There is a new release of QuickCounters up at http://codeplex.com/quickcounters.  Great new useability additions to the viewer application by Dave Comfort, as well as fewer priviliges required against remote servers.  See here for the quick tour…

I’ll be talking about using QuickCounters with BizTalk at tonight’s Minnesota BizTalk User Group meeting.  (Last night, I was presenting on using Scrum with Team Foundation Server at the Minnesota VSTS User Group – it’s been a busy week!)

We were having a problem with a BizTalk assembly (containing only maps) having a “project” reference to a standard C# library.  As soon as the project reference was added, we were unable to open the maps – errors along the lines of “unable to open source schema” and “unexpected error encountered…vsee\internal\inc\vscomptr.inl”.  See here for a good forum discussion on the issue that we came across.

It turns out this issue crops up if you have a VSTS unit testing project in the same solution…How is that for a strange product interaction?  Possible workarounds include removing the unit testing project (and putting it in a separate solution) or using a file reference instead of a project reference.  (If you go the latter route, go to the properties of your solution and set up the project build dependencies manually to account for this!)

We opted for file references rather than removing our unit test assembly, at least for the time being.  Hopefully this will be addressed soon…

So...almost three years ago (gulp) I wrote this article on the use of orchestration naming conventions.  In the post, I described the value of orchestration diagrams and their use throughout the development cycle.  In my current engagement, we are doing analysis on the integrations between ERP, warehousing, front-end commerce systems in multiple channels, and a PIM (Product Information Management) solution.  I'm finding that the BizTalk designer - used just as a modeling tool with the constructs that it provides but not filling in deep detail - eliminates so much of the ambiguity that a simple Visio can leave on the table.  So I thought I'd repeat my (now old) thoughts on this topic:

The opportunity exists to use an orchestration diagram in several interesting ways within a project lifecycle:

1. As a way to capture an initial high-level design, using all the orchestration shapes as intended but not yet bothering with real maps and schemas.  Stubbing out schemas (so you can declare messages and variables to be of proper types) and maps will allow you to flesh out the orchestration diagram(s) quite a bit, using the compiler as just a way to check for consistency.  All of the external system interactions, communication patterns, decision points, parallel vs. joined flows, etc. can be represented at this point in a shell orchestration.
2. As a way to gain consensus with the development team & business sponsor about whether the right functionality is indeed going to be built.  The high level design just described is a great tool for this discussion.  Put your orchestration(s) up on a wall with a projector and do a walk-through with as many of the project stakeholders as makes sense.  Or use a tool like CutePDF to print the orchestration as a PDF to send around via email.
3. As a way to estimate work.  The various shapes in your initial orchestration can often represent reasonable granularity for time estimates.
4. And finally, as a way to guide project work...Rather than starting with the entire orchestration that you created to support steps 1-3, you might find it easier to create a new orchestration that represents the path(s) you are tackling at a particular point.  You can cut/paste portions of that original orchestration or simply use it as a reference for what comes next it serves as your outline.

To help realize some of these benefits, naming conventions within an orchestration are quite important

While the naming conventions are good practice for variables, Messages, Multi-Part types, etc. they are even more import for the workflow shapes.  The goal is to ensure that the intent of each shape is clear, and that the text associated with the shape conveys as much as possible given the space constraints.  Make liberal use of group shapes where needed.  In this way, a non-technical audience will be able to use the orchestration as documentation.

Another release of QuickCounters is available - check it out! Improved WCF support, and enhancements to the viewer. You really have to play with the library a bit to realize what a huge benefit it can provide both to operations and performance test investments (for fairly little investment on your part...)

Our CodePlex home is here. Downloads and source code here.

Have you ever wanted to get real-time metrics on:  How many times has a particular orchestration executed on this node in the BizTalk group?  What does the average execution time for my pipeline component looking like?  How many instances of a particular orchestration are currently executing across the group, not including dehydrated instances?

Questions like this can be tough to answer - the built-in BizTalk counters are too coarse-grained, and Tracking & BAM are only partially useful here (particularly if you want high frequency samples.)  What you would like is the ability to get fine-grained "request metrics" on all of your BizTalk orchestrations and pipelines in a way that is simple to implement and easy to consume.

I've been leading an open source project called QuickCounters for quite some time now - though we've just recently moved into CodePlex.  It is a library that will be of interest to BizTalk developers - but it can be used in much broader settings.  I had a chance to present on this topic at the Twin Cities .NET Users Group (on 11/2/2006 - presentation here.)

QuickCounters is a .NET library which makes it extremely easy to implement the common, request-level performance counters that are interesting in just about any service you might write.  You can use this library to instrument  general .Net components, web services, BizTalk orchestrations, pipeline components, remoting interfaces, enterprise service components...you get the idea.  There is special support in the library for BizTalk and WCF scenarios, but it is quite easy to use in any case.

QuickCounters recognizes that the Windows Performance Counter technology that has been with us for some time is often still the best choice for providing (and consuming) detailed performance metrics.  QuickCounters also recognizes that for any given service request there are several metrics which turn out to be interesting for performance testing, operational health analysis, and historical trending. The idea with QuickCounters is to raise the level of abstraction from that of an individual performance counter up to the request itself, using a simple API.

Suppose you want to gather these metrics for each type of request in your system:

• Requests Started
• Requests Executing
• Requests Completed
• Requests Failed
• Request Execution Time
• Requests/Hour
• Requests/Min
• Requests/Sec

Although the .NET performance classes would give you a good start, it will still be a chunk of work.  With QuickCounters, you describe your requests in a simple xml format, and include a simple code snippet in each request implementation:

void SampleRequest()
{
   RequestType someRequest = RequestType.Attach("MyApplication","someRequest");
   someRequest.BeginRequest();

   try
   {
      // Do useful work...
      someRequest.SetComplete();
   }
   catch
   {
      someRequest.SetAbort();
      throw;
   }
}
        

After a quick "install" of the xml that describes your requests (via the included "QuickCounter viewer" utility) you are on your way.  You can see all eight metrics described above for all of your requests (which each become a Performance object) - in PerfMon, with MOM, or any other performance counter consumer. 

For a BizTalk orchestration, you will simply have a variable of type RequestType, which you will assign in an expression shape at the top of your orchestration via the static "RequestType.Attach" method, followed by a call to BeginRequest: 

quickCounterDemo = QuickCounters.RequestType.Attach(
    "QuickCounterDemo","demo",
    QuickCounterDemo(Microsoft.XLANGs.BaseTypes.InstanceId)); 
quickCounterDemo.BeginRequest();           

Successful completion should end with another expression shape that calls SetComplete, otherwise SetAbort.  (Full sample referenced below.)  And yes, the library understands that an orchestration's execution will often begin on one node in your BizTalk group and continue/complete on a different node. 

Of course, if you have many requests, and many nodes in your BizTalk group, it can be a hassle to add everything to PerfMon.  20 requests * 3 servers = 60 interactions with PerfMon's UI.  You might remember to save an .msc file, but you might not.  Here is where the QuickCounter viewer comes into play.  Since you've already written an xml file that describes your requests, just let the viewer know about that file -  and a comma separated list of servers you are running on:

(click)

(The viewer has benefitted substantially from John Thom's contributions...)

Now, the WCF support in the library is...amazing, and completely the result of contributions from Tomas Restrepo.  Keep an eye on his blog for a full discussion, but the bottom line is that creation of the simple xml format and all API calls (BeginRequest/SetComplete/SetAbort) are handled for you automatically simply by applying an attribute (or a behavior - your choice) to your WCF service implementation.  How cool is that?

Our CodePlex home is here.  Downloads and source code here.  Contributors here.

I expect we'll release about once a month or so for awhile - give it a spin and let us know what you think.  It is being used by some very large BizTalk projects (in production) with good success right now.

There are two interesting BizTalk 2006 bugs to be aware of - it seems like they would be encountered fairly often, so I'm a little surprised that I haven't seen more newsgroup/forum activity around these.

The first bug is encountered when you attempt to start a BizTalk 2006 application definition with orchestrations that have been bound to different hosts.  (Say, OrchA in HostA, and OrchB in HostB.)  After the application has been deployed, the first time the application is started (and 'orchestration start/enlist' is included in the start options) you will receive an error that looks something like this:

Microsoft.BizTalk.ExplorerOM.BtsException: Could not enlist orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Could not enlist orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Value does not fall within the expected range.

You can work around this error by manually starting your orchestrations in the MMC, either one-at-a-time or grouped by host.  The error only occurs the first time you attempt to start - all future attempts will succeed.  This is because the underlying bug has to do with the creation of subscriptions for the orchestrations. 

This is actually a regression from BizTalk 2004 - where you could in fact change multiple orchestrations (across differing hosts) to a started state via the BtsOrchestration 'Explorer OM' class in a single transaction (i.e. before calling SaveChanges.)  In addition to the behavior just described with BizTalk 2006, you can't do this kind of Explorer OM work once multiple hosts are involved in BizTalk 2006.

But hey, there is a hotfix.  Contact PSS and ask about KB927052.

The second bug is encountered when you have a BizTalk 2006 application with multiple MSMQ receive locations.  You will find that restarting your host instances takes an excessively long period of time.  This is due to a bug in the adapter which was not allowing the host service to stop correctly (the Service Control Manager would eventually just terminate the service...)  Contact PSS and ask about KB928222.

Update: The tracking query as a TRQ file is available here, per a comment on this post.

I just got back from a week at the "Microsoft SOA & Business Process" conference in Redmond.  Lots of discussion on BizTalk 2006 R2, the new .NET Adapter Framework, Windows Communication Foundation (WCF), Windows Workflow (WF), Microsoft Office Sharepoint Server 2007 (MOSS).  Great stuff.

I did a session at the conference on Friday titled "Applying Maximum Sustainable Throughput to a Management/Operations Strategy".  (OK, the title may have scared a few people off...) The content was an extension of what I presented to the Twin Cities BizTalk User Group in September.  The overall theme was relating what you learn during your performance testing phase to your operations strategy, and ensuring that you have application-level metrics that represent operational boundaries you would like to "live within".  In particular, I discussed how to formulate your performance goals, how to measure against them as operational parameters, and how to proactively monitor with a tool such as Microsoft Operations Manager (MOM).  We went on to discuss how to choose the workloads (e.g. send, receive, or orchestration work) for new servers that you place in your BizTalk group, depending on what you see within various BizTalk work queues and throttling states.  I've put the slides for the session here for your perusal. 

One of the demos showed a HAT query I've worked up which will report on how many times each orchestration has run within a given time window, along with the percent of total transaction volume it represents, time duration, and activity rates (per second, per minute, and per hour). You can put the following query in a .TRQ file (like OrchestrationDurationsAndRates.trq) and the put it within C:\Program Files\Microsoft BizTalk Server 2006\Tracking\Queries (or the equivalent on your server.) It will then appear within the "Queries" menu in HAT. This can be a genuinely useful query to have at your disposal, both during performance testing and in production.  You can run this in Query Analyzer if you prefer - just uncomment the lines that define UtcOffsetMin.  Note that rate calculations are dividing across the entire time window, which assumes a fairly steady traffic pattern.

If you're trying to determine compliance within an SLA such as "95% of all requests must be satisfied within 15 seconds or less", you might add an item to the select clause below along the lines of "average(service duration) + (2*stddev(service duration))", if the distribution of your transaction durations tends to be normal (per your high school stats class.)

declare @beginTime as datetime
declare @endTime as datetime

declare @TotalCountInWindow as int

--
--  SET THE TIME WINDOW FOR YOUR QUERY HERE
--
select @beginTime =DateAdd(hour, -2, GetDate())
select @endTime = GetDate()

-- Only need to uncomment this select when inside of query analyzer,
-- where UtcOffsetMin won't be defined.
-- declare @UtcOffsetMin as int
-- select @UtcOffsetMin = 
-- -1*DateDiff("mi",CONVERT(varchar,GetDate(),0),CONVERT(varchar,GetUTCDate(),0))

select @beginTime = dateadd(minute,-1*@UtcOffsetMin,@beginTime)
select @endTime = dateadd(minute,-1*@UtcOffsetMin,@endTime)

SELECT @TotalCountInWindow = count(*)
FROM dbo.dtav_ServiceFacts sf WITH (READPAST)
where [ServiceInstance/StartTime] between
@beginTime and @endTime
and [ServiceInstance/State] = 'Completed'
and [Service/Type] = 'Orchestration'


SELECT 
[Service/Name],
count(*) as TotalCount,
(cast(count(*) as float)/cast(@TotalCountInWindow as float))*100 as PercentOfTotal,
avg([ServiceInstance/Duration]/1000.0) as AverageDuration,
min([ServiceInstance/Duration]/1000.0) as MinDuration,
max([ServiceInstance/Duration]/1000.0) as MaxDuration,
stdev([ServiceInstance/Duration]/1000.0) as StdDevDuration,
count(*)/DateDiff("hh",@beginTime,@endTime) as perHour,
count(*)/DateDiff("mi",@beginTime,@endTime) as perMinute,
cast(count(*) as float)/DateDiff("ss",@beginTime,@endTime) as perSecond

FROM dbo.dtav_ServiceFacts sf WITH (READPAST)
where [ServiceInstance/StartTime] between
@beginTime and @endTime
--and  [ServiceInstance/Duration] > 0
and [ServiceInstance/State] = 'Completed'
and [Service/Type] = 'Orchestration'
group by [Service/Name]

Got your own favorite HAT query? Post it in the comments - maybe we'll start a new section for them. Some shops deploy their custom HAT queries directly with their BizTalk applications...For Deployment Framework folks, that looks as follows (assuming a project subdirectory called DeployedHATQueries):

	
<target name="deployHatQueries">
  <copy todir="${btsDir}\Tracking\Queries" overwrite="true">
      <fileset basedir="DeployedHATQueries">
         <include name="**\*.trq" />
      </fileset>
  </copy>
</target>

Late notice (again) but if you can…be sure to join us at 6:00 pm on Thursday, September 21st at the local Microsoft office in Bloomington.  You can register here.

Scott Colestock (me) will be presenting this month on the topic of relating what you learn during performance testing of a BizTalk application to your operations/monitoring strategy.  Should be good fun (really!) and of course we’ll have the usual food and beverages.

Despite all the functional testing and stress testing you do prior to releasing your BizTalk app into production, unexpected behavior can (and often will) happen just the same.  Production usage just winds up introducing all sorts of permutations (including interactions with external systems) that are hard to predict earlier in the lifecycle.

The goal, of course, is to minimize the the operational "care and feeding" that an application requires over time.  Making this happen is mostly a function of using the application's "diagnostic surface area" (logs, counters, MOM packs, etc.) to feed back into each release cycle.  But we also need post-mortem tools when the host environment terminates unexpectedly or stops responding (whether that environment is BizTalk, IIS, COM+, Sql SSIS, etc.)

While a well-designed app will be able to successfully restart and resume processing (with full data integrity) at such a point (i.e. after the host has been terminated), there is still operational expense that has been injected.  We want to find and eliminate these problems...

Using the Visual Studio debugger is almost never an option in production, of course.  We need the ability to capture the current state as a "dump file" and do offline analysis. 

The "Windows Debugging Tools" are designed for this purpose (and you will often use these during a call with Microsoft's support staff, so it is good to be familiar with them.)  The debugging tools are a pretty large subject - so here, we are just going to cover the bare minimum required to capture a dump file for your running BizTalk process when it appears to be hung with a large number of "Active" service instances.

Step By Step:

• Install or xcopy the Windows Debugging Tools to the server where BizTalk is currently hung (or crashing unexpectedly.)  It can be helpful to install in an easy location for command line access like 'c:\debuggers'.
• From command line, run the following from the command line to get process IDs for all BizTalk hosts:
  typeperf "\BizTalk:Messaging(*)\ID Process" -sc 1
• Run 'adplus.vbs' in crash or hang mode, depending on whether the process ends unexpectedly (crash) or has become unresponsive (hang).  To generate a hang dump, your command line might look like:
  c:\debuggers\cscript adplus.vbs –hang –CTCF –p (pid from last step) –o c:\temp
• Copy the dump file to an offline location if need be.
• Set an envrionment variable called '_NT_SYMBOL_PATH' to 'srv*c:\symbols*http://msdl.microsoft.com/download/symbols'.  Alternatively, launch WinDbg.exe from the debuggers directory and use the File-'Symbol File Path' menu.  This will ensure that you are automatically downloading the correct symbols when you analyze the crash dump.  
• Start WinDbg.exe, and use File-'Open Crash Dump' to open your dump file.  Then, in the command window, use:
  '.load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll' to load managed code debugging extensions.
• In the command window, use !EEStack to get a full stack trace.  Use Edit-Find to search for your custom code method name or the name of your orchestration.  Look for patterns that indicate the cause of the hang ("hmmm, all my threads seem to be inside Thread.Sleep. That's funny.")  Use !help from the command window to begin learning about the rest of SOS (to assist with diagnosing managed memory leaks, etc.)

For more information on the Windows debugging tools, see here.