trace of thought (Scott Colestock) - BizTalk Tools

New release of QuickCounters...

Scott Colestock — Thu, 17 May 2007 19:48:19 GMT

There is a new release of QuickCounters up at http://codeplex.com/quickcounters. Great new useability additions to the viewer application by Dave Comfort, as well as fewer priviliges required against remote servers. See here for the quick tour…

I’ll be talking about using QuickCounters with BizTalk at tonight’s Minnesota BizTalk User Group meeting. (Last night, I was presenting on using Scrum with Team Foundation Server at the Minnesota VSTS User Group – it’s been a busy week!)

QuickCounters update...

Scott Colestock — Thu, 01 Feb 2007 16:56:13 GMT

Another release of QuickCounters is available - check it out! Improved WCF support, and enhancements to the viewer. You really have to play with the library a bit to realize what a huge benefit it can provide both to operations and performance test investments (for fairly little investment on your part...)

Our CodePlex home is here. Downloads and source code here.

Introducing QuickCounters...

Scott Colestock — Thu, 14 Dec 2006 22:53:19 GMT

Have you ever wanted to get real-time metrics on: How many times has a particular orchestration executed on this node in the BizTalk group? What does the average execution time for my pipeline component looking like? How many instances of a particular orchestration are currently executing across the group, not including dehydrated instances?

Questions like this can be tough to answer - the built-in BizTalk counters are too coarse-grained, and Tracking & BAM are only partially useful here (particularly if you want high frequency samples.) What you would like is the ability to get fine-grained "request metrics" on all of your BizTalk orchestrations and pipelines in a way that is simple to implement and easy to consume.

I've been leading an open source project called QuickCounters for quite some time now - though we've just recently moved into CodePlex. It is a library that will be of interest to BizTalk developers - but it can be used in much broader settings. I had a chance to present on this topic at the Twin Cities .NET Users Group (on 11/2/2006 - presentation here.)

QuickCounters is a .NET library which makes it extremely easy to implement the common, request-level performance counters that are interesting in just about any service you might write. You can use this library to instrument general .Net components, web services, BizTalk orchestrations, pipeline components, remoting interfaces, enterprise service components...you get the idea. There is special support in the library for BizTalk and WCF scenarios, but it is quite easy to use in any case.

QuickCounters recognizes that the Windows Performance Counter technology that has been with us for some time is often still the best choice for providing (and consuming) detailed performance metrics. QuickCounters also recognizes that for any given service request there are several metrics which turn out to be interesting for performance testing, operational health analysis, and historical trending. The idea with QuickCounters is to raise the level of abstraction from that of an individual performance counter up to the request itself, using a simple API.

Suppose you want to gather these metrics for each type of request in your system:

Requests Started
Requests Executing
Requests Completed
Requests Failed
Request Execution Time
Requests/Hour
Requests/Min
Requests/Sec

Although the .NET performance classes would give you a good start, it will still be a chunk of work. With QuickCounters, you describe your requests in a simple xml format, and include a simple code snippet in each request implementation:

void SampleRequest()
{
   RequestType someRequest = RequestType.Attach("MyApplication","someRequest");
   someRequest.BeginRequest();

   try
   {
      // Do useful work...
      someRequest.SetComplete();
   }
   catch
   {
      someRequest.SetAbort();
      throw;
   }
}

After a quick "install" of the xml that describes your requests (via the included "QuickCounter viewer" utility) you are on your way. You can see all eight metrics described above for all of your requests (which each become a Performance object) - in PerfMon, with MOM, or any other performance counter consumer.

For a BizTalk orchestration, you will simply have a variable of type RequestType, which you will assign in an expression shape at the top of your orchestration via the static "RequestType.Attach" method, followed by a call to BeginRequest:

quickCounterDemo = QuickCounters.RequestType.Attach(
    "QuickCounterDemo","demo",
    QuickCounterDemo(Microsoft.XLANGs.BaseTypes.InstanceId)); 
quickCounterDemo.BeginRequest();

Successful completion should end with another expression shape that calls SetComplete, otherwise SetAbort. (Full sample referenced below.) And yes, the library understands that an orchestration's execution will often begin on one node in your BizTalk group and continue/complete on a different node.

Of course, if you have many requests, and many nodes in your BizTalk group, it can be a hassle to add everything to PerfMon. 20 requests * 3 servers = 60 interactions with PerfMon's UI. You might remember to save an .msc file, but you might not. Here is where the QuickCounter viewer comes into play. Since you've already written an xml file that describes your requests, just let the viewer know about that file - and a comma separated list of servers you are running on:

(click)

(The viewer has benefitted substantially from John Thom's contributions...)

Now, the WCF support in the library is...amazing, and completely the result of contributions from Tomas Restrepo. Keep an eye on his blog for a full discussion, but the bottom line is that creation of the simple xml format and all API calls (BeginRequest/SetComplete/SetAbort) are handled for you automatically simply by applying an attribute (or a behavior - your choice) to your WCF service implementation. How cool is that?

Our CodePlex home is here. Downloads and source code here. Contributors here.

I expect we'll release about once a month or so for awhile - give it a spin and let us know what you think. It is being used by some very large BizTalk projects (in production) with good success right now.

More Binding File Magic...

Scott Colestock — Fri, 12 Aug 2005 18:32:47 GMT

In my last entry, I discussed some ways that can making working with binding files a bit easier. Here is another post in that same vein that addresses a common pain point...

Un-escaping TransportTypeData

One of the annoying things about binding files is that adapters only have a string element available to store adapter-specific information for send ports and receive locations. As a result, adapters will store escaped XML (or even "doubly escaped" xml...) This can be extremely hard to manage, especially for adapters such as MQSeries that keep quite a bit of information in this form.

To solve this problem, I introduced a new command-line tool in the most recent version of the Deployment Framework called "ElementTunnel.exe" (the source for which is in the Tools download.) This utility will take in an xml file, along with a file containing xpaths to elements that should be "encoded" or "decoded". The end result is that you can choose to manage a "master" binding file (not directly useable) and run ElementTunnel on it immediately prior to deployment. (You may also run XmlPreProcess on the same file for macro expansion! The sample in the deployment framework shows both occurring - XmlPreProcess should occur first!)

So what does this mean? An example for a single Send Port snippet: It means that, in the case of MQSeries, instead of storing and maintaining this mess:

<SendPort Name="SomeQueue" IsStatic="true" IsTwoWay="false">
<TransmitPipeline Name="SomeAssembly.SomeQueue" FullyQualifiedName="SomeAssembly.SomeQueue,
SomeAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=bb955d799cc915b9" Type="2" />
<PrimaryTransport>
<Address>MQS://SomeServer/SomeQM/SomeQueue</Address>
<TransportTypeData><CustomProps><AdapterConfig vt="8">&lt;Config xmlns:xsd=
;"http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&
amp;gt;&lt;uri&gt;MQS://SomeServer/SomeQM/ SomeQueue& amp;lt;/uri&gt;&lt;queueDetails&
amp;gt;SomeServer/SomeQM/SomeQueue&lt;/queueDetails&
amp;gt;&lt;transactionSupported&gt;yes&
amp;lt;/transactionSupported&gt;&lt;dataConversion&gt;no&lt;/dataConversion&
amp;gt;&lt;segmentationAllowed&gt;no&lt;
/segmentationAllowed&gt;&lt;fragmentationSize&
amp;gt;500&lt;/fragmentationSize&gt;&lt;ordered& amp;gt;no&lt;/ordered&gt;&lt;/Config&
amp;gt;</AdapterConfig& gt;</CustomProps></TransportTypeData>
<RetryCount>3</RetryCount>
<RetryInterval>5</RetryInterval>
...
</SendPort>> >

You can store and maintain this:

<SendPort Name="SomeQueue" IsStatic="true" IsTwoWay="false">
<TransmitPipeline Name="SomeAssembly.SomeQueue" FullyQualifiedName="SomeAssembly.SomeQueue,
SomeAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=bb955d799cc915b9" Type="2" />
<PrimaryTransport>
<Address>MQS://SomeServer/SomeQM/SomeQueue</Address>
<TransportTypeData>     <CustomProps>
        <AdapterConfig vt="8">
            <Config>
                <uri>MQS://SomeServer/SomeQM/SomeQueue</uri>
                <queueDetails>SomeServer/SomeQM/SomeQueue</queueDetails>
                <transactionSupported>yes</transactionSupported>
                <dataConversion>no</dataConversion>
                <segmentationAllowed>no</segmentationAllowed>
                <fragmentationSize>500</fragmentationSize>
                <ordered>no</ordered>
            </Config>
        </AdapterConfig>
    </CustomProps> </TransportTypeData>
<RetryCount>3</RetryCount>
<RetryInterval>5</RetryInterval>
...
</SendPort>

Ahhh, isn't that better? Of course, similar goodness for all other adapters. And, in the clean version, you'll find it easier to place/maintain XmlPreProcess macros.

In the Deployment Framework sample, you'll see that we pass the following xpaths to ElementTunnel (along with the "master" binding file itself):

/BindingInfo/ReceivePortCollection/ReceivePort/ReceiveLocations/ReceiveLocation/ ReceiveLocationTransportTypeData/CustomProps/AdapterConfig
/BindingInfo/ReceivePortCollection/ReceivePort/ReceiveLocations/ReceiveLocation/ ReceiveLocationTransportTypeData
/BindingInfo/SendPortCollection/SendPort/*/TransportTypeData/CustomProps/AdapterConfig
/BindingInfo/SendPortCollection/SendPort/*/TransportTypeData

If you want to "unescape" your binding file (generally a one-time thing, just to get clean content) you'll want to pass these xpaths in a slightly different order, because of the "double escaping":

/BindingInfo/ReceivePortCollection/ReceivePort/ReceiveLocations/ReceiveLocation/ ReceiveLocationTransportTypeData
/BindingInfo/ReceivePortCollection/ReceivePort/ReceiveLocations/ReceiveLocation/ ReceiveLocationTransportTypeData/CustomProps/AdapterConfig
/BindingInfo/SendPortCollection/SendPort/*/TransportTypeData
/BindingInfo/SendPortCollection/SendPort/*/TransportTypeData/CustomProps/AdapterConfig

So! If you are managing large binding files (where escaped xml is getting in your way), you might find this technique handy...Grab the tool, and give it a go.

Messages from Scratch with Embedded Resources

Scott Colestock — Wed, 11 May 2005 21:29:14 GMT

There have been several folks who have discussed how to create messages "from scratch" within an Orchestration context - you can read Matt's thoughts and check out a BizTalk documentation excerpt. This is a common question, and I had an additional technique I thought I would share...

Background: When you have a schema that has many promoted properties (or distinguished fields), or many elements that can be set via xpath expressions easily, it can be useful to simply start with a "template" document instance and populate the element content that you are interested in.

In this situation, you will often have a "Message Assignment" shape that looks something like this:

xmlDoc.LoadXml("<ns0:BizTalkSampleS3 
      xmlns:ns0="http://BizTalkSample.Schemas.BizTalkSampleS3">
      <SomeElement></SomeElement></ns0:BizTalkSampleS3>");
someMsg = xmlDoc;
someMsg.SomeElement = "some content";
// (or xpath(someMsg,someElementXPath) = "some content" if we don't have a 
// distinguished field.)

One disadvantage of loading up "template" xml documents from either expression shapes or code (via XmlDocument.LoadXml) is that those xml fragments can get easily "lost", and are hard to update early in the development cycle when schemas may still be in flux. Loading the template files from the file system is problematic because the question arises "where should I store these files, so that I can find them in any environment I deploy to?" (Solvable, but a hassle.)

Instead, why not embed the template xml documents as assembly resources? For those unfamiliar with that process, I have a short tutorial here (& a helper class.)

You will need a C# project as part of your overall BizTalk solution. Place your template xml file(s) in the directory corresponding to this project, and add them as an "existing item" to the project.
Select this file within the Solution Explorer, and within the Properties window, select "Embedded Resource" as the "Build Action" as shown here:
Place this class (text here) within the same C# project that houses the resources you have added.

To construct a message, drag out a "Message Assignment" shape, and within the associated expression write some code like the following. Simply pass the file name of the template document as an argument to GetXmlDocResource (or GetStringResource.)

sampleUsingTemplate = 
   BizTalkSample.Components.EmbeddedResourceCache.
      GetXmlDocResource("BizTalkSampleS3_output.xml");
   // Populate the "rest" of the message with distinguished fields, promoted 
   // properties, xpath expressions, etc.
   sampleUsingTemplate.SomeElement = "foo";

The class I have supplied will cache the loaded resources in a hashtable for performance sake, and allow you to load resources as both strings and XmlDocuments.

A last thought: Many people ask, "Why can't I just create a message using a new operator or a default constructor of some sort?" Well, because few XSD schemas sufficiently constrain the set of valid instance documents enough for that to be useful - what form would a "default message" take? (Would it have the optional elements you need? Some elements that you don't want?)

Enjoy - feedback appreciated!