There has to be a better way to do this – if you know of one, kindly post a comment.

Until then, here is a technique that will allow you to ensure that if a stored procedure that BizTalk is calling is unable to complete its work successfully, you can rollback the work and be notified of that fact within your orchestration.

1. In the stored procedure, use RAISERROR with a severity greater than 10 in the event of an error.  This should abort the (DTC) transaction that the BizTalk Sql Adapter initiated (that wraps the work within the stored procedure.)
2. Use a scope around the Send/Recieve shapes in your orchestration that deal with this stored procedure.  You will want to catch Microsoft.XLANGs.Core.XlangSoapException…but you can’t.  The designer won’t let you.  So…be brave, open the ODX, and look for the catch block so you can modify it as shown below (see “ExceptionType”):

<om:Element Type="Catch" OID="e7590870…" ParentLink="Scope_Catch" …>
  <om:Property Name="ExceptionName" Value="soapex" /> 
  <om:Property Name="ExceptionType" Value="Microsoft.XLANGs.Core.XlangSoapException" />
  <om:Property Name="IsFaultMessage" Value="False" /> 
  <om:Property Name="ReportToAnalyst" Value="True" />
  <om:Property Name="Name" Value="CatchException_2" />
  <om:Property Name="Signal" Value="True" /> 
</om:Element>

3. With this solution, you will wind up with the message you are sending to the stored procedure in a suspended state whenever the RAISEERROR occurs.  This isn't a huge deal, but you have to plan how these will get cleaned up.  It could be done manually, with a script, or with an orchestration or send port that subscribes to these suspended messages (try a filter with ErrorReport.FailureCode == 0xc0c0167a to get started, though you probably want to be more specific.)

After you’ve gone to these lengths, it is always worth considering whether you want to have your orchestration call a component so you are a little closer to the action.  But, if you have need of the adapter, this should work out.

BizTalk 2006 R3 was announced today!  This answers the frequently recurring question of “When will BizTalk live inside of Visual Studio 2008?”

Well, this release will make that happen…In addition, R3 will provide support for Windows Server 2008 and Sql Server 2008.  You can expect a release after general availability of Sql Server 2008…

Like R2, this is a pretty incremental release.  Steven Martin indicates that releasing fresh bits is a better user experience than a “large Service Pack”…well, at any rate, it always seems like a good idea to avoid delivering new features in a service pack.  Expected to see some new (and updated) LOB, legacy, and database adapters – in addition to a mobile RFID piece…

I sat down with Jeff Brand from Microsoft to discuss “BizTalk for .NET developers”.  We touched on how BizTalk fits in with a typical SOA effort, dealing with the learning curve, typical usage scenarios, and several other topics.  Great fun.  You can grab the MP3 here if you like.

This bit of documentation explains that when you want to send a message out over the MQSeries adapter and receive a correlated response, the typical pattern is to:

1. Generate a value for the MQMD_MsgID property on the outbound message
2. Set the MQMD_CorrelID property on the outbound message to that same generated value, so that your “Send” shape can initialize a correlation that refers to the same property that will be present on the message you receive.
3. Send the message (initialize correlation with MQMD_CorrelID)
4. Receive the response (follow correlation with MQMD_CorrelID)

This assume that the destination system is following the usual pattern of setting the correlation identifier on the reply message equal to the message identifier on the request message.

Great.  Except when you don’t have the luxury of specifying the MQMD_CorrelID message on the outbound message to a value of your liking.  This occurs when you’re using a CICS bridge, or anything else that requires you to set the MQMD_CorrelID property to IBM.WMQ.MQC.MQCI_NEW_SESSION (signifying a new session interaction…)

In this case…you might be attempted to use the other pattern suggested here.  In this case, you engage in a solicit-response interaction with the adapter – where you send your request message and receive an immediate reply that contains a message with BizTalk_CorrelationID populated as a context property.  The message ID has also been generated for you.  You are then supposed to do a “dummy send” to initialize a correlation based on the BizTalk_CorrelationID property (sigh), and then follow that correlation with another receive shape that is the actual transaction response. 

To see this in action, see SDK\Samples\AdaptersUsage\MQSeriesAdapter\MQSCorrelationSetOrchestration under your BizTalk installation, or see the orchestration here.

Unfortunately, this design lends itself to a race condition.  The response coming back can occur prior to your correlation being initialized by the “dummy send”, resulting in a routing failure.  (This isn’t hypothetical – I just spent some time helping someone debug this issue.)

To work around this, you could try to do content-level correlation instead (i.e. use a promoted property in your request and response content.)

Wayyyy back in November of 2006, I wrote about a bug in BizTalk 2006:

This bug is encountered when you attempt to start a BizTalk 2006 application definition with orchestrations that have been bound to different hosts.  (Say, OrchA in HostA, and OrchB in HostB.)  After the application has been deployed, the first time the application is started (and 'orchestration start/enlist' is included in the start options) you will receive an error that looks something like this:

Microsoft.BizTalk.ExplorerOM.BtsException: Could not enlist orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Could not enlist orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Value does not fall within the expected range.

There was a fix for this bug  back at that time, but somehow it crept back in (even in R2.)  The KB number for the problem is the same, however (KB927052) and a fix is available.

Minor update to the BizTalk naming conventions here.  The big thing is that for a long while now I’ve thought physical send/receive ports should be named with just a plainly-worded phrase that will be as transparent as possible to operations staff.  (Whereas, back in the days of BizTalk 2004, you wanted something brief enough to fit in a HAT column…)

A few folks have discussed the topic of building BizTalk solutions with TFS, namely here and here.  As most have surmised in the time since TFS originally shipped, you have to build BizTalk solutions with “devenv.exe” if you are going to use Team Build, by overriding the “AfterCompile” target.  This is because there is no “standalone” xlang compiler (at least not that you & I can use) – it instead loads in-process to the IDE.

Your steps will be something like the following:

• Do a “developer tools only” install of BizTalk on your build server.  No need to install the BizTalk services or databases, etc. (unless you intend to deploy & run post-deployment tests on the build machine.)  Note that since you need to install Team System Dev or Test to run post-build unit tests anyway, you aren’t really installing that many more “bits” on your build server (if you are skipping the BizTalk runtime/database infrastructure.)
• Comment out the line in your TFSBuild.proj file that looks like this: <SolutionToBuild Include="$(SolutionRoot)\path\MySolution.sln" />
• Create an AfterCompile target that looks something like this:

  <Target Name="AfterCompile">
 
  <!--Call DevEnv to build the BizTalk Solution-->
  <Exec Command="&quot;C:\%programfiles%\Microsoft Visual Studio 8\Common7\IDE\devenv&quot; &quot;$(SolutionRoot)\BizTalk\Dev\MyBizTalk.sln&quot; /Build &quot;$(BuildFlavor)|$(BuildPlatform)&quot;"/>

  <!-- Create a drop location -- >
  <MakeDir
   Directories="$(DropLocation)\$(BuildNumber)\$(BuildFlavor)"
   Condition="!Exists('$(BinariesRoot)\$(BuildFlavor)')" />

    <!-- Do whatever work you would like here - assemble binaries, build an MSI via WiX, etc. and
         place the outputs in the drop location. -->

 </Target>

What isn’t immediately obvious is that if you want to execute unit tests as part of your build process (i.e. tests that don’t require your BizTalk solution to be deployed…think build verification) then the DevEnv method of building can cause you trouble.  The testing portion of the build process expects your binaries to land in the “Binaries” folder — a peer of Sources, BuildType, and TestResults in your build location.

Therefore…you will want to add something like this at the end of your AfterCompile target:

    <!-- Get our build outputs into the "binaries" folder to facilitate unit testing. -->
    <MakeDir Directories="$(OutDir)"/>
    <CreateItem Include="$(SolutionRoot)\**\*.dll" >
      <Output ItemName="FilesToCopy" TaskParameter="Include" />
    </CreateItem>
    <Copy
          SourceFiles="@(FilesToCopy)"
          DestinationFolder="$(OutDir)"
          ContinueOnError="true" />

 

With this in place, the unit testing portion of the build process can execute as if you hadn’t resorted to calling devenv.exe…

Update: Are you getting an error like this: “The specified configuration and platform ('Debug/Any CPU') is not defined.” — right at the point that test results are supposed to be published?  I don’t entirely understand this, except to say that something in the core TFS targets is apparently responsible for populating the Platform_Flavors table in the TfsBuild database.  This step might not happen if you take over compiling by calling devenv.exe.  Take a look in that table to see if the platform/flavor you are trying to use is listed there…

I believe there is a problem with the "pre-converted" (from MOM 2005) management pack that is offered for BizTalk 2006 on the SCOM 2007 management pack download page.

I believe the conversion started with the BizTalk 2006 (flat) MOM 2005 management pack, rather than the BizTalk 2006 R2 version.  The latter is available here.

Assuming you have already installed the MOM 2005 Backward Compatibility Management Pack, version 6.0.5000.12, you should be able to import the 2006 R2 version (via the Operations Manager Migration Wizard.)

The reason this is important is because the computer attribute that the (older) management pack is looking for includes:

MatchWildcard(AttributeValue(BizTalkServer2006Attribute), "3.5.*")

BizTalk 2006 R2 has a version stamp of 3.6.1404.0.

(I believe other folks have run into this issue as well.)

Update: Because the import/conversion process is a bit of a hassle, I thought I’d post the SCOM-converted files here for download.  These were produced by starting with the R2 AKM files, converting to xml via mp2xml.exe (in the MOM 2005 resource kit), and then finally apply MPConvert.exe from the SCOM tools.

My article on BizTalk host throttling appeared in BizTalk Hotrod Issue 2 – Todd VanNurden and Sal Cincotta have done a great job with assembling and editing content, so be sure to take a look at the whole issue.  I’m hoping to write a follow-up piece that discusses custom throttling techniques in a future issue.

(And before you ask, I don’t own a suit nearly half as nice as the one Todd has photoshopped me into for the magazine.)

Have you ever created a custom monitoring or troubleshooting tool (or data store) for working with BizTalk, and wanted the ability to launch the HAT (Health and Activity Tracker) tool in a particular context?  Here are the command line parameters that will enable you to do just that…

First, though you can typically launch the HAT tool from "C:\Program Files\Microsoft BizTalk Server 2006\BTSHatApp.exe", for a production tool you probably want to read the install location of BizTalk from the registry at “SOFTWARE\Microsoft\BizTalk Server\3.0\InstallPath.”

The command line syntax looks like:

BTSHatApp.exe “btshat:“ViewName:<OrchDebugger|MessageFlow|SvcInstQuery> MgmtSvr:<your server> MgmtDb:<your mgmt db> SvcInstId:<service instance id> [MsgId:]<optional msg id>””

OrchDebugger and MessageFlow bring up the expected HAT views that you are familiar with.  SvcInstQuery will actually open the BizTalk Admin console, with a query set up so you can see a live service instance with associated messages, etc.

You can also launch with the name of a saved tracking query, as in:

BTSHatApp.exe "c:\program files\microsoft biztalk server 2006\tracking\queries\MessagesSentInPastDay.trq"

The BizTalk management pack for Microsoft Operations Manager appears to use this command line interface so that it can provide built-in tasks to launch HAT with the context present in captured event log entires.  This means that if you have the MOM operator console installed on a machine that is joined to a BizTalk group, you have a nicely integrated set of troubleshooting tools.

A flat file schema can be configured to generate empty elements for empty content – using the “Generate Empty Nodes” option.

It seems that at runtime, the flat file dissasembler can generate “<foo></foo>” for these empty nodes in cases where “Validate Instance…” in the IDE would have generated “<foo/>”.

Where this can cause pain is if you receive a message like: “<foo><bar></bar></foo>”

into your orchestration, and then do a series of assignments like:

xmlDocVariable = myMessage;
// xmlDocVariable2 initialized because xmlDocVariable is needed for other things...
xmlDocVariable2.LoadXml(xmlDocVariable.OuterXml);
myMessage2 = xmlDocVariable2;

At this point, myMessage2 is going to be the serialized representation of xmlDocVariable2.  By default, you will now get:

<foo>
..<bar>
..</bar>
</foo>

Notice you now have carriage returns, line feeds, and spaces (shown here as periods) involved.  Depending on how you now go after /foo/bar, this can be bad (i.e. you won’t get empty content when you might expect it.)  You can avoid this behavior by doing:

xmlDocVariable2.LoadXml(xmlDocVariable.OuterXml);
xmlDocVariable2.PreserveWhitespace = true;
myMessage2 = xmlDocVariable2;

Thanks to Tomas and Carlos for setting me straight on PreserveWhitespace – I was trying to set it before the call to LoadXml, and this doesn’t work.  Never a dull day in BizTalk land…

The documentation for PreserveWhitespace is a little strange.  This functionality would seem to be better represented as parameters to Load/Save, rather than as a property…

“If PreserveWhitespace is true before Load or LoadXml is called, white space nodes are preserved; otherwise, if this property is false, significant white space is preserved, white space is not.  If PreserveWhitespace is true before Save is called, white space in the document is preserved in the output; otherwise, if this property is false, XmlDocument auto-indents the output.”

We were having a problem with a BizTalk assembly (containing only maps) having a “project” reference to a standard C# library.  As soon as the project reference was added, we were unable to open the maps – errors along the lines of “unable to open source schema” and “unexpected error encountered…vsee\internal\inc\vscomptr.inl”.  See here for a good forum discussion on the issue that we came across.

It turns out this issue crops up if you have a VSTS unit testing project in the same solution…How is that for a strange product interaction?  Possible workarounds include removing the unit testing project (and putting it in a separate solution) or using a file reference instead of a project reference.  (If you go the latter route, go to the properties of your solution and set up the project build dependencies manually to account for this!)

We opted for file references rather than removing our unit test assembly, at least for the time being.  Hopefully this will be addressed soon…

So...almost three years ago (gulp) I wrote this article on the use of orchestration naming conventions.  In the post, I described the value of orchestration diagrams and their use throughout the development cycle.  In my current engagement, we are doing analysis on the integrations between ERP, warehousing, front-end commerce systems in multiple channels, and a PIM (Product Information Management) solution.  I'm finding that the BizTalk designer - used just as a modeling tool with the constructs that it provides but not filling in deep detail - eliminates so much of the ambiguity that a simple Visio can leave on the table.  So I thought I'd repeat my (now old) thoughts on this topic:

The opportunity exists to use an orchestration diagram in several interesting ways within a project lifecycle:

1. As a way to capture an initial high-level design, using all the orchestration shapes as intended but not yet bothering with real maps and schemas.  Stubbing out schemas (so you can declare messages and variables to be of proper types) and maps will allow you to flesh out the orchestration diagram(s) quite a bit, using the compiler as just a way to check for consistency.  All of the external system interactions, communication patterns, decision points, parallel vs. joined flows, etc. can be represented at this point in a shell orchestration.
2. As a way to gain consensus with the development team & business sponsor about whether the right functionality is indeed going to be built.  The high level design just described is a great tool for this discussion.  Put your orchestration(s) up on a wall with a projector and do a walk-through with as many of the project stakeholders as makes sense.  Or use a tool like CutePDF to print the orchestration as a PDF to send around via email.
3. As a way to estimate work.  The various shapes in your initial orchestration can often represent reasonable granularity for time estimates.
4. And finally, as a way to guide project work...Rather than starting with the entire orchestration that you created to support steps 1-3, you might find it easier to create a new orchestration that represents the path(s) you are tackling at a particular point.  You can cut/paste portions of that original orchestration or simply use it as a reference for what comes next it serves as your outline.

To help realize some of these benefits, naming conventions within an orchestration are quite important

While the naming conventions are good practice for variables, Messages, Multi-Part types, etc. they are even more import for the workflow shapes.  The goal is to ensure that the intent of each shape is clear, and that the text associated with the shape conveys as much as possible given the space constraints.  Make liberal use of group shapes where needed.  In this way, a non-technical audience will be able to use the orchestration as documentation.

There are two interesting BizTalk 2006 bugs to be aware of - it seems like they would be encountered fairly often, so I'm a little surprised that I haven't seen more newsgroup/forum activity around these.

The first bug is encountered when you attempt to start a BizTalk 2006 application definition with orchestrations that have been bound to different hosts.  (Say, OrchA in HostA, and OrchB in HostB.)  After the application has been deployed, the first time the application is started (and 'orchestration start/enlist' is included in the start options) you will receive an error that looks something like this:

Microsoft.BizTalk.ExplorerOM.BtsException: Could not enlist orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Could not enlist orchestration 'BizTalkSample.Orchestrations.Echo,BizTalkSample.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=05a4a8d4071035d9'. Value does not fall within the expected range.

You can work around this error by manually starting your orchestrations in the MMC, either one-at-a-time or grouped by host.  The error only occurs the first time you attempt to start - all future attempts will succeed.  This is because the underlying bug has to do with the creation of subscriptions for the orchestrations. 

This is actually a regression from BizTalk 2004 - where you could in fact change multiple orchestrations (across differing hosts) to a started state via the BtsOrchestration 'Explorer OM' class in a single transaction (i.e. before calling SaveChanges.)  In addition to the behavior just described with BizTalk 2006, you can't do this kind of Explorer OM work once multiple hosts are involved in BizTalk 2006.

But hey, there is a hotfix.  Contact PSS and ask about KB927052.

The second bug is encountered when you have a BizTalk 2006 application with multiple MSMQ receive locations.  You will find that restarting your host instances takes an excessively long period of time.  This is due to a bug in the adapter which was not allowing the host service to stop correctly (the Service Control Manager would eventually just terminate the service...)  Contact PSS and ask about KB928222.

Update: The tracking query as a TRQ file is available here, per a comment on this post.

I just got back from a week at the "Microsoft SOA & Business Process" conference in Redmond.  Lots of discussion on BizTalk 2006 R2, the new .NET Adapter Framework, Windows Communication Foundation (WCF), Windows Workflow (WF), Microsoft Office Sharepoint Server 2007 (MOSS).  Great stuff.

I did a session at the conference on Friday titled "Applying Maximum Sustainable Throughput to a Management/Operations Strategy".  (OK, the title may have scared a few people off...) The content was an extension of what I presented to the Twin Cities BizTalk User Group in September.  The overall theme was relating what you learn during your performance testing phase to your operations strategy, and ensuring that you have application-level metrics that represent operational boundaries you would like to "live within".  In particular, I discussed how to formulate your performance goals, how to measure against them as operational parameters, and how to proactively monitor with a tool such as Microsoft Operations Manager (MOM).  We went on to discuss how to choose the workloads (e.g. send, receive, or orchestration work) for new servers that you place in your BizTalk group, depending on what you see within various BizTalk work queues and throttling states.  I've put the slides for the session here for your perusal. 

One of the demos showed a HAT query I've worked up which will report on how many times each orchestration has run within a given time window, along with the percent of total transaction volume it represents, time duration, and activity rates (per second, per minute, and per hour). You can put the following query in a .TRQ file (like OrchestrationDurationsAndRates.trq) and the put it within C:\Program Files\Microsoft BizTalk Server 2006\Tracking\Queries (or the equivalent on your server.) It will then appear within the "Queries" menu in HAT. This can be a genuinely useful query to have at your disposal, both during performance testing and in production.  You can run this in Query Analyzer if you prefer - just uncomment the lines that define UtcOffsetMin.  Note that rate calculations are dividing across the entire time window, which assumes a fairly steady traffic pattern.

If you're trying to determine compliance within an SLA such as "95% of all requests must be satisfied within 15 seconds or less", you might add an item to the select clause below along the lines of "average(service duration) + (2*stddev(service duration))", if the distribution of your transaction durations tends to be normal (per your high school stats class.)

declare @beginTime as datetime
declare @endTime as datetime

declare @TotalCountInWindow as int

--
--  SET THE TIME WINDOW FOR YOUR QUERY HERE
--
select @beginTime =DateAdd(hour, -2, GetDate())
select @endTime = GetDate()

-- Only need to uncomment this select when inside of query analyzer,
-- where UtcOffsetMin won't be defined.
-- declare @UtcOffsetMin as int
-- select @UtcOffsetMin = 
-- -1*DateDiff("mi",CONVERT(varchar,GetDate(),0),CONVERT(varchar,GetUTCDate(),0))

select @beginTime = dateadd(minute,-1*@UtcOffsetMin,@beginTime)
select @endTime = dateadd(minute,-1*@UtcOffsetMin,@endTime)

SELECT @TotalCountInWindow = count(*)
FROM dbo.dtav_ServiceFacts sf WITH (READPAST)
where [ServiceInstance/StartTime] between
@beginTime and @endTime
and [ServiceInstance/State] = 'Completed'
and [Service/Type] = 'Orchestration'


SELECT 
[Service/Name],
count(*) as TotalCount,
(cast(count(*) as float)/cast(@TotalCountInWindow as float))*100 as PercentOfTotal,
avg([ServiceInstance/Duration]/1000.0) as AverageDuration,
min([ServiceInstance/Duration]/1000.0) as MinDuration,
max([ServiceInstance/Duration]/1000.0) as MaxDuration,
stdev([ServiceInstance/Duration]/1000.0) as StdDevDuration,
count(*)/DateDiff("hh",@beginTime,@endTime) as perHour,
count(*)/DateDiff("mi",@beginTime,@endTime) as perMinute,
cast(count(*) as float)/DateDiff("ss",@beginTime,@endTime) as perSecond

FROM dbo.dtav_ServiceFacts sf WITH (READPAST)
where [ServiceInstance/StartTime] between
@beginTime and @endTime
--and  [ServiceInstance/Duration] > 0
and [ServiceInstance/State] = 'Completed'
and [Service/Type] = 'Orchestration'
group by [Service/Name]

Got your own favorite HAT query? Post it in the comments - maybe we'll start a new section for them. Some shops deploy their custom HAT queries directly with their BizTalk applications...For Deployment Framework folks, that looks as follows (assuming a project subdirectory called DeployedHATQueries):

	
<target name="deployHatQueries">
  <copy todir="${btsDir}\Tracking\Queries" overwrite="true">
      <fileset basedir="DeployedHATQueries">
         <include name="**\*.trq" />
      </fileset>
  </copy>
</target>

Despite all the functional testing and stress testing you do prior to releasing your BizTalk app into production, unexpected behavior can (and often will) happen just the same.  Production usage just winds up introducing all sorts of permutations (including interactions with external systems) that are hard to predict earlier in the lifecycle.

The goal, of course, is to minimize the the operational "care and feeding" that an application requires over time.  Making this happen is mostly a function of using the application's "diagnostic surface area" (logs, counters, MOM packs, etc.) to feed back into each release cycle.  But we also need post-mortem tools when the host environment terminates unexpectedly or stops responding (whether that environment is BizTalk, IIS, COM+, Sql SSIS, etc.)

While a well-designed app will be able to successfully restart and resume processing (with full data integrity) at such a point (i.e. after the host has been terminated), there is still operational expense that has been injected.  We want to find and eliminate these problems...

Using the Visual Studio debugger is almost never an option in production, of course.  We need the ability to capture the current state as a "dump file" and do offline analysis. 

The "Windows Debugging Tools" are designed for this purpose (and you will often use these during a call with Microsoft's support staff, so it is good to be familiar with them.)  The debugging tools are a pretty large subject - so here, we are just going to cover the bare minimum required to capture a dump file for your running BizTalk process when it appears to be hung with a large number of "Active" service instances.

Step By Step:

• Install or xcopy the Windows Debugging Tools to the server where BizTalk is currently hung (or crashing unexpectedly.)  It can be helpful to install in an easy location for command line access like 'c:\debuggers'.
• From command line, run the following from the command line to get process IDs for all BizTalk hosts:
  typeperf "\BizTalk:Messaging(*)\ID Process" -sc 1
• Run 'adplus.vbs' in crash or hang mode, depending on whether the process ends unexpectedly (crash) or has become unresponsive (hang).  To generate a hang dump, your command line might look like:
  c:\debuggers\cscript adplus.vbs –hang –CTCF –p (pid from last step) –o c:\temp
• Copy the dump file to an offline location if need be.
• Set an envrionment variable called '_NT_SYMBOL_PATH' to 'srv*c:\symbols*http://msdl.microsoft.com/download/symbols'.  Alternatively, launch WinDbg.exe from the debuggers directory and use the File-'Symbol File Path' menu.  This will ensure that you are automatically downloading the correct symbols when you analyze the crash dump.  
• Start WinDbg.exe, and use File-'Open Crash Dump' to open your dump file.  Then, in the command window, use:
  '.load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll' to load managed code debugging extensions.
• In the command window, use !EEStack to get a full stack trace.  Use Edit-Find to search for your custom code method name or the name of your orchestration.  Look for patterns that indicate the cause of the hang ("hmmm, all my threads seem to be inside Thread.Sleep. That's funny.")  Use !help from the command window to begin learning about the rest of SOS (to assist with diagnosing managed memory leaks, etc.)

For more information on the Windows debugging tools, see here.

Tired of havng incredibly long xpath statements when using the "xpath" keyword in your orchestration expression shapes?  (They are generally long because of the xml namespace issues, unless your documents do not use namespaces.  When namespaces are in play, "local-name()" tends to overwhelm your actual path content.)

For BizTalk 2006, we can take advantage of the fact that System.Xml 2.0 added the ability for documents to supply namespace mappings themselves when doing xpath work - without the need for manually populating and using a XmlNamespaceManager instance, as was the case for .Net 1.x.  Note that XPathNavigator now (as of .net 2.0) has a SelectSingleNode method which can accept an implementation of the new IXmlNamespaceManager interface (which XPathNavigator itself implements.) This means that the 'xmlns:someprefix=someuri' declarations in your instance document can actually be used automatically, such that your xpaths can use the same prefixes when issuing a select. .Net 1.x had no such mechanism.

So, in whatever "utility" class you have sitting around to help you out when doing orchestration work, add the following static method.  (I generally put such things outside of a namespace - that way they are a bit more concise inside the orchestration expression editor.) In high-performance scenarios, may want to measure the performance of this approach vs. the built-in xpath keyword.  Recall that messages can be passed as XmlDocument instances, so your expression window might have: myString = XmlUtility.GetXPathStringResult(myMessage,"/spf:someparent/spf:somechild");

public static string GetXPathStringResult(
    XmlDocument document,
    string xpath)
{
    XPathNavigator nav = document.CreateNavigator();

    // (so namespace mappings are in scope when we select)
    nav.MoveToChild(XPathNodeType.Element); 

    XPathNavigator node = nav.SelectSingleNode(xpath, nav);

    return node.Value;
}    
    

Update: Andy pointed out that Tom Abraham has given this a thorough treatment here.

There appear to be cases where having BizTalk 2004 and the .NET 2.0 framework installed on the same machine may cause you some difficulty.  I've seen a few instances where the BizTalk service will not start...or where the BizTalk 2004 process is loading (gulp) the 2.0 runtime (so says Process Explorer.)  This latter behavior will result in the VS2003 debugger not attaching correctly (among other things...)  See this kb.

To fix this, you can modify the BTSNTSvc.exe.config file to include the following (after 'configSections'):          

<startup>
<supportedRuntime version="v1.1.4322"/>
</startup>

In my last post I indicated there was a better story in BizTalk 2006 for working with the C# code that is generated as an intermediate when compiling ODX (orchestration) files - that is, better than having to deal with temporary files and the BizTalk 2004 File Dump utility.  I

If you've worked with BizTalk 2006, you might have already stumbled across this.  The xlang compiler for BizTalk 2006 is kind enough to place the intermediate C# file for your ODX directly in the project directory.  If you have multiple orchestrations in an assembly, you will find a C# file for each orchestration - but only one of those files will actually have content (all the generated C# code is consolidated into one file.)

What this means is that you can do symblic debugging of orchestrations in a fashion far easier than what I described for BizTalk 2004.  The Edit/Debug cycle might look like this:

1. Build your orchestration project.
2. Open the associated (generated) C# file - perhaps make it a solution item for convenience.
3. Search for something in one of your expression shapes (say "MyClass.Execute"), and set a breakpoint.
4. From the Debug menu, choose Processes and attach to BTSNTSvc.exe. (Have more than one?) Choose CLR debugging only - not native. The symbols should be loaded automatically - no need to copy PDBs to the GAC for this case.
5. Trigger your orchestration however you normally would.
6. Find your problem, Debug-Detach All, and fix the problem in the orchestration.
7. Decide that you really want Gilles' MessageContext Debugger Visualizer

There is another benefit to having the generated C# files for orchestrations be more accessible...If you deploy your PDBs to the GAC, either using either my upcoming deployment work for BizTalk 2006 or just the GetGacPath2 utility that I made available here, you will find that stack traces that are recorded in the event log actually reference line numbers within the generated C# code.

As I said in an earlier post, if you are responsible for troubleshooting BizTalk applications in production, you are likely hungry for all the information you can get about why something is failing.  Since all BizTalk project assemblies are in the GAC, the stack traces you get (either from your own logging, or the event logs BizTalk generates for unhandled exceptions) do not contain file and line number information by default.  This makes post-mortem analysis a lot harder...

When a BizTalk 2006 application is deployed to production, you might consider archiving the generated C# code for production support.  If you use GetGacPath2 to put PDBs in the GAC, you will have file and line number information within captured stack traces to aid in troubleshooting. (Note that GetGacPath2 supports MSIL, 32 bit, and 64 bit assembly caches.  MSIL is the right choice for BizTalk 2006 projects.)

I've noticed that - despite a fair bit of discussion on the topic - many folks wind up using atomic scopes simply to avoid serialization requirements for .Net types within an orchestration.  It isn't surprising this is the past of least resistance - after all, the relevant compiler error sort of suggests this as a fix:

error X2141: a non-serializable object type 'SomeAssembly.SomeComponent yourComponent' can only be declared within an atomic scope or service

Recall that persistence will occur when messages are sent, when another orchestration is started asynchronously (like via the Start Orchestration shape), when an orchestration instance is suspended, when a host instance does a controlled shutdown, when a "wait" state occurs (like a blocking receive or delay shape that makes the schedule a candidate for dehydration), or when the orchestration completes. 

It will also occur at the end of a transactional scope to assist with resuming in an unambiguous state (and executing compensation logic.)

So...although the introduction of an atomic scope prevents serialization within the scope, it introduces serialization at the end of the scope.  This is often a persistence point (read: database hit) you wouldn't have had to live with -- if an atomic scope was introduced purely to get around the compiler error above.

Now...sometimes the .Net component you are using indeed requires state - and you should go ahead and implement a serialization strategy for that case.  If the component can be stateless, then you might consider making the methods you access "static" to avoid the compiler-enforced requirement altogether.

There is a third case, however - where you know that serialization is not required (given how your orchestration is structured) - but you don't own the code for it, either.  For instance, you might want to use an XmlNode to hold the result of a SelectSingleNode call on an XmlDocument.  If your use of the XmlNode instance is confined to a single expression shape...no state actually needs to be preserved.  So, in this case, consider wrapping the class as shown below.  Here, we honor the serialization requirement - but we don't, in fact, do any serialization work.  Now, no atomic scope is needed (avoiding the associated performance hit).  Use this technique with care - the state management facilities in BizTalk are usually a great asset.

      
/// <summary>
/// It can be helpful when dealing with xpath expressions within orchestrations to
/// make use of a System.Xml.XmlNode (or derivations) without having to deal 
/// with the fact that XmlNode is not serializable.  If usage is confined to a single
/// expression shape, we actually won't have any serialization requirements, and so they
/// have empty implementation here.
/// </summary>
[Serializable]
public class BizTalkXmlNode : ISerializable
{
	private XmlNode _xmlNode;

	public BizTalkXmlNode()
	{

	}

	public XmlNode XmlNode
	{
		get{return _xmlNode;}
		set{_xmlNode = value;}
	}

	public string InnerText
	{
		get{return _xmlNode.InnerText;}
		set{_xmlNode.InnerText = value;}
	}

	// This is helpful to use as a static helper method because the xpath expression in 
	// biztalk returns a XmlNode - so direct comparisons to string literals won't work.
	// (And the expression editor won't let you get the InnerText property...)
	public static string GetInnerText(XmlNode node)
	{
		return node.InnerText;
	}

	private BizTalkXmlNode(SerializationInfo info, StreamingContext context)
	{
		// no state intended...
	}

	public void GetObjectData(SerializationInfo info, StreamingContext context)
	{
		// no state intended...
	}

} 
    

This great post from Tomas on receiving ack/nack notifications from the MSMQ[C] adapter prompted me to write up something I had researched a few weeks back...

But first, a bit of background...

When doing work with message queueing systems, a very common (and well supported) convention exists when two systems are exchanging messages: System 'A' sends a message to System 'B'.  When System 'B' wishes to reply, it uses the message identifier of the 'request' message as the correlation identifier of the 'reply' message.  The reply queue used is either well known by both parties, or is communicated as a property on the request message.  For purposes of discussion, let's call this the "queueing contract". 

Both MSMQ and MQSeries have MessageId and CorrelationId properties on messages.  When looking at messages in a reply queue, System 'A' can peek at the messages, looking for a particular CorrelationId prior to issuing a receive.  Alternatively, it might use the CorrelationId as a look-up into working state of some sort.  To facilitate request-response interactions, MQSeries has had a ReceiveByCorrelationId API for a long while, whereas MSMQ added this in 3.0 (that is, on Windows XP and Windows 2003.) (Actual property and method names may vary in real life...)

There are, of course, many deployed systems in the wild that implement the queueing contract - using both MSMQ and MQSeries.  You may find yourself needing to integrate with these.  At first blush, it seems that consuming a service such as this should be a breeze from within BizTalk orchestrations - after all, accomplishing correlation is a first-class-citizen feature for BizTalk, so how tough can this be?

Correlation in BizTalk is indeed a first-class citizen...for data within your documents (or promoted message context.)  The classic example is to define a property schema that embodies a business identifier like "PONumber", and then visit all schemas that will be used across a set of message interactions - promoting the appropriate field in each as a "PONumber".  Then, a correlation type and correlation set are defined which reference that property, and Send and Receive shapes are set to either initialize or follow the correlation as appropriate.

The problem with an orchestration consuming the queuing contract we described earlier is that this contract occurs at the transport layer.  To be a consumer of the queueing contract, an orchestration would like to send a message, and initialize a correlation based on the MessageId...but the MessageId is produced further downstream, at the point an actual MSMQ or MQSeries message is generated by an adapter.  Moreover, an orchestration would subsequently like to receive a message, following an initialized correlation - but the CorrelationId isn't promoted by the MSMQ adapter (though it is for the MQSeries adapter...)

Solving for the MSMQ Adapter

The MSMQ Adapter gives us a small amount of help to get past this...We can in fact do our initial send operation through a solicit-response port (rather than a one-way port) and get back the MessageId that was generated by the adapter -- it actually comes back in a single-element document with a tag named "MSMQMsgId".  Starting here, the approach I took to solving the whole of the problem looked like this:

1. Send a message through a MSMQ solicit-response port.  The response operation in the orchestration designer can be of type "String".
2. On the response half of the solicit-response port, execute a pipeline & pipeline-component that extracts the returned MessageId and promotes it as the CorrelationId (using the standard MSMQ adapter namespace for that property.)  (For tracing purposes & useability, the MessageId is also returned in a <string> wrapper by the pipeline component rather than the native <MSMQMsgId> wrapper.)
3. When the initial message (containing the correlation id in both content and context!) is received back into the orchestration, it is sent back out to initialize a correlation set (typed as MSMQ.CorrelationId.)  This Send shape is just a "dummy send"...it exists just to initialize the correlation, because that is the only means given to us in the orchestration world.  I used Patrick Wellink's "NOPE" Adapter for this purpose, but there would be other options as well (including Tomas' Null Send Adapter, which I just recently rediscovered and which apparently performs better under load.)
4. Next, we receive the "real" response message - following the correlation that was just initialized.  Because MSMQ.CorrelationId is not promoted by default, the response is brought through a second pipeline component that performs that service for us.

See the diagram below (and the downloadable sample) for more detail. 

(click)

What can go wrong with this approach?  Well...there is race condition that can occur if the "real" response message (step 4 above) arrives before steps 2 and 3 can execute and establish the correlation (and associated instance subscription.)  In other words, if the "real" response message comes back before the orchestration has had a chance to express interest in that particular message, a routing failure will occur.  This might be highly unlikely for your scenario (given response times of the service you are interacting with) but it is something to test for, nonetheless.  (With BizTalk 2006's ability to route failed messages, you could potentially catch and retry in this scenario...)

(Note - if you are going to rely on the technique in the sample, consider asking Microsoft support for BizTalk 2004 QFE 1647, which addresses an issue with truncated correlation identifiers being returned to BizTalk...)

Solving for the MQSeries Adapter

Consuming the queueing contract with the MQSeries adapter for BizTalk can be done with a technique quite close to the one just discussed - but with a little less work.  The response to the initial solicit-response interaction returns with MQMD_MsgID already copied to a  MQSeries.BizTalk_CorrelationId property.  Likewise, the when receiving the "real" response message, MQMD_CorrelationId is copied to the MQSeries.BizTalk_CorrelationId property.  This means all correlation can be done with this one property - no pipelines/pipeline-components are needed!

From the documentation:

(click)

(The "dummy send" - though not pictured - would be required here as well to initialize the correlation set.)

In addition, there is a much cleaner solution that the BizTalk 2004 MQSeries adapter provides.  It turns out that MQSeries allows the "caller" to assign the message ID (unlike MSMQ), so within BizTalk, you can in fact set MQMD_MsgID and MQMD_CorrelationId to the same value prior to doing your initial send.  Now, you can initialize a correlation set based on MQMD_CorrelationId with the first outgoing message, and follow the correlation for the "real" response message.  Slick!

From the documentation:

(click)

Solving for the MQSeries Adapter in BizTalk 2006

Ahhhh, here we have what looks like real simplicity...There is a new feature in the BizTalk 2006 MQSeries adapter termed "Dynamic Receive" (referenced in the BizTalk 2006 Adapter Enhancements document.)  For a solicit-response port, Dynamic Receive allows context properties on the outbound message to determine which server, queue manager, and queue should be "watched" for the response message.  This is used in conjunction with a "match" option - which allows you to wait for a particular message based on CorrelationID (or any of several other properties, such as MessageID, GroupID, SequenceNumber, Offset, or MsgToken...)

Not only does this give us quite a bit of flexiblity for where response messages should be found (we don't need a fixed receive location anymore) but it also elminates the whole of the problem of implementing the "queueing" contract within orchestrations.  With this feature in place, an orchestration simply uses one solicit-response port, and the work is done.

Needless to say, it would be fantastic to get similar support from the MSMQ Adapter (at least the ability to have a single solicit-response port with a similar "match" criteria option, even without the dynamic receive location.)  But I don't think that is in the cards for BizTalk 2006...

Notes on the download:

• See the BTMSMQCorrelation.PortBindings.xml file for the names of the two queues and two file directories that must be created for this sample.
• Install Patrick Wellink's "NOPE" Adapter or choose a different strategy for dummy sends (like Tomas' Null Send Adapter.)
• Use the included Deployment Framework-based deployment, or deploy manually.
• To run, launch the "TestQueueServer" console app that will implement the queueing contract (request/response)
• The sample will import and run just fine in BizTalk 2006, though you will have to deploy manually.  (Still working on the 2006 version of the deployment framework - more later...)

Hopefully this is helpful to those doing queueing work with BizTalk - enjoy!

You can find my thoughts on a really simple message replay (message re-processing) technique for BizTalk in Stephen Thomas' The BizTalker newsletter.  Check it out here.  For additional clarification, I've created a quick Visio here.

I worked through a problem recently with a client that really took me by surprise - because I would think that many BizTalk shops would be running into this issue regularly.  So!  Here goes with an explanation and a solution. 

We ran into this problem initially using the MSMQ (not MSMQT) adapter with BizTalk 2004.  We had roughly 10 MSMQ receive locations, as well as a few send ports that were using the loopback adapter.  These were all executing in the same host.

The initial symptom was that the loopback adapter appeared to not work - messages were just not getting through!  They sat in the "delivered, not consumed state" for no good reason.  But we quickly reproduced the problem with just MSMQ receive locations (i.e. without the loopback adapter.)

On a single processor virtual machine, the repro looked like this:  Create four MSMQ receive locations, and one MSMQ send port (with the send port subscribed to one of the receive ports, just to keep things easy.)  No messages will flow through the send port at all. 

To repro: This download has a binding file with receive ports/locations for local (non-tx) private queues Q1-Q4, plus a send port for local private queue NONTXQ with a filter for the first receive port.  There is also a bit of VB script to put a message into a queue...If you turn off one of the receive locations for Q2-Q4, you'll find things work just fine.  If you don't, then (reiterating) no messages will flow through the send port.

What was the resolution?  Well, with BizTalk 2004 Service Pack 1 installed, you can create a "CLR Hosting" key under the registry service definition.

• Open the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
• Under BTSSvc{some GUID}, create a key called 'CLR Hosting'. (Note that there will be a BTSSvc3.0 entry present...but you must add the key under a BTSSvc{some GUID} key, where the GUID corresponds to the host you are dealing with, as shown by the DisplayName value.). Example: BTSSvc{DDEF238B-2D21-4B1E-8845-C6C67C6A86C0}.
• Under the "CLR Hosting" key (which you will create), create the following DWORD entries with the following values:
  • MaxIOThreads – 75 (actual # tbd)
  • MaxWorkerThreads – 75 (actual # tbd)
  • MinIOThreads – 55 (actual # tbd)
  • MinWorkerThreads – 55 (actual # tbd)
• Restart the BizTalk host service.

In our case, we actually had to increase these values - you should determine the values you need through testing.  Consider having min worker threads equal to 7x the number of MSMQ receive locations, and max worker threads equal to 10x the number of MSMQ receive locations.  (More on these numbers later...)

Does the documentation address this?  Good question.  If you look at the topic "Managing Multiple Receive Locations" in the MSMQ adapter documentation, you will find some reference to this.  It indicates you should create a "CLR Hosting" key as described above...but no actual values are mentioned (clearly just a documentation mishap.)

But why do these have to be tweaked at all?  Good question.  The documentation for the MSMQ adapter has some unfortunate quotables, like:

To increase performance, Microsoft BizTalk® 2004 Adapter for MSMQ is multi-threaded. If you have many receive locations, there may not be enough threads available for all the receive locations. This prevents some of the receive locations from picking up messages.

The reality is that you really shouldn't have to starve any particular receive location because of a lack of threads...you should just wind up with increased latency.  But, such is not the implementation of the MSMQ adapter (at least for BizTalk 2004.) 

Some background: The MSMQ adapter has a "Batch Size" parameter and a "Serial Processing" parameter that can be set per receive location.  "Batch Size" determines how many messages the adapter will attempt to read from the queue (and submit to the message box) on each iteration.  "Serial Processing" determines whether one thread is engaged in the peek/get/submit activity per receive location (Serial Processing = 'true') or multiple threads (Serial Processing = 'false').  If "Serial Processing" is true, the "Batch Size" is forced to one regardless of the actual setting.

So what is the execution flow for a given receive location?  The internal class MsmqReceiverEndpoint is instantiated per receive location, and when it initializes, it calls ThreadPool.QueueUserWorkItem with a reference to itself.   If "Serial Processing" is false...it does this exactly seven (7) times.  

What does it do with the QueueUserWorkItem callback?  Well, when MsmqReceiverEndpoint.ProcessWorkItem is called, it enters into a do/while loop that doesn't exit until the endpoint (receive location) becomes invalid (i.e. the receive location is shut town.)  In other words, ProcessWorkItem sits on a .NET thread pool thread - and if Serial Processing is false, it sits on seven of them.  The do/while loop executes a peek on the queue (with a hard-coded 10 second timeout), and if there are messages waiting, it receives up to "Batch Size" and submits them to the message box.  (It will give up attempting to receive a "Batch Size" worth of messages if the 10 second timeout is reached on any attempt within the batch receive loop - i.e. if you drop a single message on a queue, and the batch size is greater than one, expect to wait 10 seconds before further activity begins...)  The behavior of consuming seven threads per queue leads to the recommendation of MinWorkerThreads = 7x MSMQ receive locations provided above.

Now, I confess - I'm not a BizTalk adapter expert.  But, this design seems to be in conflict with the advice offered in "Writing Effective BizTalk Server Adapters", where it says:

Don't starve the .NET thread pool: ...While starving the .NET thread pool is a risk to all asynchronous programming in .NET, it is particularly important for the BizTalk Server adapter programmer to watch out for this.  It has impacted many BizTalk Server adapters: take great care not to starve the .NET thread pool.  The .NET thread pool is a limited but widely shared resource.  It is very easy to write code that uses one of its threads and holds onto it for ages and in so doing blocks other work items from ever being executed....If you have multiple pieces of work to do (for example copying messages out of MQSeries into BizTalk Server), you should execute one work item (one batch of messages into BizTalk Server) and simply requeue in the thread pool if there is more work to do. What ever you do, don't sit in a while loop on the thread.  

Is this fixed in BizTalk 2006?  Surely it is...  And, in fact, it sure seems to be in Beta 1.  The design of the adapter is a bit different...First, "Serial Processing" refers to whether additional messages will be received from the queue prior to the "EndBatchComplete" event being set (downstream of IBTDTCCommitConfirm.Done.)  (This part of "Serial Processing" is true for BizTalk 2004 as well, along with forcing the batch size to one.)  "Serial Processing" in BizTalk 2006 does not affect how many threads will be reading from your queue - you will have just one (despite what the Beta 1 docs say...), unless you have multiple host instances in play.  (That one thread using a large batch size and operating with serial processing set to 'false' - not blocking on the actual message box submission - should keep up with a fairly large message arrival rate, but multiple host instances might be needed for your particular case.)

More importantly, the ProcessWorkItem implementation returns immediately after a single peek/get/submit operation (and simply calls QueueUserWorkItem again, per the advice cited above.)  (Side note: There seems to be some room in the design for the idea that you in fact woudn't return immediately if more than a threshold number of messages were received, but currently this condition is "if # of messages received > BatchSize", which won't ever happen.) 

So what should I do for now with BizTalk 2004?  For those using the MSMQ Adapter with BizTalk 2004...consider whether you can set "Serial Processing" equal to true.  Keep in mind this forces you to a batch size of 1, so this might not work depending on your message arrival rate.  If you test this configuration and find an unacceptable performance loss, consider setting the MinWorkerThreads value to 7x the number of MSMQ receive locations you are maintaining, and MaxWorkerThreads to roughly 10x (to provide breathing room.)  As an alternative, spread your receive locations among multiple hosts (though avoid an over-proliferation of hosts - that has its own issues.) 

And never draw any conclusions until you have performance tested at load with your final host configuration - that is, your final allocation of send handlers, receive handlers, send ports, and receive locations among your hosts!  Other adapters may affect the outcome if they involve polling on the receive side, or polling on the "response" side of a solit-response send port.  (If they use a thread pool thread to do their work, they can be affected by any adapter that consumes threads whether they themselves are written correctly or not!)  Finally, I've heard from a gentlemen who has done extensive testing that the threading parameters above are useful/necessary when using large numbers of MSMQT receive locations as well.

Never a dull day in BizTalk land...!