Despite all the functional testing and stress testing you do prior to releasing your BizTalk app into production, unexpected behavior can (and often will) happen just the same.  Production usage just winds up introducing all sorts of permutations (including interactions with external systems) that are hard to predict earlier in the lifecycle.

The goal, of course, is to minimize the the operational "care and feeding" that an application requires over time.  Making this happen is mostly a function of using the application's "diagnostic surface area" (logs, counters, MOM packs, etc.) to feed back into each release cycle.  But we also need post-mortem tools when the host environment terminates unexpectedly or stops responding (whether that environment is BizTalk, IIS, COM+, Sql SSIS, etc.)

While a well-designed app will be able to successfully restart and resume processing (with full data integrity) at such a point (i.e. after the host has been terminated), there is still operational expense that has been injected.  We want to find and eliminate these problems...

Using the Visual Studio debugger is almost never an option in production, of course.  We need the ability to capture the current state as a "dump file" and do offline analysis. 

The "Windows Debugging Tools" are designed for this purpose (and you will often use these during a call with Microsoft's support staff, so it is good to be familiar with them.)  The debugging tools are a pretty large subject - so here, we are just going to cover the bare minimum required to capture a dump file for your running BizTalk process when it appears to be hung with a large number of "Active" service instances.

Step By Step:

• Install or xcopy the Windows Debugging Tools to the server where BizTalk is currently hung (or crashing unexpectedly.)  It can be helpful to install in an easy location for command line access like 'c:\debuggers'.
• From command line, run the following from the command line to get process IDs for all BizTalk hosts:
  typeperf "\BizTalk:Messaging(*)\ID Process" -sc 1
• Run 'adplus.vbs' in crash or hang mode, depending on whether the process ends unexpectedly (crash) or has become unresponsive (hang).  To generate a hang dump, your command line might look like:
  c:\debuggers\cscript adplus.vbs –hang –CTCF –p (pid from last step) –o c:\temp
• Copy the dump file to an offline location if need be.
• Set an envrionment variable called '_NT_SYMBOL_PATH' to 'srv*c:\symbols*http://msdl.microsoft.com/download/symbols'.  Alternatively, launch WinDbg.exe from the debuggers directory and use the File-'Symbol File Path' menu.  This will ensure that you are automatically downloading the correct symbols when you analyze the crash dump.  
• Start WinDbg.exe, and use File-'Open Crash Dump' to open your dump file.  Then, in the command window, use:
  '.load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll' to load managed code debugging extensions.
• In the command window, use !EEStack to get a full stack trace.  Use Edit-Find to search for your custom code method name or the name of your orchestration.  Look for patterns that indicate the cause of the hang ("hmmm, all my threads seem to be inside Thread.Sleep. That's funny.")  Use !help from the command window to begin learning about the rest of SOS (to assist with diagnosing managed memory leaks, etc.)

For more information on the Windows debugging tools, see here.

Tired of havng incredibly long xpath statements when using the "xpath" keyword in your orchestration expression shapes?  (They are generally long because of the xml namespace issues, unless your documents do not use namespaces.  When namespaces are in play, "local-name()" tends to overwhelm your actual path content.)

For BizTalk 2006, we can take advantage of the fact that System.Xml 2.0 added the ability for documents to supply namespace mappings themselves when doing xpath work - without the need for manually populating and using a XmlNamespaceManager instance, as was the case for .Net 1.x.  Note that XPathNavigator now (as of .net 2.0) has a SelectSingleNode method which can accept an implementation of the new IXmlNamespaceManager interface (which XPathNavigator itself implements.) This means that the 'xmlns:someprefix=someuri' declarations in your instance document can actually be used automatically, such that your xpaths can use the same prefixes when issuing a select. .Net 1.x had no such mechanism.

So, in whatever "utility" class you have sitting around to help you out when doing orchestration work, add the following static method.  (I generally put such things outside of a namespace - that way they are a bit more concise inside the orchestration expression editor.) In high-performance scenarios, may want to measure the performance of this approach vs. the built-in xpath keyword.  Recall that messages can be passed as XmlDocument instances, so your expression window might have: myString = XmlUtility.GetXPathStringResult(myMessage,"/spf:someparent/spf:somechild");

public static string GetXPathStringResult(
    XmlDocument document,
    string xpath)
{
    XPathNavigator nav = document.CreateNavigator();

    // (so namespace mappings are in scope when we select)
    nav.MoveToChild(XPathNodeType.Element); 

    XPathNavigator node = nav.SelectSingleNode(xpath, nav);

    return node.Value;
}    
    

(Update: The original download files were missing the PDB-to-GAC functionality I've discussed before.  Please download again if you have already...)

The BizTalk Deployment Framework has been updated to work with BizTalk 2006...It is hard to believe that this project has been going on since May of 2004!

The Deployment Framework for BizTalk 2006 still has the same goals as the 2004 version:

• Streamline the deployment process for developers, ensuring repeatability
• Make it easier for a team of BizTalk developers to stay in sync - not just with BizTalk artifacts, but with all the other infrastructure that surrounds the solution (virtual directories, queues, file folders, etc.)
• Extremely close parity between server-side deployments and developer deployments - so you are always testing your server deployments as you go through a typical developer work day.

BizTalk 2006 itself introduced quite a few features to make deployment easier, and can work fine for small (or solo) projects.  Here are a few limitations I've encountered:

• Though much improved, it is still possible to get into "dependency chaos" - where you spend time manually undeploying/deploying individual artifacts.
• Binding changes have to be communicated "manually" between developers on a team, since the import process is done through the Admin MMC or command-line.  Just "getting latest" from version control isn't sufficient.
• Binding files for your various environments have to be maintained as separate files (manually), rather than "merging in" environment-specific settings
• Artifacts such as rules, virtual directories, queues, folders, additional dependent assemblies, etc. can be represented as "Resources" within the Admin MMC (and in exported MSIs), but not in a fashion that easily moves between developers on a team.

So!  To get started with this version, download the Deployment Framework (Tools) zip and run the MakeBizTalkExternalTools_VS2005.vbs script.  This will add entries to the Visual Studio tools menu for deploying and undeploying using the framework.  You can download the full sample to see the framework in action (first build it, then do Tools-BizTalk Deploy.  You'll see something like this.)

The high-level approach is the same as the 2004 framework - you supply a small project-specific NAnt script that indicates via properties what elements of a deployment your solution requires.  You include BizTalkDeploymentInclude.nant to get all the core deployment functionality, and make sure the DeployTools directory is copied to your project.  (Unzipping the Deployment Framework Core into your project is a good way to do this.)  See the documentation for a more complete discussion.

The primary difference in the upgraded framework for BizTalk 2006 is that we now create a BizTalk Application definition, and use BTSTask to add all BizTalk artifacts as resources within that application.  Starting and stopping the application is done at an application level rather than per port/per orchestration.

Packaging up your solution as an MSI can be done with the WiX-based scripts that have been in the framework for awhile, or by using the MSI export mechanism in BizTalk 2006, if you prefer.  (The latter solution will require a few additional steps, and doesn't give you the parity described earlier.  But it works if you need to go that route.)

Other Notes:

• NAnt .85 RC4 (and NAntContrib ) is required.  Be sure to copy the new BizTalk.NAnt.Tasks.dll to NAnt's bin directory.  We need to call BizTalk's .net 2.0 assemblies, and NAnt wasn't built against 2.0 -- so change nant.exe.config to have only<supportedRuntime version="v2.0.50727" /> in the<startup> element.
• Log4net usage in the sample (which isn't required of course for the framework) has been updated to log4net 1.2.10 , as has my serializable wrapper.  You can find the new log4net.Ext.Serializable in the Tools download (which is useful all by itself, apart from the framework.)
• You can actually use the BizTalkDeploymentInclude.nant file in this release with BizTalk 2004, if you like, to aid in your migration.  There is NAnt property to indicate 2004 vs. 2006.
• Scan all the previous release notes...

Leave comments with any questions/issues/etc.  Enjoy!

Download: Full Sample, Framework Core, Tools Source , Docs

Update: Andy pointed out that Tom Abraham has given this a thorough treatment here.

There appear to be cases where having BizTalk 2004 and the .NET 2.0 framework installed on the same machine may cause you some difficulty.  I've seen a few instances where the BizTalk service will not start...or where the BizTalk 2004 process is loading (gulp) the 2.0 runtime (so says Process Explorer.)  This latter behavior will result in the VS2003 debugger not attaching correctly (among other things...)  See this kb.

To fix this, you can modify the BTSNTSvc.exe.config file to include the following (after 'configSections'):          

In my last post I indicated there was a better story in BizTalk 2006 for working with the C# code that is generated as an intermediate when compiling ODX (orchestration) files - that is, better than having to deal with temporary files and the BizTalk 2004 File Dump utility.  I

If you've worked with BizTalk 2006, you might have already stumbled across this.  The xlang compiler for BizTalk 2006 is kind enough to place the intermediate C# file for your ODX directly in the project directory.  If you have multiple orchestrations in an assembly, you will find a C# file for each orchestration - but only one of those files will actually have content (all the generated C# code is consolidated into one file.)

What this means is that you can do symblic debugging of orchestrations in a fashion far easier than what I described for BizTalk 2004.  The Edit/Debug cycle might look like this:

1. Build your orchestration project.
2. Open the associated (generated) C# file - perhaps make it a solution item for convenience.
3. Search for something in one of your expression shapes (say "MyClass.Execute"), and set a breakpoint.
4. From the Debug menu, choose Processes and attach to BTSNTSvc.exe. (Have more than one?) Choose CLR debugging only - not native. The symbols should be loaded automatically - no need to copy PDBs to the GAC for this case.
5. Trigger your orchestration however you normally would.
6. Find your problem, Debug-Detach All, and fix the problem in the orchestration.
7. Decide that you really want Gilles' MessageContext Debugger Visualizer

There is another benefit to having the generated C# files for orchestrations be more accessible...If you deploy your PDBs to the GAC, either using either my upcoming deployment work for BizTalk 2006 or just the GetGacPath2 utility that I made available here, you will find that stack traces that are recorded in the event log actually reference line numbers within the generated C# code.

As I said in an earlier post, if you are responsible for troubleshooting BizTalk applications in production, you are likely hungry for all the information you can get about why something is failing.  Since all BizTalk project assemblies are in the GAC, the stack traces you get (either from your own logging, or the event logs BizTalk generates for unhandled exceptions) do not contain file and line number information by default.  This makes post-mortem analysis a lot harder...

When a BizTalk 2006 application is deployed to production, you might consider archiving the generated C# code for production support.  If you use GetGacPath2 to put PDBs in the GAC, you will have file and line number information within captured stack traces to aid in troubleshooting. (Note that GetGacPath2 supports MSIL, 32 bit, and 64 bit assembly caches.  MSIL is the right choice for BizTalk 2006 projects.)

In the last release of the BizTalk Deployment Framework, an additional feature was added that I thought deserved its own post for explanation...

If you are responsible for troubleshooting BizTalk applications in production, you are likely hungry for all the information you can get about why something is failing.  Since all BizTalk project assemblies are in the GAC, the stack traces you get (either from your own logging, or the event logs BizTalk generates for unhandled exceptions) do not contain file and line number information.  This makes post-mortem analysis a lot harder...

A new custom NAnt task (getgacpath) and corresponding support within BizTalkDeploymentInclude.nant was added for placing PDB (program database) files for all BizTalk and component assemblies into the GAC.  (You can use just this NAnt task without the rest of the Deployment Framework, of course.  In addition, a standalone command line utility called GetGacPath has been included in the "Tools" download.)

Why is this interesting?  Well, because it allows you to get stack traces that include file and line number information even when assemblies reside in the GAC.  (This is obviously useful and useable outside the domain of BizTalk...)  Without extra work, file and line number information is generally missing from a stack trace if a PDB file is not co-located physically with the corresponding assembly.  Putting the PDB file directly into the GAC is an easy way of solving this problem.

Set the "deployPDBsToGac" NAnt property to "true" in your project-specific NAnt file to get this functionality.

A couple of points are worth mentioning:

• If the directory structure of the GAC were to ever change, this would need to be updated.  (See my upcoming post on using this technique with .NET 2.0.)
• It would be great if Gacutil.exe provided this functionaity, but it doesn't. 
• You should strongly consider having your release-mode binaries (aka 'deployment' configuration in BizTalk) build PDB files - there is no reason not to.  Look under Project properties/Configuration properties/Build/Generate Debugging Information.
• Stack traces that show up in the event log either through your own logging, or via BizTalk (because the exception was unhandled) will now have file and line number info.  For orchestrations, the file name will correspond to the temporary C# file generated at compile time.  If you use the techniques described in this post , you can correlate back to the actual statement that failed.  For instance, the event log might say:

  [4740] ERROR BizTalkSample.Orchestrations.TopLevelOrch - An exception was caught. 
  [a08701ff-bf80-4940-9f9f-c2bb8597684b]
  System.Exception: Something exceptional happened.
     at BizTalkSample.Orchestrations.TopLevelOrch.segment2(StopConditions stopOn) in 
     c:\Documents and Settings\Scott\Local Settings\Temp\h0li3wbi.0.cs:line 2538
  

• If you grab the file mentioned (h0li3wbi.0.cs) using BTSFileDump (again,see here) you can cross-reference this message to a location in the generated C# code for TopLevelOrch:

  

  You might save off "h0li3wbi.0.cs" (the temp file name chosen when TopLevelOrch compiled) and the other generated files when you do your build, using BTSFileDump. (That would be tough to do if you build with CruiseControl, etc.) But you wouldn't have to do that - if you can retrieve the source code used for a particular deployment, then the line number alone should be sufficient to get you to the exception site (since you can find the correct temp file by looking for comments, etc. that are in the orchestration - just like if you were using this technique to do live debugging.)  This all gets better with BizTalk 2006 - see a future post on this topic. 
• Final point: If you set deployPDBsToGac to "true", deployments will begin by stopping the BizTalk services (instead of just bouncing the services at the end.)  The reason for this is that the BTSNTSvc.exe process will not "let go" of an assembly for which it has loaded a PDB file (so you have to terminate it.)  This means you may not always want this switch on for the developer edit/run/debug cycle.

There are some updates to the Deployment Framework and log4net story that I've been meaning to release for quite some time. After this, I'll focus my deployment efforts on BizTalk 2006 (didn't I say that last release?)

Below are a list of revisions. On this blog's home page under "Downloads", you will find an updated full download (with sample), as well as updated "Core" and "Tools" downloads.  "Core" contains only the deployment framework itself and can be used for upgrades of existing projects.  "Tools" contains source for all utilities used by the framework.  Be sure to put a fresh copy of BizTalk.NAnt.Tasks.dll in your nant\bin directory. 

• Update to the log4net.Ext.Serializable library.  Note that the "Tools" download now has variants of this library for both log4net 1.2.0 and 1.2.9, though the sample is built against 1.2.9.  The important change here is that the original version of the library had a threading bug...as a result, the usage pattern for the library has changed somewhat.  Now, an instance of log4net.helpers.PropertiesCollectionEx is declared (as an orchestration variable), and an expression shape at the top of your orchestration will do something like below (notice logProps is passed to all logging calls so as to preserve context despite thread changes across dehydrations.)  See the sample for additional detail.

  logger = log4net.Ext.Serializable.SLogManager.
     GetLogger(@"BizTalkSample",log4net.helpers.CallersTypeName.Name);
  ...
  logProps.Set("InstanceId",TopLevelOrch(Microsoft.XLANGs.BaseTypes.InstanceId));
  logger.Debug(logProps,"Received top level request...");               
                 

• Update to the SSOSettingsFileReader class to include an "Update" method.  Likewise, an additional custom NAnt task called "updatessoconfigitem" that allows you to add or update config items within an SSO affiliate application.  What is the use case for this?  Suppose you have a piece of information that you a) don't want to store in SettingsFileGenerator.xls (perhaps because you can't manage that file securely) and b) you need access to at run time.  (For instance, it might be an Oracle username/password.)  You could ask an operator/admin for this information at deployment time using SetEnvUI.exe/InstallWizard.xml (from the deployment framework) -- and then in your project-specific NAnt file do this:

     <target name="customSSO" if="${property::exists('serverDeploy') and serverDeploy}">
        <updatessoconfigitem ssoitemname="databaseUserName" ssoitemvalue="${sys.env.databaseUserName}"/>
        <updatessoconfigitem ssoitemname="databasePassword" ssoitemvalue="${sys.env.databasePassword}"/>
     </target>            
        

  At runtime, you can simply use SSOSettingsFileReader to retrieve this information.  Note that this example checks for whether we are doing a server deployment - so in this case, we would have the development environment values in SettingsFileGenerator.xls. 

• Added a small utility called SSONameValueDump to the tools download to view current name/value pairs of an SSO affiliate application (if you have appropriate permissions.)
• Update to the <controlbiztalkports> custom NAnt task to allow Send Port Groups to be shared across projects.  That is, Send Port Groups will only be removed if they no longer contain Send Ports (and putting new Send Ports into an already-existing Send Port Group will happen automatically.)
• XmlPreprocess will be run against log4net configuration files, if you are using log4net. This allows you to use SettingsFileGenerator.xls to control logging levels for different physical environments.
• SetEnvUI.exe no longer defaults to "last directory" when doing a file browse, to avoid confusion between applications when managing multiple BizTalk application installs.
• If you use ElementTunnel, you will find that the xpath file will be reversed automatically for unescape operations (thanks to Frank de Groot for that update!)
• Fix: When deploying HTTP/SOAP infrastructure, file system permissions granted to the specified application pool account.
• Fix: The "quick update" target (updateOrchestrations) now respects true/false properties for includeSchemas, includeTransforms, etc. (A menu option in Visual Studio is created for this target when you use the MakeBizTalkExternalTools script. The target updates orchestrations, components, schemas, and transforms (see Flanders' post on this) Using this as part of your normal edit/run/debug cycle is a must for productivity…)
• Fix: Handle multiple assemblies (of one artifact type) correctly in more cases, including where we are not deploying to the management database (i.e. gac deployment only.)

Download: Full Sample, Framework Core, Tools Source

Late notice, I know, but if you can...be sure to join us at 6:00 pm at the local Microsoft office in Bloomington!

Jim Gaudette will be discussing reusability in BizTalk, and hey, food and beverages provided.

I've noticed that - despite a fair bit of discussion on the topic - many folks wind up using atomic scopes simply to avoid serialization requirements for .Net types within an orchestration.  It isn't surprising this is the past of least resistance - after all, the relevant compiler error sort of suggests this as a fix:

Recall that persistence will occur when messages are sent, when another orchestration is started asynchronously (like via the Start Orchestration shape), when an orchestration instance is suspended, when a host instance does a controlled shutdown, when a "wait" state occurs (like a blocking receive or delay shape that makes the schedule a candidate for dehydration), or when the orchestration completes. 

It will also occur at the end of a transactional scope to assist with resuming in an unambiguous state (and executing compensation logic.)

So...although the introduction of an atomic scope prevents serialization within the scope, it introduces serialization at the end of the scope.  This is often a persistence point (read: database hit) you wouldn't have had to live with -- if an atomic scope was introduced purely to get around the compiler error above.

Now...sometimes the .Net component you are using indeed requires state - and you should go ahead and implement a serialization strategy for that case.  If the component can be stateless, then you might consider making the methods you access "static" to avoid the compiler-enforced requirement altogether.

There is a third case, however - where you know that serialization is not required (given how your orchestration is structured) - but you don't own the code for it, either.  For instance, you might want to use an XmlNode to hold the result of a SelectSingleNode call on an XmlDocument.  If your use of the XmlNode instance is confined to a single expression shape...no state actually needs to be preserved.  So, in this case, consider wrapping the class as shown below.  Here, we honor the serialization requirement - but we don't, in fact, do any serialization work.  Now, no atomic scope is needed (avoiding the associated performance hit).  Use this technique with care - the state management facilities in BizTalk are usually a great asset.

      
/// <summary>
/// It can be helpful when dealing with xpath expressions within orchestrations to
/// make use of a System.Xml.XmlNode (or derivations) without having to deal 
/// with the fact that XmlNode is not serializable.  If usage is confined to a single
/// expression shape, we actually won't have any serialization requirements, and so they
/// have empty implementation here.
/// </summary>
[Serializable]
public class BizTalkXmlNode : ISerializable
{
	private XmlNode _xmlNode;

	public BizTalkXmlNode()
	{

	}

	public XmlNode XmlNode
	{
		get{return _xmlNode;}
		set{_xmlNode = value;}
	}

	public string InnerText
	{
		get{return _xmlNode.InnerText;}
		set{_xmlNode.InnerText = value;}
	}

	// This is helpful to use as a static helper method because the xpath expression in 
	// biztalk returns a XmlNode - so direct comparisons to string literals won't work.
	// (And the expression editor won't let you get the InnerText property...)
	public static string GetInnerText(XmlNode node)
	{
		return node.InnerText;
	}

	private BizTalkXmlNode(SerializationInfo info, StreamingContext context)
	{
		// no state intended...
	}

	public void GetObjectData(SerializationInfo info, StreamingContext context)
	{
		// no state intended...
	}

}