In my last entry, I discussed some ways that can making working with binding files a bit easier.  Here is another post in that same vein that addresses a common pain point...

Un-escaping TransportTypeData

One of the annoying things about binding files is that adapters only have a string element available to store adapter-specific information for send ports and receive locations.  As a result, adapters will store escaped XML (or even "doubly escaped" xml...)  This can be extremely hard to manage, especially for adapters such as MQSeries that keep quite a bit of information in this form. 

To solve this problem, I introduced a new command-line tool in the most recent version of the Deployment Framework called "ElementTunnel.exe" (the source for which is in the Tools download.)  This utility will take in an xml file, along with a file containing xpaths to elements that should be "encoded" or "decoded".  The end result is that you can choose to manage a "master" binding file (not directly useable) and run ElementTunnel on it immediately prior to deployment.  (You may also run XmlPreProcess on the same file for macro expansion! The sample in the deployment framework shows both occurring - XmlPreProcess should occur first!) 

So what does this mean?  An example for a single Send Port snippet: It means that, in the case of MQSeries, instead of storing and maintaining this mess:

With BizTalk 2004, it can be quite helpful to eventually maintain binding files as "source code".  After a solution has reached a certain point of stability (where port definitions are not changing often), many projects will use the Deployment Wizard to do one last export of the binding information -- and then maintain it by hand for any future changes (storing it in version control along with the rest of the solution.)

There are some interesting benefits that come along with this.  One such benefit is the ability to use the XmlPreProcess tool to merge environment-specific elements into the binding file (like URIs, retry counts, etc.), using the SettingsFileGenerator.xls spreadsheet to assist -- as has been discussed on this blog before.  Even if you are not using the Deployment Framework (which uses XmlPreProcess extensively), you should consider using XmlPreProcess as a standalone tool.  The ability to easily maintain a matrix of logical names (for physical endpoints, etc.) versus "environment names" (development, QA, production, etc.) is a huge help.  See the example spreadsheet below.  (The Deployment Framework also shows how to extend use of this same spreadsheet to manage run-time configuration settings that are stored in the BizTalk SSO.)

(click)

On to a bit more advanced topic: If you have a set of port definitions that you want to conditionally deploy into a given environment, you can define a true/false value within the spreadsheet and use simple "ifdef" logic in your binding file around the port definition.  For instance, you might want a particular File Send Port or Receive Location to only be active in your development and test environments.  To do this, define a name such as "LogInboundPODocsToFile", and set the default value to "true" - and set it to "false" in the "production" column.  Mark up your binding file accordingly.  See the example spreadsheet and binding file snippet below.  (When XmlPreProcess is run on this binding file, the port definition will only be included for environments where the LogInboundPODocsToFile value is true.)

(click)

<!-- ifdef ${LogInboundPODocsToFile} -->
<SendPort Name="LogSalesOrderResponse_FILE" IsStatic="true" IsTwoWay="false">
<TransmitPipeline Name="SendWithDefaultNamespaceFormat"         FullyQualifiedName="SendWithDefaultNamespaceFormat, XYZCo.BizTalk.Pipelines, Version=1.0.0.0, Culture=neutral,     PublicKeyToken=343bd7a15fff8d6e"
Type="2" />
<PrimaryTransport>
<Address>C:\Dev\FileLog\%MessageID%.xml</Address>
<TransportType Name="FILE" Capabilities="11" ConfigurationClsid="5e49e3a6-b4fc-4077-b44c-22f34a242fdb" />
...
</SendPort>
<!-- endif -->

Why would you want to conditionally deploy ports?  Like many, I have found it useful to have an additional file-based Receive Location (associated with a Receive Port bound to an orchestration) to kick off orchestrations during development - even if the actual transport used in production will be something different.  In addition, binding an orchestration to a Send Port Group allows you to have an additional file-based Send Port that will create an easy log of outbound traffic.  Finally, you might create a file-based Send Port that acts as an "additional" subscriber (by Receive Port Name) to your inbound messages for an easy log of inbound traffic.  (And, combined with a file-based receive port, these two mechanisms give you an easy re-processing mechanism - just drag/drop in Explorer.)  But you might want all of this machinery shut off in production, hence the technique we just discussed.

Macro Recursion

Another feature within XmlPreProcess is the ability to use "macro" recursion with XmlPreProcess.  This means you can define a macro (logical name) such as QueueServer (with a different value for development, QA, and production, etc.) and then define additional values in the spreadsheet that build on this such as: POAckQueue = {$QueueServer}\private$\POAckQueue.  This indirection can make maintaining large numbers of endpoint URIs even easier...See the example below - where POAckQueue can now appear in the "default" column (applicable to all environments.)

(click)

Multiple Environments

Note that the SettingsFileGenerator.xls spreadsheet provided with the Deployment Framework sample (and with XmlPreProcess) has room for four environments (development, QA, staging, and production.)  However, you can simply add columns to manage additional environments if need be.  One such use of this would be to create a column for "unit testing", where the URIs and other binding file substitutions point to resources under the control of your unit testing framework.

More to say on binding file management in another post...

Version 2.6 of the Deployment Framework is now available (see the download links - the full download, core, and tools have all been updated.)  Future work in this area will be about augmenting the already-great deployment story in BizTalk 2006! 

The release notes below will (likely) only be interesting for folks that are using the framework extensively.

Interesting Features:

• If you populate the bizTalkHosts property with the name of the Hosts that you are using, the 'servicecontroller' nant task is used for the stop/start of BizTalk services, with a configurable timeout stored in the 'biztalkHostResetTime' nant property. This requires recopying the BizTalk.nant.tasks.dll to your \program files\nant\bin directory. This alleviates the problem some users were having with timeouts in service restarts.
• IISReset now uses configurable reset time from nant property iisResetTime.
• Modified the sample application to demonstrate how you can maintain adapter configuration in binding files (i.e. the TransportTypeData sections) in its "unescaped" form -- rather than having to maintain it as escaped xml-within-xml. This relies on a new tool - ElementTunnel.exe, which is also in the Tools download. This is a huge help with adapters such as MQSeries that have many configuration elements.  More on this in another post.

Administrivia:

• SSOSettingsFileImport.exe opens files with read access only, eliminating some file locking issues.
• Permissions on the physical directories associated with virtual directories are now granted to the account specified as the application pool account (instead of just aspnet account.)
• Now using NAnt .85 RC3
• Now using log4net 1.2.9 (thanks Campbell McNeill for work on this!)
• Now using the July 2005 release of WiX (thanks to Loren Halvorsen for work on compatibility with this.)
• Now using XmlPreProcess 1.0 RC3 (and the associated SettingsFileGenerator.xls.) One new feature here is the ability to use URLs that contain query strings in the spreadsheet. The name/value pair XML files that are generated will wrap as CDATA where appropriate. Thanks Loren!
• Modified the sample application to demonstrate how file receive and drop locations can be made relative to the a) project directory on a developer workstation or b) installation directory on a server. This has actually be in the sample application before - but it relied on the xmlpoke nant task, whereas now it uses the XmlPreProcess mechanism we use for all other binding file modifications.
• Modified the deployment verification unit test in the sample application to rely on a NUnitUtility helper class that you can pull into your own projects. One included method allows you to examine the event log (on multiple machines) looking for errors that BizTalk might have raised during the test - very handy, and a potentially important component of your test.
• Support for a local_settings file used with XmlPreProcess. It can be handy to have an "environment specific" set of macro values that are particular to a local workstation, and the sample application shows how this is done now.
• Added transforms and schemas to the list of what is re-gac'd for the updateOrchestrations target. See here for more detail. In a recent release, I made changes that allowed for the use of a single binding file, so that binding files as emitted by the deployment wizard could be used (rather than splitting them by hand, which was required early on with the Deployment Framework.) This was a good change…but the implementation broke the "messaging only" scenario (where you have no orchestrations.) I don't have a good fix for this yet. The workaround is as follows: In your project's nant file, set includeOrchestrations to "true" (even though you don't have any), and define the orchestration assembly list to be an empty string, as shown here:
  <property name="includeOrchestrations" value="true" />
<property name="orchestrations" value="" />

How do you upgrade?  Extract the "core" zip file on "top" of your existing project, and then fix up the WiX directory manually (since it will be named BizTalkSample.WiXSetup after the unzip.)  All tools and scripts will be updated as a result.  Then test...Leave a comment with any issues you encounter.  Good luck!

Will this be the last place on the web that trumpets the BizTalk 2006 beta?  Likely not, but I did see it in quite a few places today.

Head to http://beta.microsoft.com with your passport and 'BizTalkBetaTeam' for a guest ID, and then wait patiently.  (While you're waiting, consider building out a VPC image with VS2005 beta 2 and, presumably, the latest SQL 2005 bits.  SQL 2000 would be fine as well, as BizTalk 2006 will not require SQL 2005.)

This...is going to be a great release.  Nothing so revolutionary that you can't leverage all the skills that you (or your staff) have already learned.  Yet, there are many, many important feature additions and "rough edges" removed.

Rattling off a few of the new items:

• In-order delivery for any adapter that supports it (i.e. MSMQ, MQSeries, etc.)  In 2004, only MSMQ/T supported this.  (Of course, a faulty orchestration can break first-in-first-out - more on that in a later post.)
• The introduction of an "Application" concept for grouping BizTalk assets - which extends to orchestrations, role links, send port groups, send ports, receive ports, receive locations, policies, schemas, maps, pipelines, other resources (e.g. soap proxies),  you name it!  Just as importantly, the management infrastructure understands applications - so health/management views can be narrowed down appropriately.
• The management infrastructure has been completely encapsulated in an MMC - HAT is largely hidden.  More interesting is that the MMC can manage multiple BizTalk groups - and can do so remotely (by definition...)
• A packaging/deployment solution that looks good - we'll have more to say about that in the coming weeks!  The developer experience in particular looks to be quite good.  Likely still some value-add to be done on the server side.
• Ability to route failed messages - and subscribe in your orchestrations.
• Calling pipelines from within orchestrations (no more loopback adapter or similar solutions needed...)
• Zoom and expand/collapse-state-preservation within orchestrations.  (So when you collapse that big group or scope shape, it will stay collapsed across close/re-open.)
• BAM integration with SQL Notification Services.
• "Operator Role" has been defined to make allocating administration tasks a bit easier from a security perspective.
• Pipelines can have per-instance configuration - saving you from recreating what were essentially a lot of duplicate pipelines!  (This was possible in 2004, I believe - but not exposed cleanly.)

This will be fun...I look forward to exploring the beta bits (man, the CTP was pretty short-lived...!)

As has been noted before in the lore of BizTalk, it sure would be useful to use a real debugger with orchestrations - at least occasionally.

There are times when an expression shape winds up getting a bit sticky (not that I would know...) and a debugger would be just the ticket.  Other times, the exception you are getting from the orchestration engine isn't at all clear.

Still other times, seeing the actual contents of messages or context as you step through would be interesting.

Jon has posted on debugging orchestrations (in IL) with an ILDASM/ILASM loop, and had also discussed building your orchestration assemblies manually (using xsharpp.exe).  (He correctly noted these approaches wouldn't generate anything supportable, but they work for spelunking.) 

I wanted to suggest something a bit different - I wanted to go back instead to the venerable BTS File Dump utility that Charles released in May of 2004 (before the GenerateCSFiles registry key was discovered...) and propose a different technique for symbolic debugging.

First, install Charles' BizTalk File Dump utility - you can download it from here.  Fire it up, and change the output path in the utility to something easy, like c:\temp\BTSFileDump.

Here is the Edit/Debug cycle...(Start by making sure the file dump utility is running, and click the "Start dumping files" button.)

1. Build your orchestration project (or whole solution if need be.)
2. Do a Ctrl-Shift-F (Find in Files) in Visual Studio, and change the "Look in:" folder to c:\temp\BTSFileDump
3. Search for something in one of your expression shapes, say, "MyClass.Execute".  The correct generated file (that the file dump utility grabbed) will appear in your Find Results - open it up, and set a breakpoint.  (Not on the xml designer portion - on the actual code!)
4. From the Debug menu, choose Processes and attach to BTSNTSvc.exe.  (Have more than one?)  Choose CLR debugging only - not native.  The symbols should be loaded automatically - no need to copy PDBs to the GAC.
5. Trigger your orchestration however you normally would.  Bask in a picture like this:  (Puts a Petzold-style WindowProc to shame...) 
   
   
   
6. Use QuickWatch to examine message contents/context, if you like - you'll be interested in expansions like this one (where sampleRequest is a message in the orchestration.)
7. Find your problem, Debug-Detach All, and fix the problem in the orchestration.
8. Click "Delete all files in output path" in the file dump utility (to avoid duplicates) and rebuild.  Repeat the process if you need to.  (To save time, consider an external tool that just re-GACs the orchestration assembly and resets the host process.  See here.)

Note that this technique will work in a production setting.  You could copy the PDBs and sources to the production server, and use DbgClr.exe (in the Framework SDK) or cordbg.exe to attach to the appropriate host process.

Happy debugging...

Andy Morrison wrote about a technique for identifying the correct host instance to attach to when debugging components associated with orchestrations.

A potentially easier solution is to keep perfmon running with the counter shown below (BizTalk:Messaging/ID Process - all instances.)  The counter values will update every time a host instance recycles.  The Debug-Processes dialog in Visual Studio will show you (and allow you to sort by) process IDs to make this easy. 

Then put PerfMon in "report view" to easily see process IDs by host instance: