trace of thought (Scott Colestock) - Deployment Framework

New version of XmlPreprocess...

Scott Colestock — Thu, 06 Nov 2008 15:29:05 GMT

A tool that has been an important part of the BizTalk Deployment Framework has been updated by Loren – see the codeplex project here.

XmlPreProcess is a general purpose tool in its own right for managing configuration files across multiple environments. The tool has pulled in previously separated functionality (the excellent stuff done by Tom) so that it can consume spreadsheets (that describe environment variations) directly, rather than needing a separate process for that. Very slick stuff !

Slides on Unattended Deployments...

Scott Colestock — Thu, 19 Jun 2008 19:03:48 GMT

As I mentioned previously, I did some talks recently (user group, Teched 2008) on “unattended” deployments for BizTalk. You can download the slides here.

Soon, I’ll be posting a generic Team Build script for BizTalk projects as well as a MSBuild script that can be used with TFS Deployer (for unattended deployments.)

Deployment Framework Release on CodePlex...

Scott Colestock — Fri, 15 Feb 2008 22:40:24 GMT

So, I should have moved the BizTalk Deployment Framework to CodePlex long ago, but better late than never. Find it here.

A few small items:

Better support for 64 bit platforms in scripting operations
Support for using the 64 bit isapi extension for http adapter
A "startApplicationOnDeploy" property you can use to control whether your application immediately starts after deploying
Bug fixes for the lightweight "Update Orchs/Comps" tool menu item
Automatic creation of "ProjectPath" sso config item – that lets you know where your developer workspace or server installation directory resides.
Better support for pre-processing bindings on developer workstations

But the cool item is integration of Tom Abraham’s improved “SettingsFileGenerator” – see here. It moves the process of generating individual environment setting files from a VB macro inside of the SettingsFileGenerator.xls spreadsheet to an external command-line tool.

It also allows you to use the Excel XML format for the SettingsFileGenerator spreadsheet. The Excel XML format allows for the ability to do standard diff/merge activities, a big improvement…

The way we’ve intergated this tool into the deployment framework allows you to treat the individual settings files as "transient" - they will get auto-generated by the deployment process now on the fly, and do not need to be version controlled or manually generated after spreadsheet edits.

Enjoy!

Deployment Framework for BizTalk 2006...

Scott Colestock — Fri, 30 Jun 2006 20:46:41 GMT

(Update: The original download files were missing the PDB-to-GAC functionality I've discussed before. Please download again if you have already...)

The BizTalk Deployment Framework has been updated to work with BizTalk 2006...It is hard to believe that this project has been going on since May of 2004!

The Deployment Framework for BizTalk 2006 still has the same goals as the 2004 version:

Streamline the deployment process for developers, ensuring repeatability
Make it easier for a team of BizTalk developers to stay in sync - not just with BizTalk artifacts, but with all the other infrastructure that surrounds the solution (virtual directories, queues, file folders, etc.)
Extremely close parity between server-side deployments and developer deployments - so you are always testing your server deployments as you go through a typical developer work day.

BizTalk 2006 itself introduced quite a few features to make deployment easier, and can work fine for small (or solo) projects. Here are a few limitations I've encountered:

Though much improved, it is still possible to get into "dependency chaos" - where you spend time manually undeploying/deploying individual artifacts.
Binding changes have to be communicated "manually" between developers on a team, since the import process is done through the Admin MMC or command-line. Just "getting latest" from version control isn't sufficient.
Binding files for your various environments have to be maintained as separate files (manually), rather than "merging in" environment-specific settings
Artifacts such as rules, virtual directories, queues, folders, additional dependent assemblies, etc. can be represented as "Resources" within the Admin MMC (and in exported MSIs), but not in a fashion that easily moves between developers on a team.

So! To get started with this version, download the Deployment Framework (Tools) zip and run the MakeBizTalkExternalTools_VS2005.vbs script. This will add entries to the Visual Studio tools menu for deploying and undeploying using the framework. You can download the full sample to see the framework in action (first build it, then do Tools-BizTalk Deploy. You'll see something like this.)

The high-level approach is the same as the 2004 framework - you supply a small project-specific NAnt script that indicates via properties what elements of a deployment your solution requires. You include BizTalkDeploymentInclude.nant to get all the core deployment functionality, and make sure the DeployTools directory is copied to your project. (Unzipping the Deployment Framework Core into your project is a good way to do this.) See the documentation for a more complete discussion.

The primary difference in the upgraded framework for BizTalk 2006 is that we now create a BizTalk Application definition, and use BTSTask to add all BizTalk artifacts as resources within that application. Starting and stopping the application is done at an application level rather than per port/per orchestration.

Packaging up your solution as an MSI can be done with the WiX-based scripts that have been in the framework for awhile, or by using the MSI export mechanism in BizTalk 2006, if you prefer. (The latter solution will require a few additional steps, and doesn't give you the parity described earlier. But it works if you need to go that route.)

Other Notes:

NAnt .85 RC4 (and NAntContrib ) is required. Be sure to copy the new BizTalk.NAnt.Tasks.dll to NAnt's bin directory. We need to call BizTalk's .net 2.0 assemblies, and NAnt wasn't built against 2.0 -- so change nant.exe.config to have only<supportedRuntime version="v2.0.50727" /> in the<startup> element.
Log4net usage in the sample (which isn't required of course for the framework) has been updated to log4net 1.2.10 , as has my serializable wrapper. You can find the new log4net.Ext.Serializable in the Tools download (which is useful all by itself, apart from the framework.)
You can actually use the BizTalkDeploymentInclude.nant file in this release with BizTalk 2004, if you like, to aid in your migration. There is NAnt property to indicate 2004 vs. 2006.
Scan all the previous release notes...

Leave comments with any questions/issues/etc. Enjoy!

Download: Full Sample, Framework Core, Tools Source , Docs

PDBs in the GAC...

Scott Colestock — Wed, 01 Mar 2006 18:43:19 GMT

In the last release of the BizTalk Deployment Framework, an additional feature was added that I thought deserved its own post for explanation...

If you are responsible for troubleshooting BizTalk applications in production, you are likely hungry for all the information you can get about why something is failing. Since all BizTalk project assemblies are in the GAC, the stack traces you get (either from your own logging, or the event logs BizTalk generates for unhandled exceptions) do not contain file and line number information. This makes post-mortem analysis a lot harder...

A new custom NAnt task (getgacpath) and corresponding support within BizTalkDeploymentInclude.nant was added for placing PDB (program database) files for all BizTalk and component assemblies into the GAC. (You can use just this NAnt task without the rest of the Deployment Framework, of course. In addition, a standalone command line utility called GetGacPath has been included in the "Tools" download.)

Why is this interesting? Well, because it allows you to get stack traces that include file and line number information even when assemblies reside in the GAC. (This is obviously useful and useable outside the domain of BizTalk...) Without extra work, file and line number information is generally missing from a stack trace if a PDB file is not co-located physically with the corresponding assembly. Putting the PDB file directly into the GAC is an easy way of solving this problem.

Set the "deployPDBsToGac" NAnt property to "true" in your project-specific NAnt file to get this functionality.

A couple of points are worth mentioning:

If the directory structure of the GAC were to ever change, this would need to be updated. (See my upcoming post on using this technique with .NET 2.0.)
It would be great if Gacutil.exe provided this functionaity, but it doesn't.
You should strongly consider having your release-mode binaries (aka 'deployment' configuration in BizTalk) build PDB files - there is no reason not to. Look under Project properties/Configuration properties/Build/Generate Debugging Information.
Stack traces that show up in the event log either through your own logging, or via BizTalk (because the exception was unhandled) will now have file and line number info. For orchestrations, the file name will correspond to the temporary C# file generated at compile time. If you use the techniques described in this post , you can correlate back to the actual statement that failed. For instance, the event log might say:
```
[4740] ERROR BizTalkSample.Orchestrations.TopLevelOrch - An exception was caught. 
[a08701ff-bf80-4940-9f9f-c2bb8597684b]
System.Exception: Something exceptional happened.
   at BizTalkSample.Orchestrations.TopLevelOrch.segment2(StopConditions stopOn) in 
   c:\Documents and Settings\Scott\Local Settings\Temp\h0li3wbi.0.cs:line 2538
```
If you grab the file mentioned (h0li3wbi.0.cs) using BTSFileDump (again,see here) you can cross-reference this message to a location in the generated C# code for TopLevelOrch:

You might save off "h0li3wbi.0.cs" (the temp file name chosen when TopLevelOrch compiled) and the other generated files when you do your build, using BTSFileDump. (That would be tough to do if you build with CruiseControl, etc.) But you wouldn't have to do that - if you can retrieve the source code used for a particular deployment, then the line number alone should be sufficient to get you to the exception site (since you can find the correct temp file by looking for comments, etc. that are in the orchestration - just like if you were using this technique to do live debugging.) This all gets better with BizTalk 2006 - see a future post on this topic.
Final point: If you set deployPDBsToGac to "true", deployments will begin by stopping the BizTalk services (instead of just bouncing the services at the end.) The reason for this is that the BTSNTSvc.exe process will not "let go" of an assembly for which it has loaded a PDB file (so you have to terminate it.) This means you may not always want this switch on for the developer edit/run/debug cycle.