(Update: See the latest on the Deployment Framework
here.)
Well, its time for another revision of the BizTalk/NAnt deployment tools...
For folks recently joining us, these tools allow you to quickly create both developer workstation deployments (inside Visual Studio) and MSI-based deployments (for servers) for BizTalk 2004 solutions. Just about all aspects of BizTalk deployments are handled, including items such as virtual directories, SOAP proxies, and WinXP/2k/2003 differences.
I get quite a few emails from people who are using these tools (which is great!) A couple folks have suggested that I start referring to the whole thing as a" p framework?. < deployment>
That sounds about right - like all frameworks, there is a core set of functionality that takes care of quite a bit of work, and a given project can supply what needs customization or what is missing.
In this case, the core NAnt scripts, custom NAnt tasks, WiX generation, and other custom tools form the framework itself - and a given BizTalk project can supply project-specific details, what needs to be customized, and any missing functionality. So I'll refer to this work as the "Deployment Framework for BizTalk" from now on...
What has changed in this rev?
- When last posting on this topic, I should have been even clearer that as of the last release, the deployment framework requires NAnt .85.
- The biggest improvement is the ability to use binding files as they are emitted by the BizTalk Deployment Wizard directly. In the past, the framework had insisted that binding files be split between orchestration bindings and messaging bindings - this is no longer required. I had several requests for this fix...Also, note that having multiple binding files is supported (like one for each of several assemblies.) Now, if you wish to continue to use the "old style" split bindings, define the orchBindings property in your project NAnt file as shown below:
<property name="orchBindings" value="${nant.project.name}.OrchBindings.xml" overwrite="false" />
- You can now set the value of NAnt properties from the SettingsFileGenerator spreadsheet. Recall that this spreadsheet is used (in conjunction with XmlPreProcess) to manage environment-relative configuration - that is, to manage those values that vary depending on whether you are in a development, QA, or production environment. Until now, it has been difficult to use those configuration values within the NAnt deployment file! You typically just used them in your binding files or other application-level XML files. Now, however, you can define a property as shown below to make these values available as NAnt properties. A good example usage is the case where you want the actual names of the SSO Application Users group and SSO Administrators group to be defined in the SettingsFileGenerator spreadsheet (because the group names may vary by environment you deploy to.) Just define a property in your project NAnt file called "propsFromEnvSettings" with a comma-separated list of values you want pulled in. (The sample has been updated to show this, too.)
<property name="propsFromEnvSettings" value="ssoAppUserGroup,ssoAppAdminGroup"/>
- Many people had asked for the DeployResults.txt file (which is created in the DeployResults subdirectory during a deployment) to be available in a history log, so that it would be easy to walk up to a production server and determine what deploy/undeploy activity had taken place (especially useful for diagnosing any failures.) To that end, the deployment framework will preserve past DeployResult.txt files with this convention: `DeployResults TRACESATURN_05032005_0922.txt' Note that the machine name, date, and time are preserved - so you can gather these files across all the servers in a BizTalk group, if need be.
- SSOSettingsFileReader class improvements: In the last release, I introduced the ability to manage settings within the BizTalk SSO using the SettingsFileGenerator spreadsheet that is also used for managing environment-relative configuration. Part of that functionality included a class called SSOSettingsFileReader, used to retrieve settings at runtime. This class now has some strongly typed accessors, and no longer sits inside a lengthy namespace, which means that in an orchestration expression shape you can simply do:
someVar = SSOSettingsFileReader.ReadString("BizTalkSample","SomeAppConfigItem") // or ReadInt32
- Ben Cops had found that on large-scale deployments, it was useful to restrict which orchestrations were deployed/started to speed the development cycle. So, he modified the orchestration NAnt task accordingly. To use (if you need to), just set the orchestrationsToIgnore property in your project NAnt file with a comma-separated list that reflects what you are currently working on. Make sure to copy BizTalk.NAnt.Tasks.dll to your program files\nant\bin directory again! (you'll find it in the DeployTools directory in the download.)
- A bug was fixed with undeployment for the case where an SSO affiliate app had already been deleted...
- All references in NAnt files to xmlns="http://nant.sf.net/schemas/nant-0.84.win32.net-1.0.xsd" were removed, because they were simply causing heartache (for many reasons.) Remove this namespace in your project NAnt file when upgrading.
- Rather than trying to keep the source for the C# tools (NAnt tasks, SetEnvUI, DeployBTRules, SSOSettingsFileImport\SSOSettingsFileReader, etc.) current within the GotDotNet workspace, I'm making them available as a zip.
Remember, you do not need to piece together the current functionality set from past blog entries - the complete documentation (in Word form) is in the download and separately here.
Some last thoughts:
- Some folks have been reluctant to use the deployment framework because of its reliance on NAnt, and a worry over using NAnt in a server environment. One thing to remember (that might allay this concern) is that a subset of NAnt is delivered with the generated MSI, and NAnt will not require a separate installation procedure by your production operations team. Moreover, NAnt doesn't create any kind of persistent service or footprint on the production machine (i.e. no automatic file associations that would cause scripts to run with a double-click or anything like that...)
- Add a Visual Studio External Tool for the updateOrchestration target if you haven't already! This can speed your development cycle significantly. See the tools zip for a script that creates all relevant external tools.
- When using the WiX-based MSI generation, make sure your project NAnt file (for WiX generation, like BizTalkSample.WiXSetup.Build) is choosing "debug" or "release" based on what you want to put in the MSI. See comments in that file for more detail, and Jon Flanders' post on this topic.
How do I upgrade?
- With a safe copy of your BizTalk solution, expand the "Core" zip contents right on your project directory, such that the core scripts will overwrite what you have now (but your project-specific scripts will be preserved.)
- Manually copy the WiX-related updates to your *.WiXSetup directory, since they will get unzipped to BizTalkSample.WiXSetup
- Remove xmlns="http://nant.sf.net/schemas/nant-0.84.win32.net-1.0.xsd" from your project NAnt file
- Copy DeplyTools\BizTalk.NAnt.Tasks.dll to your program files\nant\bin directory
- Do a deployment on a developer workstation and a server to test things out...
Downloading
- You can always use the download links on this blog.
- Full sample here, core scripts here, extra tools here.
(But always check the download links to see if there has been an update…)
- Standalone documentation here (and diagram here.)
- GotDotNet workspace here.
(Update: See the latest on the Deployment Framework
here.)
In the latest release of the Deployment Framework for BizTalk “extra tools”, you will find a script that will automate the creation of “external tools” in Visual Studio that are useful for working with BizTalk 2004 and the Deployment Framework. Modify the script to include all the tools that your team relies on, too.
The idea that this should be automated came from Ben Cops (though I’ve heard the complaint before!) Manually creating all the useful external tools was somewhat painful for a large team of BizTalk developers.
After you run the script, your Tools menu in Visual Studio should include entries for deploying and un-deploying the current solution, as well as for updating just the orchestrations, .NET components, and SSO configuration data (one menu entry.) In addition, there will be entries for running NAnt on a highlighted target name (useful if you have your build script open), and for running HAT and the Subscription Viewer. Enjoy…
A short while ago, I did a presentation for the Twin Cities .NET user group on log4net. (You might recall an earlier blog entry where I discussed using log4net with BizTalk 2004...)
The presentation was not specific to BizTalk 2004 - instead, it attempts to describe why I think the log4net library is so very well thought out. You can grab it here, if you like.
The last slide references Loren Halvorsen's comparison of log4net and the new Microsoft Enterprise Library Logging Block, which I would recommend taking a look at (the comparison, that is.) Tom Hollander (among many others!) later weighed in with this piece.
Lastly, 1.2.9 beta of log4net has recently become available (release notes here)....However, my extensions to log4net have not yet been updated to reflect this new drop. I'll be sure to post when I'm able to update.
(Update: See the latest on the Deployment Framework
here.)
Just when you thought automated deployments for BizTalk 2004 were safe, another
update to the “Deploy with NAnt” template comes along…
:)
This is a very worthwhile "upgrade" to consider taking advantage of. One
of the big items is support for "configuration info within SSO", but a complete
list of features/fixes, etc. is below.
SSO Integration: If you are currently using the
SettingsFileGenerator.xls spreadsheet (discussed in previous entries) in
conjunction with the XmlPreProcess
tool to handle "environment-relative" configuration - that is, to handle
variations between your dev/QA/production environments - you will be glad to
know you can get more leverage out of that process now…
To review, with this process, you use the names that are defined in the first
column of the SettingsFileGenerator spreadsheet (shown below) as "tokens"
within your binding file, as in:
<!-- ifdef ${ xml preprocess} -->
<!-- <Address>${FileSendLocation}\%MessageID%.xml</Address>
-->
<!-- else -->
<Address>C:\temp\BizTalkSample OutDir\%MessageID%.xml</Address>
<!-- endif -->
At install time, the value for FileSendLocation that is appropriate to the
environment you are deploying to is "plugged in" by
XmlPreProcess (but for developers, the file remains legal XML.)
The spreadsheet might look as follows:
(click)
Now, however, you can also put in name-value pairs (whether they
vary by environment or not) into the spreadsheet that you need access to at run
time (i.e. general configuration information.)
The name-value pairs will be deployed into a newly created affiliate
application within the SSO, and can be read using the SSOSettingsFileReader
class (which serves as a cache, too – and is provided in the download.)
The static methods on this class can be used from an orchestration
expression shape, or from assemblies that orchestrations call – as in
string test =
SSOSettingsFileReader.ReadValue("BizTalkSample","SomeAppConfigItem"); (By
the way, the “nested names” shown in the spreadsheet above are just
meant to suggest a partitioning mechanism you might want to use, rather than
one flat “namespace” of config data.)
Of course, the deployment scripts handle all the interaction with the SSO
database for deployments/un-deployments. This whole mechanism
(SettingsFileGenerator.xls, XmlPreProcess, and now SSOSettingsFileImport.exe)
is worth using even if you don’t use the rest of this script
infrastructure. Note that Jon Flanders had a
great article on SSO-based configuration using a strongly-typed
approach, where as this approach is loosely-typed, but leverages a mechanism
you might already be using (i.e. the spreadsheet concept.) (Note:
Jon’s code for creating/deleting affiliate apps helped me complete the
SSOSettingsFileImport.exe utility quickly…)
Other features/fixes:
-
Support for multiple schema assemblies, including the case where schema
assemblies reference each other due to schema imports. (Undeployments
occur in reverse order of deployments.)
-
Support for multiple orchestration assemblies, including the case where
orchestration assemblies reference each other due to the use of Call/Start
Orchestration shapes. (Undeployments occur in reverse order of
deployments.)
-
Support for multiple orchestration and port binding files (just list them as
comma separated values in the orchBindings/portBindings properties.)
-
Fixed bug that occurred when undeploying ISAPI extensions in Windows 2003 / IIS
6.
-
Fixed bug in undeploying pipeline components (thanks to John Adams)
-
Added /NOFORCE flag to IISRESET calls in accordance with
http://support.microsoft.com/kb/286196
-
New property "includeInstallUtilForComponents" was added, which can be set to
'true' in your project-specific nant file. This will cause your component
assemblies to be called with "installutil.exe" (with the /u flag for
undeployment.) This is useful if you have .NET Installer-derived classes
that need to get called for creating event sources, perf counters, etc.
(Yes, you could certainly argue these should be called by the WiX piece of the
script infrastructure instead, but this method works and is more in keeping
with the spirit of the entire process.) Note there is no harm if a given
component assembly listed in your components property does not have an
installer class.
-
Eliminated the need for the includeCustomTarget property, and provided
additional ways to supply custom functionality. Project-specific nant
file can now supply any or all of customDeployTarget, customPostDeployTarget,
customUndeployTarget, customPostUndeployTarget -- and they will be called if
they are present.
-
Integrated the deployment of log4net-specific stuff, if the includelog4net
property is set to true. See the work on
BizTalk/log4net integration for more information. (Log4net sample
usage is included in the new sample.)
-
Now using NAnt version .85 - so be sure to grab the latest
NAnt/NAntContrib
since the nant scripts are using .85-specific syntax to avoid "deprecated"
warnings that were occurring otherwise. Note that even once you have
installed NAnt .85, you will still be able to deploy BizTalk projects that are
using old versions of the deployment scripts. Note also that this is a
pretty basic port to .85 - there are no doubt more elegant ways of doing lots
of things that I haven't investigated yet, now that nant has expression
support.
Other notes
The scripts like to see BizTalk projects using Debug and Release for target
names. A long while back I had suggested a file-level search/replace
within btproj files to change these, but two more elegant options are
available:
-
In project properties, change the output location for the
“Development” configurations to “bin\debug” and output
location of “Deployment” configurations to
“bin\release”.
-
(Via Puneet Gupta): Change the template for BizTalk projects, found in
BTSApp.btproj file in %BizTalkInstallDir%\Developer
Tools\BizTalkWizards\BTSApp\Templates\1033, to reflect Debug and Release
targets.
Another note: Lots of folks have noticed that XmlPreProcess can be used for a
lot more than port binding files! Log4net config files, orchestration
binding files, you name it. See the full Word documentation in the
download for an example of how to do this within customDeployTarget.
How do I upgrade?
-
With a safe copy of your BizTalk solution, expand the “Core” zip
contents right on your project directory, such that the core scripts will
overwrite what you have now (but your project-specific scripts will be
preserved.)
-
Manually copy the WiX-related updates to your *.WiXSetup directory, since they
will get unzipped to BizTalkSample.WiXSetup
-
Do a deployment on a developer workstation and a server to test things
out…
Downloading
-
You can always use the right-hand links on this blog.
-
Full sample
here, core scripts
here
-
GotDotNet workspace
here. Because of issues with GotDotNet, I’m only going to be
keeping the EXE utility source code up to date – not the entire sample
& core tree.
Enjoy – comments always welcome and appreciated.
Steve and I wound up working on the same problem at the same time (probably for the same person…)
When working with the MSMQ adapter, keep in mind that you must reference the Microsoft.BizTalk.Adapter.MSMQ.MsmqAdapterProperties.dll to have access to MSMQ-specific properties. (The list of properties available is in the adapter documentation.)
Why? The intellisense in the expression shape (when using the parentheses syntax on messages, ports, etc.) is looking for classes derived from Microsoft.XLANGs.BaseTypes.PropertyBase to present in the drop down list. Some of those classes are part of your "native" BizTalk installation, some are provided by add-on adapters, and (of course) some are provided by property schemas that you develop. A reference to the containing assembly is necessary to find them.
Other things to note relative to the MSMQ adapter:
- For dynamic send ports, the syntax can look like: SomePort(Microsoft.XLANGs.BaseTypes.Address) = @"MSMQ://FORMATNAME:DIRECT=OS:SOMEMACHINE\PRIVATE$\SOMEQ";
- In the case of dynamic send ports, the runtime will use non-transactional sends by default. If you want transactional sends, you will need to set the transactional property: outboundMsg(MSMQ.Transactional) = true;
- Transactional messages which fail to deliver to the remote queue will be found in the local Transactional Dead Letter queue. No error will be raised by the adapter. This may mean that your design requires acknowledgement messages to achieve what you are looking for.
I was just reading Mike Holdorf's post on the bts_CleanupMsgbox stored procedure. The team I work with has been able to make good use of this sproc as well, especially while stress testing.
One thing to beware of is this: Your habit for doing this kind of thing might lead you to open up Sql Query Analyzer, and right click on the stored procedure - selecting “script to new window as execute.” This will generate the following:
DECLARE @RC int
DECLARE @fLeaveActSubs int
-- Set parameter values
EXEC @RC = [BizTalkMsgBoxDb].[dbo].[bts_CleanupMsgbox] @fLeaveActSubs
Ahhh, but the stored procedure actually defaults fLeaveActSubs to “1”, whereas this code will leave fLeaveActSubs as “0”. This will cause all your subscriptions to be deleted, and you will have to redeploy your BizTalk applications. Not that I would know....
So just do “exec bts_CleanupMsgbox”...
(Note - this work has been updated for log4net 1.2.9. See the 'Tools' download in the Deployment Framework)
Early on when working with BizTalk 2004, it might be tempting to view the
ability to track orchestration events (and the use of the orchestration
debugger) as a substitute for traditional diagnostic logging.;
After all, the orchestration debugger can tell you exactly how far you got in
an orchestration, and what path through the workflow you took to get there
However, consider these limitations:
-
The orchestration debugger is used either a) after the orchestration has
completed or b) in conjunction with a breakpoint. Neither of these
choices is ideal, and only the latter gives you visibility into the
intermediate values of variables/messages.
-
Once an orchestration involves looping constructs, multiple orchestrations
connected via messaging, etc. the orchestration debugger often loses quite a
bit of diagnostic value.
Given this, it can be useful to integrate diagnostic logging into your
orchestrations and the components they call. If we are going to invest in
this effort, we’d like to use something richer than the built-in
System.Diagnostics.Trace/Debug infrastructure.
Log4net is an Apache-sponsored
initiative within the “Apache Logging
Services” project. It provides a rich diagnostic infrastructure
for .NET, with support for hierarchical logging and configuration, multiple
logging targets, and support for logging context.
A recent
article outlined the advantages of log4net over Microsoft’s Enterprise
Instrumentation Framework (EIF), though Microsoft/Avanade will be revamping
this logging infrastructure with the release of the
Enterprise Template Library
Given that the log4net initiative has broad support at the API layer (i.e.
log4j, log4PLSQL, etc.) and platform layer (including .NET CF) as well as the
benefit of maturity, log4net seems like a very reasonable choice for
instrumenting BizTalk 2004 applications. (The Enterprise Template Library
logging component will likely be a good choice as well when released.)
Following is a discussion of log4net itself, along with what is required to use
it in a BizTalk 2004 setting.
Quick Insight into log4net
Without diving immediately into the log4net API, let’s get a sense for what it
can do. What do we mean by hierarchical logging and configuration with
multiple target support?
Suppose we have classes with these namespace-qualified names:
MyCompany.MyDepartment.MyClassA
MyCompany.MyDepartment.MyClassB
Each of these classes can obtain a logger specific to their class by declaring
a static member as such:
private static readonly ILog log = LogManager.GetLogger(typeof(Foo));
Each class (MyClassA/MyClassB) can now emit trace with statements such as log.Info(“hello
world”)> or log.Warn(“a potential problem…”).
Default log levels include: Debug, Info, Warn, Error, and Fatal (though these
can be expanded.)
The concept of “context” can be added to a stream of trace messages via
NDC.Push(“some context”)/NDC.Pop or MDC.Set(“somekey”,”somevalue”).
(These class names are short for “Nested Diagnostic Context” and “Mapped
Diagnostic Context” respectively.) These can be extremely useful for
multithreaded/multi-user applications, to distinguish the streams of related
trace messages from each other. This context can be optionally formatted
into the associated tracing statements, as will be shown shortly.
If we want to turn on tracing at the ‘Error’ level by default, at the ‘Warn’
level for MyCompany, and at the ‘Debug’ level for MyClassA, we would simply
configure as follows (notice the use of the namespace hierarchy):
<root>
<level value="ERROR" />
</root>
<logger name="MyCompany">
<level value="WARN" />
</logger>
<logger name="MyCompany.MyDepartment.MyClassA">
<level value="DEBUG" />
</logger>
This can be changed on-demand simply by changing the configuration file, which
can be reloaded automatically without restarting the application. This
allows us to a) not be blinded by too much trace information and b) not impose
undue server load by tracing, whereas a non-hierarchical approach only allows
for one logging level for an entire application.
Suppose it is desired to route all trace information (by default) to
OutputDebugString, where a utility such as dbmon.exe or
DebugView can be used to view it. Suppose further it is desired to
prefix our output with the date, thread Id, logging level, the name of the
logger (i.e. MyCompany.MyDepartment.MyClassA), and the current context.
To do this with log4net, we would have our configuration file declare an
“appender” of the appropriate type, and include a declaration of a “layout” as
follows. The “appender” is then referenced by a particular logger (or the
root logger).
<appender name="OutputDebugStringAppender"
type="log4net.Appender.OutputDebugStringAppender" >
<layout type="log4net.Layout.PatternLayout">
<!-- The format specifiers let us add a wide
variety of additional info -->
<param name="ConversionPattern" value="%d [%t]
%-5p %c [%x] - %m%n" />
</layout>
</appender>
<root>
<level value="ERROR" />
<appender-ref ref="OutputDebugStringAppender" />
</root>
This might yield trace output as follows:
16:39:33,022 [316] INFO MyCompany.MyDepartment.MyClassA
[SomeContext] - Hello World
Again, the specifics of what additional information can be included in trace
output can all be configured dynamically at runtime. Note that other
useful information such as method name, file name, line number, caller’s window
identity can all be added with the PatternLayout class as well, though it is
quite expensive (further adding value to the hierarchy concept!)
Moreover, each element of the hierarchy can be directed to a different
appender, or to multiple appenders. Built in appenders include support
for ADO.NET, .NET remoting, SMTP, files, rolling files, buffering, and the
Event Log (among others.) Because the Event Log is supported, there is no
need to use a separate API to support such logging.
Core log4net Concepts
Without attempting to present a full tutorial (see
this introduction for a more complete discussion), the core concepts in
log4net include:
Loggers: Use by code to emit trace output at various levels.
-
Loggers are organized into hierarchies just like .NET namespaces (they don't
have to correspond, but often should.) Example: System is parent of
System.Text, and an ancestor of System.Text.StringBuilder.
-
It is typical to have the logger name equal to a fully-qualified class
name.
-
Loggers are retrieved using a static method of LogManager, e.g.
LogManager.GetLogger(someStringOrTypeName);
-
An ILog interface is returned with Debug, Info, Warn, Error, Fatal methods on
it for tracing.
-
Logging level is generally inherited from the hierarchy, and be configured
anywhere in the hierarchy.
-
Rule: logging level is equal to first non-null value working UP the tree.
-
Predefined levels: DEBUG, INFO, WARN, ERROR, FATAL, ALL, OFF
-
Default level for root in hierarchy is DEBUG
Appenders: An output destination for logging.
-
Built in: event log, asp.net trace, console, file, rolling file, smtp, etc.
-
Appenders can optionally have "layouts" to specialize how formatting is done
and add additional information (windows identity, source code info, context,
thread id, timestamp, etc.)
-
Appenders are attached to a logger. Log requests will go to a given
logger's appender, as well as all appenders attached up the hierarchy (unless
"additivity=false" at some level, in which case moving up hierarchy
stops…)
Filters: A given appender can apply a filter such that only log requests
that match some criteria will actually be output. Filters are applied
sequentially. Built-in filters include string match and regex
match.
Configurators: Classes which initialize the log4net environment.
Configuration is done either through assembly attributes that specify a config
file, or an explicit call to specify a config file. The config file can
be a standard .net config file, or standalone. Configuration is by
default at the appdomain level, but can be finer-grained through the use of
log4net "logging domains".
Using log4net with BizTalk 2004
There are a few challenges to using log4net in a BizTalk 2004
environment. They can be summarized as follows:
-
Assembly-attribute-driven configuration won’t work because BizTalk 2004 does
not appear to support the addition of custom attributes for BizTalk
assemblies. (And even if it did, this method would pose issues for
BizTalk environments.)
-
Using the log4net configuration classes for configuration is problematic
because there is no clear point at which to make the call. (How does as
an orchestration know if it is “first” and should initialize? How does it
know at any
point in an orchestration whether the BizTalk service has been recently
recycled, losing the log4net configuration?) Moreover, there is not a
good way to identify where the configuration file should be located.
-
ILog-derived classes (log4net loggers) are not serializable “out-of-the-box”,
making it difficult to use them in an orchestration setting.
-
Log4net context-storage mechanisms are thread-relative, which isn’t
workable for orchestrations, since orchestrations often a) dehydrate and then
re-hydrate on a different thread and b) make use of parallel branches.
We’d like to correctly associate context such as the orchestration instance ID
on a logger that is scoped to an orchestration.
To address these challenges, I’ve created
extensions to log4net that attempt to stay within log4net’s design
intent. These are housed in an assembly called
“log4net.Ext.Serializable”. (Update – these are now
currently found in the “Extra Tools” download for the Deployment Framework –
see the download links at the blog home
page.)
The main classes defined are as follows:
SLog: A serializable implementation of the ILog interface, to be used by
orchestrations (or other components) for logging. This class has an
InstanceId property as well as a general properties collection that is carried
with the logger itself. (These can be made available in appender output
via a PatternLayout class and the %P{yourpropname} format specifier. Use
%P{InstanceId} for the InstanceId.) When deserialized, this class can
detect an un-initialized state for log4net and recreate the correct
configuration.
SLogManager: To be used in place of log4net’s LogManager to dispense
SLog classes. This is the accepted way of dispensing custom loggers in
log4net’s design.
RegistryConfigurator: A class which consults a registry key for the
location of a log4net config file, and creates the configuration in the
specified “logging domain” using the log4net DomConfigurator class.
Specifics for BizTalk 2004 Usage
Using the assembly just discussed, the specifics of using log4net with a
particular BizTalk project can be described as follows:
-
Get 1.2.0 Beta8 of log4net at log4net
extension described above, as well as a
full sample project if you like. (Update
– the current version of this sample will now always be found in the
Deployment Framework sample – see download links at the
blog home page.)
-
If you are using NAnt to deploy your BizTalk solution, extend your
customDeployTarget as shown below to create a registry key that will point to a
log4net configuration file within your project’s root directory (whether on a
developer desktop or an MSI base directory). (Handled
for you in current rev of deployment framework.) Also, create a
file with the log4net extension that corresponds to your solution name (i.e.
BizTalkSample.log4net)
<target name="customDeployTarget">
<!-- Write registry key with location of our log4net
configuration file. This step is NOT needed in current rev
of Deployment Framework… -->
<exec program="cscript.exe" failonerror="true"
commandline="/nologo ${deployTools}\WriteRegValue.vbs
HKLM\SOFTWARE\${nant.project.name}\log4netConfig
${nant.project.basedir}\${nant.project.name}.log4net" />
</target>
-
Orchestration assemblies should reference log4net.Ext.Serializable as well as
log4net.dll, and declare a variable in each orchestration (perhaps named
“logger”) of type log4net.Ext.Serializable.SLog.
-
Orchestrations should begin (after an activating receive, if necessary) with an
expression shape that looks like:
logger =
log4net.Ext.Serializable.SLogManager.GetLogger(@"BizTalkSample",log4net.helpers.CallersTypeName.Name);
logger.RegistryConfigurator();
logger.InstanceId = MyOrchName(Microsoft.XLANGs.BaseTypes.InstanceId);
logger.Debug("New request arrived...");
Important: If an orchestration has parallel branches, you will want to declare
multiple variables of type log4net.Ext.Serializable.SLog, assigning 2-n to the
first one, as in:
logger2 = logger;
This prevents the need for synchronizing scopes. (See
this article for more.)&nbps;An alternative is to use scope-local
logger variables that are initialized by assigning them to a global instance.
-
At various points in your orchestration, simply make calls to
logger.Debug/logger.Info/logger.Error, etc. Rely on logger.Error for
event log entries (with appropriate logger/appender configuration.)
-
Use log4net with the components that your orchestration calls as well. If
those components are serializable, they will want to make use of
SLogManager/SLog classes as well in non-static methods. If you want
context (such as the orchestration instance ID) to flow, it might be worthwhile
to make an SLog-typed property visible on the class that the orchestration can
set. (This needs more thought.)
Using log4net with Paul Bunyan
Paul Bunyan is a logging tool that
has a very nice server-side buffering service as well as a great trace-viewing
client that can connect to multiple servers simultaneously (a great plus for
BizTalk groups!) Although Paul Bunyan has its own API, we aren’t
interested in it for purposes of this discussion.
A log4net appender can be easily written for Paul Bunyan, such that log output
can be directed there. Moreover, this appender can be made to explicitly
recognize an InstanceId property carried with a log4net logging event, and map
it to the “Context” concept within Paul Bunyan! This allows for viewing
the orchestration instance ID within a distinct, filterable column in the Paul
Bunyan viewer as such:
(click)
Sample .log4net configuration File
Note that types have to be referred to with fully-qualified names since
log4net.dll is being deployed to and loaded from the GAC.
<?xml version="1.0" encoding="utf-8" ?>
<log4net>
<!-- Define some output appenders -->
<appender name="OutputDebugString" type="log4net.Appender.OutputDebugStringAppender,
log4net,Version=1.2.0.30714,Culture=Neutral,PublicKeyToken=b32731d11ce58905">
<layout type="log4net.Layout.PatternLayout,log4net,
Version=1.2.0.30714,Culture=Neutral,PublicKeyToken=b32731d11ce58905">
<param name="ConversionPattern" value="%d [%t] %-5p %c - %m [%P{InstanceId}]%n" />
</layout>
</appender>
<appender name="EventLog" type="log4net.Appender.EventLogAppender, log4net,
Version=1.2.0.30714, Culture=Neutral, PublicKeyToken=b32731d11ce58905">
<param name="ApplicationName" value="BizTalkSample" />
<layout type="log4net.Layout.PatternLayout,log4net,
Version=1.2.0.30714,Culture=Neutral,PublicKeyToken=b32731d11ce58905">
<param name="ConversionPattern" value="[%t] %-5p %c - %m [%P{InstanceId}]%n" />
</layout>
<!-- Only send error/fatal messages to the event log. -->
<filter type="log4net.Filter.LevelRangeFilter">
<param name="LevelMin" value="ERROR" />
<param name="LevelMax" value="FATAL" />
</filter>
</appender>
<!-- Setup the root category, add the appenders and set the default level -->
<root>
<!-- Possible levels (in order of increasing detail):
OFF, FATAL, ERROR, WARN, INFO, DEBUG, ALL -->
<level value="ERROR" />
<appender-ref ref="EventLog" />
</root>
<logger name="BizTalkSample">
<level value="ALL" />
<appender-ref ref="OutputDebugString" />
</logger>
</log4net>
BizTalk 2004 has some very useful behavior around parallel execution and
scope-level timeouts that it is helpful to have a good understanding of.
What follows is a series of experiments & associated findings that should
shed some light on this area.
Experiment One:
One restriction when using the parallel shape is that if multiple branches make
use of a given orchestration variable, you can get a compiler error:
error X2226: 'someVar': if shared data is updated in a parallel then all
references in every task must be in a synchronized or atomic scope
In the experiment below, we are accesing ‘someVar’ in both parallel
branches - each within an expression shape that also calls Thread.Sleep (the
left hand for 10 seconds, the right hand for 30.) To address the compiler
error just noted, we have a scope around each usage with the
‘Synchronized’ flag on the scopes set to ‘true’.
The trace messages shown to the right of the diagram tell us that, indeed, the
execution of Scope 2 and Scope 3 is completely serialized (in this case, Scope
2 completes before Scope 3 begins.)
This experiment also tells us something unrelated (but useful): that an
exception will not interrupt “user code”. By
“user code” we refer to code in an expression shape, i.e. not
using an orchestration “intrinsic” such as a standard Delay shape
or a Receive shape, etc. Notice via the timings that the exception thrown
in the left-hand branch doesn’t abort the Thread.Sleep in the right-hand
branch. The exception is caught only after the Thread.Sleep in the
right-hand branch completes (though the final trace message in the right hand
branch – ‘after 30 sec’ – does not execute.) This
is important to understand if you have expression shapes in orchestrations
which are making blocking calls to .NET components, DCOM servers, etc.
Experiment Two:
If we eliminate the ‘someVar’ reference in the expression shapes
above, we find that the scopes do not execute serially – whether
the scope synchronization flag is set to true OR false. Notice
below (via the timings) that the sleep operations are executed at the same time
- so it is the presence of the common variable in the expression that forces
the synchronization!
We now have a 20 second gap after sleeping for 10 seconds (because the
exception we throw still doesn’t interrupt us).
Experiment Three:
We would like to use a given instance of a .NET object without imposing
the use of synchronized scopes. As it happens, if we have a .NET object
that is pointed at by two references (i.e. someVar and someVar2 - where
someVar2 was set equal to someVar with a simple assignment), the requirement
that the orchestration compiler normally imposes regarding the use of a
synchronized scope goes away.
In the orchestration below, the left branch is using someVar, and the right
branch uses someVar2. The trace indicates that the sleep operations
happen at same time (though once again the exception doesn’t interrupt
the right-hand side and is caught only when the right-hand completes.)
Lesson: If you have a .NET component (that you know to be thread-safe) you wish
to use in an orchestration - and you wish to use a single instance of it - you
will need to have multiple references to that same instance to use in each
branch of a parallel shape. (The synchronized-scope alternative is likely
unacceptable!) The first variable declaration of your component might use
the default constructor, while the others will have “Use Default
Constructor” set to false and will be assigned to the first.
 
Experiment Four:
Now, there is more to be said regarding exceptions and what they will interrupt
in parallel branches. If instead of using a Thread.Sleep we use a Delay
shape (or a Receive shape, etc.) we find that throwing an exception in one
branch of the parallel shape will indeed interrupt the other branch(es).
Notice in the timings below that the Delay 30sec shape does not complete
once the exception in the left-hand branch in thrown. Particularly when
you are structuring real-world business flows, this is an important and quite
useful behavior.
Experiment Five:
The behavior of exceptions in parallel flows is closely related to another
behavior in BizTalk 2004 - that of what BizTalk is prepared to "interrupt" when
a long-running scope has exceeded the timeout value that has been configured
for the scope. A Delay shape (or Receive, etc.) will indeed be
interrupted when the timeout expires (and a TimeoutException will be
thrown). (Note that for an atomic scope, the timeout governs the maximum
time to allow prior to aborting the transaction.) See the timings below
and note that the Delay 30sec shape does not complete.
 
On the other hand, as you might expect at this point, a blocking call in an
expression shape (like a Thread.Sleep or a DCOM call, etc.) will not be
interrupted. However, the exception will be raised when the blocking call
eventually returns:
 
Summary:
-
The Synchronized flag on a scope will only cause synchronized (serialized)
behavior among “peers in parallel branches” when shared variable or
message state is involved. (This is ignoring any transactional semantics
you might layer on top – which is beyond the scope of this article!)
-
If you have a thread-safe .NET component that you wish to use in an
orchestration from multiple parallel branches, strongly consider having
multiple variables point to a common instance. The first instance might
have “Use Default Constructor = true”, while the remaining
variables will have that flag set equal to false and be assigned to the first
instance in an expression shape:
someVar2 = someVar1;
someVar3 = someVar1;
someVar4 = someVar1, etc.
An alternative is to use scope-local variables that are assigned to a global
instance.
-
A line of code in an Expression shape will not be interrupted by either
an exception in a parallel branch or a TimeOutException arising from a
timed-out scope.
-
A Delay shape or Receive shape, etc. will be interrupted by either an
exception in a parallel branch or a TimeOutException arising from a
timed-out scope.
(Update: See the latest on the Deployment Framework
here.)
It has been about three months since my initial submission to the BizTalk 2004
developer competition that I discussed
here. The contents of that contest entry took a prize (woohoo!)
but still left a lot to be desired…
Since then, I’ve had a chance to make what I believe are major
improvements to the deployment story I’ve been discussing since last
May. In addition, I’ve had a chance to see the practice being used
across several large BizTalk projects, and have done a lot of learning as a
result. (No guarantees or warranty implied, but derivatives of the
scripts/tools I discuss here have been in use for some time by a very large
BizTalk customer I work with.)
The most important changes you will see in this release are as follows:
-
The “core” NAnt functionality has now been separated from the piece
that an individual BizTalk project must “own” and maintain.
This is a huge simplification, and allows for more rapid adoption -- as
well as the ability to deploy bug fixes & feature enhancements in a way
that just wasn’t possible when stuff was intermingled. (Peter
Provost was right
about that aspect of my first attempts.) The NAnt script that a
particular project must own can now be reduced to the following for a simple
case (notice the ‘include’ reference for the core):
<project name="BizTalkSample" default="debugDeploy">
<sysinfo/>
<include buildfile="BizTalkDeploymentInclude.nant"/>
<-- Set following properties to true or false to
include various pieces of a BizTalk deployment. -->
<property name="includeSchemas" value="true" />
<property name="includeOrchestrations" value="true" />
<property name="includeTransforms" value="true" />
<property name="includePipelines" value="true" />
<property name="includeComponents" value="true" />
<property name="includePipelineComponents" value="false" />
<property name="includeCustomFunctoids" value="false" />
<property name="includeVocabAndRules" value="false" />
<property name="includeVirtualDirectories" value="true" />
<property name="includeMessagingBindings" value="true" />
<property name="includeDeploymentTest" value="true" />
<property name="includeCustomTarget" value="true" />
<project/>
-
You now have the ability to supply a simple xml-driven Wizard-based UI when you
are deploying after an MSI install. You can use this UI to
gather Windows identity information for virtual directories (for SOAP/HTTP
receive ports), to indicate whether the scripts should deploy to the BizTalk
management database (for working in BizTalk groups), and for many other
purposes. I call this “Install-Time Configuration” in the
documentation below. Here are some example Wizard screens:
 
-
You now have the ability to have an Operations-managed
“spreadsheet-driven” mechanism for environment-relative
configuration. What do I mean? I mean those aspects of your
deployment that are particular to your development, QA, or production BizTalk
environments. Aspects that wind up appearing in your BizTalk binding
files! Queue names, database names, file shares, FTP locations,
etc. This is information that you don’t want to manage with the
Wizard UI above (there is too much of it) and you want to maintain “as a
set” for each environment. The spreadsheet (pictured below) can
start life with your developers, and ownership can gradually migrate to
Operations. This spreadsheet generates environment-specific
“settings” files, and at the point you deploy, the values for the
environment you are deploying to are automatically substituted into your
binding file…Slick!
(click)
-
You now can use a highly templated (reusable)
WiX based setup, rather than a Visual Studio Setup Project, to generate
your MSI. WiX, in a nutshell, is a set of tools and an Xml grammar that
allow you to specify the contents of an MSI. One of the clear pieces of
feedback I got on the last rev I released is that reproducing a Visual Studio
Setup Project was far too manual.
There are a large number of other changes that I’ll just enumerate
quickly here:
-
You no longer need to have your NAnt file enumerate your orchestration names or
their deployment order – a new custom NAnt task eliminates the
need for this.
-
You no longer need to have your NAnt file enumerate how many receive ports/send
ports/etc. you have – a new custom NAnt task eliminates the need
for this.
-
Support for BizTalk groups, where machines 2-n do not require deploying to the
BizTalk management database.
-
Support for not following the naming recommendations I made in previous
releases – you can have custom project names, assembly names, directory
names – its just a little more work.
-
Support for multiple assemblies of the various types (i.e. multiple
orchestration assemblies, schema assemblies, etc.)
-
Inclusion of a utility I call SetEnvUI.exe for creating the xml-based wizards
described above.
-
Inclusion of a utility I call DeployBTRules.exe for deploying BizTalk Rule
Engine policies and vocabularies.
-
Inclusion of Loren Halvorsen’s
XmlPreProcess tool for managing the environment-relative configuration
discussed above.
-
Support for creating IIS6 application pools (with specified Windows identity)
for HTTP and SOAP-based receive locations, and for adding virtual directories
to application pools.
-
Support for registering the btshttpreceive.dll ISAPI extension with IIS6.
-
Support for selectively restarting multiple BizTalk host instances, for
deploying custom functoids, and for dealing with send port groups, and
more…
There are now two downloads. The first contains all of the core scripts
and utilities plus a sample BizTalk solution that uses them. The
second download contains just the core scripts and utilities, and is designed
to allow a BizTalk solution to accept updates/bug-fixes/etc. over time (i.e.
the zip can be expanded into your directory structure on a developer
workstation so you can test it out.)
It should be noted, though, that some manual work will be required to
“upgrade” from the previous version I released in September –
this release is quite different, but I think you will find it is well worth the
time.
Full download is here.
Core scripts and utilities only are
here.
GotDotNet Workspace is
here.
The full documentation for this release (which includes more detail than this
blog entry…) is included in the zip files, but it can be viewed directly
here (with a diagram here.)
Enjoy! And remember: if you didn’t make into a given environment
(QA, production, etc.) with a scripted deployment of some kind, you
didn’t get there at all.
(Not necessarily this stuff, but something automated, at any rate…)
|