Comments [0] -
BizTalk Insights
Comments [0] -
Wednesday, October 12, 2005
You can find my thoughts on a really simple message replay (message re-processing) technique for BizTalk in Stephen Thomas' The BizTalker newsletter. Check it out here. For additional clarification, I've created a quick Visio here.
Comments [2] -
Monday, September 26, 2005
I worked through a problem recently with a client that really took me by surprise - because I would think that many BizTalk shops would be running into this issue regularly. So! Here goes with an explanation and a solution.
We ran into this problem initially using the MSMQ (not MSMQT) adapter with BizTalk 2004. We had roughly 10 MSMQ receive locations, as well as a few send ports that were using the loopback adapter. These were all executing in the same host.
The initial symptom was that the loopback adapter appeared to not work - messages were just not getting through! They sat in the "delivered, not consumed state" for no good reason. But we quickly reproduced the problem with just MSMQ receive locations (i.e. without the loopback adapter.)
On a single processor virtual machine, the repro looked like this: Create four MSMQ receive locations, and one MSMQ send port (with the send port subscribed to one of the receive ports, just to keep things easy.) No messages will flow through the send port at all.
To repro: This download has a binding file with receive ports/locations for local (non-tx) private queues Q1-Q4, plus a send port for local private queue NONTXQ with a filter for the first receive port. There is also a bit of VB script to put a message into a queue...If you turn off one of the receive locations for Q2-Q4, you'll find things work just fine. If you don't, then (reiterating) no messages will flow through the send port.
What was the resolution? Well, with BizTalk 2004 Service Pack 1 installed, you can create a "CLR Hosting" key under the registry service definition.
In our case, we actually had to increase these values - you should determine the values you need through testing. Consider having min worker threads equal to 7x the number of MSMQ receive locations, and max worker threads equal to 10x the number of MSMQ receive locations. (More on these numbers later...)
Does the documentation address this? Good question. If you look at the topic "Managing Multiple Receive Locations" in the MSMQ adapter documentation, you will find some reference to this. It indicates you should create a "CLR Hosting" key as described above...but no actual values are mentioned (clearly just a documentation mishap.)
But why do these have to be tweaked at all? Good question. The documentation for the MSMQ adapter has some unfortunate quotables, like:
To increase performance, Microsoft BizTalk® 2004 Adapter for MSMQ is multi-threaded. If you have many receive locations, there may not be enough threads available for all the receive locations. This prevents some of the receive locations from picking up messages.
The reality is that you really shouldn't have to starve any particular receive location because of a lack of threads...you should just wind up with increased latency. But, such is not the implementation of the MSMQ adapter (at least for BizTalk 2004.)
Some background: The MSMQ adapter has a "Batch Size" parameter and a "Serial Processing" parameter that can be set per receive location. "Batch Size" determines how many messages the adapter will attempt to read from the queue (and submit to the message box) on each iteration. "Serial Processing" determines whether one thread is engaged in the peek/get/submit activity per receive location (Serial Processing = 'true') or multiple threads (Serial Processing = 'false'). If "Serial Processing" is true, the "Batch Size" is forced to one regardless of the actual setting.
So what is the execution flow for a given receive location? The internal class MsmqReceiverEndpoint is instantiated per receive location, and when it initializes, it calls ThreadPool.QueueUserWorkItem with a reference to itself. If "Serial Processing" is false...it does this exactly seven (7) times.
What does it do with the QueueUserWorkItem callback? Well, when MsmqReceiverEndpoint.ProcessWorkItem is called, it enters into a do/while loop that doesn't exit until the endpoint (receive location) becomes invalid (i.e. the receive location is shut town.) In other words, ProcessWorkItem sits on a .NET thread pool thread - and if Serial Processing is false, it sits on seven of them. The do/while loop executes a peek on the queue (with a hard-coded 10 second timeout), and if there are messages waiting, it receives up to "Batch Size" and submits them to the message box. (It will give up attempting to receive a "Batch Size" worth of messages if the 10 second timeout is reached on any attempt within the batch receive loop - i.e. if you drop a single message on a queue, and the batch size is greater than one, expect to wait 10 seconds before further activity begins...) The behavior of consuming seven threads per queue leads to the recommendation of MinWorkerThreads = 7x MSMQ receive locations provided above.
Now, I confess - I'm not a BizTalk adapter expert. But, this design seems to be in conflict with the advice offered in "Writing Effective BizTalk Server Adapters", where it says:
Don't starve the .NET thread pool: ...While starving the .NET thread pool is a risk to all asynchronous programming in .NET, it is particularly important for the BizTalk Server adapter programmer to watch out for this. It has impacted many BizTalk Server adapters: take great care not to starve the .NET thread pool. The .NET thread pool is a limited but widely shared resource. It is very easy to write code that uses one of its threads and holds onto it for ages and in so doing blocks other work items from ever being executed....If you have multiple pieces of work to do (for example copying messages out of MQSeries into BizTalk Server), you should execute one work item (one batch of messages into BizTalk Server) and simply requeue in the thread pool if there is more work to do. What ever you do, don't sit in a while loop on the thread.
Is this fixed in BizTalk 2006? Surely it is... And, in fact, it sure seems to be in Beta 1. The design of the adapter is a bit different...First, "Serial Processing" refers to whether additional messages will be received from the queue prior to the "EndBatchComplete" event being set (downstream of IBTDTCCommitConfirm.Done.) (This part of "Serial Processing" is true for BizTalk 2004 as well, along with forcing the batch size to one.) "Serial Processing" in BizTalk 2006 does not affect how many threads will be reading from your queue - you will have just one (despite what the Beta 1 docs say...), unless you have multiple host instances in play. (That one thread using a large batch size and operating with serial processing set to 'false' - not blocking on the actual message box submission - should keep up with a fairly large message arrival rate, but multiple host instances might be needed for your particular case.)
More importantly, the ProcessWorkItem implementation returns immediately after a single peek/get/submit operation (and simply calls QueueUserWorkItem again, per the advice cited above.) (Side note: There seems to be some room in the design for the idea that you in fact woudn't return immediately if more than a threshold number of messages were received, but currently this condition is "if # of messages received > BatchSize", which won't ever happen.)
So what should I do for now with BizTalk 2004? For those using the MSMQ Adapter with BizTalk 2004...consider whether you can set "Serial Processing" equal to true. Keep in mind this forces you to a batch size of 1, so this might not work depending on your message arrival rate. If you test this configuration and find an unacceptable performance loss, consider setting the MinWorkerThreads value to 7x the number of MSMQ receive locations you are maintaining, and MaxWorkerThreads to roughly 10x (to provide breathing room.) As an alternative, spread your receive locations among multiple hosts (though avoid an over-proliferation of hosts - that has its own issues.)
And never draw any conclusions until you have performance tested at load with your final host configuration - that is, your final allocation of send handlers, receive handlers, send ports, and receive locations among your hosts! Other adapters may affect the outcome if they involve polling on the receive side, or polling on the "response" side of a solit-response send port. (If they use a thread pool thread to do their work, they can be affected by any adapter that consumes threads whether they themselves are written correctly or not!) Finally, I've heard from a gentlemen who has done extensive testing that the threading parameters above are useful/necessary when using large numbers of MSMQT receive locations as well.
Never a dull day in BizTalk land...!
Comments [1] -
Friday, August 12, 2005
With BizTalk 2004, it can be quite helpful to eventually maintain binding files as "source code". After a solution has reached a certain point of stability (where port definitions are not changing often), many projects will use the Deployment Wizard to do one last export of the binding information -- and then maintain it by hand for any future changes (storing it in version control along with the rest of the solution.)
There are some interesting benefits that come along with this. One such benefit is the ability to use the XmlPreProcess tool to merge environment-specific elements into the binding file (like URIs, retry counts, etc.), using the SettingsFileGenerator.xls spreadsheet to assist -- as has been discussed on this blog before. Even if you are not using the Deployment Framework (which uses XmlPreProcess extensively), you should consider using XmlPreProcess as a standalone tool. The ability to easily maintain a matrix of logical names (for physical endpoints, etc.) versus "environment names" (development, QA, production, etc.) is a huge help. See the example spreadsheet below. (The Deployment Framework also shows how to extend use of this same spreadsheet to manage run-time configuration settings that are stored in the BizTalk SSO.)
(click)
On to a bit more advanced topic: If you have a set of port definitions that you want to conditionally deploy into a given environment, you can define a true/false value within the spreadsheet and use simple "ifdef" logic in your binding file around the port definition. For instance, you might want a particular File Send Port or Receive Location to only be active in your development and test environments. To do this, define a name such as "LogInboundPODocsToFile", and set the default value to "true" - and set it to "false" in the "production" column. Mark up your binding file accordingly. See the example spreadsheet and binding file snippet below. (When XmlPreProcess is run on this binding file, the port definition will only be included for environments where the LogInboundPODocsToFile value is true.)
(click)
<!-- ifdef ${LogInboundPODocsToFile} -->
<SendPort Name="LogSalesOrderResponse_FILE" IsStatic="true" IsTwoWay="false">
<TransmitPipeline Name="SendWithDefaultNamespaceFormat" FullyQualifiedName="SendWithDefaultNamespaceFormat, XYZCo.BizTalk.Pipelines, Version=1.0.0.0, Culture=neutral, PublicKeyToken=343bd7a15fff8d6e"
Type="2" />
<PrimaryTransport>
<Address>C:\Dev\FileLog\%MessageID%.xml</Address>
<TransportType Name="FILE" Capabilities="11" ConfigurationClsid="5e49e3a6-b4fc-4077-b44c-22f34a242fdb" />
...
</SendPort>
<!-- endif -->
Why would you want to conditionally deploy ports? Like many, I have found it useful to have an additional file-based Receive Location (associated with a Receive Port bound to an orchestration) to kick off orchestrations during development - even if the actual transport used in production will be something different. In addition, binding an orchestration to a Send Port Group allows you to have an additional file-based Send Port that will create an easy log of outbound traffic. Finally, you might create a file-based Send Port that acts as an "additional" subscriber (by Receive Port Name) to your inbound messages for an easy log of inbound traffic. (And, combined with a file-based receive port, these two mechanisms give you an easy re-processing mechanism - just drag/drop in Explorer.) But you might want all of this machinery shut off in production, hence the technique we just discussed.
Macro Recursion
Another feature within XmlPreProcess is the ability to use "macro" recursion with XmlPreProcess. This means you can define a macro (logical name) such as QueueServer (with a different value for development, QA, and production, etc.) and then define additional values in the spreadsheet that build on this such as: POAckQueue = {$QueueServer}\private$\POAckQueue. This indirection can make maintaining large numbers of endpoint URIs even easier...See the example below - where POAckQueue can now appear in the "default" column (applicable to all environments.)
(click)
Multiple Environments
Note that the SettingsFileGenerator.xls spreadsheet provided with the Deployment Framework sample (and with XmlPreProcess) has room for four environments (development, QA, staging, and production.) However, you can simply add columns to manage additional environments if need be. One such use of this would be to create a column for "unit testing", where the URIs and other binding file substitutions point to resources under the control of your unit testing framework.
More to say on binding file management in another post...
Comments [0] -
Friday, June 17, 2005
A whole raft of whitepapers for BizTalk have been released over the last several weeks - see here and here.
I completed a whitepaper a short while back (though just released) on getting the most out of the BizTalk 2004 Management Pack and Microsoft Operations Manager 2005. The paper serves as a good reference for the management pack, but I hope it also serves another useful purpose. Specifically, the "operational hand-off" phase of the software development life cycle often gets short shrift - and it can cost organizations a lot of money, downtime, and late nights. So, much of the paper discusses the importance of having a development team accurately communicate the "instrumentation surface area" of their completed efforts to an operations team.
What do I mean by "instrumentation surface area"? To start, we can consider the sum of all diagnostic logging, event logging, WMI events, performance counters (custom or built-in), and all other mechanisms your application uses to communicate its current operational & health state. Moreover, we need to capture "interpretations" of this information stream that are specific to the application. (Not just "this send port isn't working..." but "We are currently not talking to our primary shipping provider...") Finally, we need to capture suggested responses and remediation - also specific to the application.
"Communicated to the operations team"...how exactly? With a Word doc? Well, in particular, I talk about how to do this for a BizTalk-focused solution using a custom MOM Management Pack that "derives" from the Microsoft-supplied BizTalk 2004 Management Pack. Done right, this will provide the highest fidelity knowledge transfer from development to operations.
See what you think - the paper is titled "Advanced Microsoft BizTalk Server 2004 and Microsoft Operations Manager 2005 Scenarios." (What a mouthful...)
Comment on this post (if you like) with your thoughts on the paper or experience in this area...
Tuesday, May 24, 2005
As has been noted before in the lore of BizTalk, it sure would be useful to use a real debugger with orchestrations - at least occasionally.
There are times when an expression shape winds up getting a bit sticky (not that I would know...) and a debugger would be just the ticket. Other times, the exception you are getting from the orchestration engine isn't at all clear.
Still other times, seeing the actual contents of messages or context as you step through would be interesting.
Jon has posted on debugging orchestrations (in IL) with an ILDASM/ILASM loop, and had also discussed building your orchestration assemblies manually (using xsharpp.exe). (He correctly noted these approaches wouldn't generate anything supportable, but they work for spelunking.)
I wanted to suggest something a bit different - I wanted to go back instead to the venerable BTS File Dump utility that Charles released in May of 2004 (before the GenerateCSFiles registry key was discovered...) and propose a different technique for symbolic debugging.
First, install Charles' BizTalk File Dump utility - you can download it from here. Fire it up, and change the output path in the utility to something easy, like c:\temp\BTSFileDump.
Here is the Edit/Debug cycle...(Start by making sure the file dump utility is running, and click the "Start dumping files" button.)
Note that this technique will work in a production setting. You could copy the PDBs and sources to the production server, and use DbgClr.exe (in the Framework SDK) or cordbg.exe to attach to the appropriate host process.
Happy debugging...
Comments [0] -
Thursday, May 12, 2005
Comments [4] -
Andy Morrison wrote about a technique for identifying the correct host instance to attach to when debugging components associated with orchestrations.
A potentially easier solution is to keep perfmon running with the counter shown below (BizTalk:Messaging/ID Process - all instances.) The counter values will update every time a host instance recycles. The Debug-Processes dialog in Visual Studio will show you (and allow you to sort by) process IDs to make this easy.
Then put PerfMon in "report view" to easily see process IDs by host instance:
Comments [3] -
Wednesday, May 11, 2005
There have been several folks who have discussed how to create messages "from scratch" within an Orchestration context - you can read Matt's thoughts and check out a BizTalk documentation excerpt. This is a common question, and I had an additional technique I thought I would share...
Background: When you have a schema that has many promoted properties (or distinguished fields), or many elements that can be set via xpath expressions easily, it can be useful to simply start with a "template" document instance and populate the element content that you are interested in.
In this situation, you will often have a "Message Assignment" shape that looks something like this:
xmlDoc.LoadXml("<ns0:BizTalkSampleS3
xmlns:ns0="http://BizTalkSample.Schemas.BizTalkSampleS3">
<SomeElement></SomeElement></ns0:BizTalkSampleS3>");
someMsg = xmlDoc;
someMsg.SomeElement = "some content";
// (or xpath(someMsg,someElementXPath) = "some content" if we don't have a
// distinguished field.)
One disadvantage of loading up "template" xml documents from either expression shapes or code (via XmlDocument.LoadXml) is that those xml fragments can get easily "lost", and are hard to update early in the development cycle when schemas may still be in flux. Loading the template files from the file system is problematic because the question arises "where should I store these files, so that I can find them in any environment I deploy to?" (Solvable, but a hassle.)
Instead, why not embed the template xml documents as assembly resources? For those unfamiliar with that process, I have a short tutorial here (& a helper class.)
sampleUsingTemplate =
BizTalkSample.Components.EmbeddedResourceCache.
GetXmlDocResource("BizTalkSampleS3_output.xml");
// Populate the "rest" of the message with distinguished fields, promoted
// properties, xpath expressions, etc.
sampleUsingTemplate.SomeElement = "foo";
The class I have supplied will cache the loaded resources in a hashtable for performance sake, and allow you to load resources as both strings and XmlDocuments.
A last thought: Many people ask, "Why can't I just create a message using a new operator or a default constructor of some sort?" Well, because few XSD schemas sufficiently constrain the set of valid instance documents enough for that to be useful - what form would a "default message" take? (Would it have the optional elements you need? Some elements that you don't want?)
Enjoy - feedback appreciated!
Comments [6] -
Monday, March 14, 2005
Steve and I wound up working on the same problem at the same time (probably for the same person…)
When working with the MSMQ adapter, keep in mind that you must reference the Microsoft.BizTalk.Adapter.MSMQ.MsmqAdapterProperties.dll to have access to MSMQ-specific properties. (The list of properties available is in the adapter documentation.)
Why? The intellisense in the expression shape (when using the parentheses syntax on messages, ports, etc.) is looking for classes derived from Microsoft.XLANGs.BaseTypes.PropertyBase to present in the drop down list. Some of those classes are part of your "native" BizTalk installation, some are provided by add-on adapters, and (of course) some are provided by property schemas that you develop. A reference to the containing assembly is necessary to find them.
Other things to note relative to the MSMQ adapter:
Comments [2] -
Thursday, February 24, 2005
I was just reading Mike Holdorf's post on the bts_CleanupMsgbox stored procedure. The team I work with has been able to make good use of this sproc as well, especially while stress testing.
One thing to beware of is this: Your habit for doing this kind of thing might lead you to open up Sql Query Analyzer, and right click on the stored procedure - selecting “script to new window as execute.” This will generate the following:
DECLARE @RC int
DECLARE @fLeaveActSubs int
-- Set parameter values
EXEC @RC = [BizTalkMsgBoxDb].[dbo].[bts_CleanupMsgbox] @fLeaveActSubs
Ahhh, but the stored procedure actually defaults fLeaveActSubs to “1”, whereas this code will leave fLeaveActSubs as “0”. This will cause all your subscriptions to be deleted, and you will have to redeploy your BizTalk applications. Not that I would know....
So just do “exec bts_CleanupMsgbox”...
Comments [0] -
Friday, December 31, 2004
BizTalk 2004 has some very useful behavior around parallel execution and scope-level timeouts that it is helpful to have a good understanding of.
What follows is a series of experiments & associated findings that should shed some light on this area.
Experiment One:
One restriction when using the parallel shape is that if multiple branches make use of a given orchestration variable, you can get a compiler error:
error X2226: 'someVar': if shared data is updated in a parallel then all
references in every task must be in a synchronized or atomic scope
In the experiment below, we are accesing ‘someVar’ in both parallel branches - each within an expression shape that also calls Thread.Sleep (the left hand for 10 seconds, the right hand for 30.) To address the compiler error just noted, we have a scope around each usage with the ‘Synchronized’ flag on the scopes set to ‘true’.
The trace messages shown to the right of the diagram tell us that, indeed, the execution of Scope 2 and Scope 3 is completely serialized (in this case, Scope 2 completes before Scope 3 begins.)
This experiment also tells us something unrelated (but useful): that an exception will not interrupt “user code”. By “user code” we refer to code in an expression shape, i.e. not using an orchestration “intrinsic” such as a standard Delay shape or a Receive shape, etc. Notice via the timings that the exception thrown in the left-hand branch doesn’t abort the Thread.Sleep in the right-hand branch. The exception is caught only after the Thread.Sleep in the right-hand branch completes (though the final trace message in the right hand branch – ‘after 30 sec’ – does not execute.) This is important to understand if you have expression shapes in orchestrations which are making blocking calls to .NET components, DCOM servers, etc.
Experiment Two:
If we eliminate the ‘someVar’ reference in the expression shapes above, we find that the scopes do not execute serially – whether the scope synchronization flag is set to true OR false. Notice below (via the timings) that the sleep operations are executed at the same time - so it is the presence of the common variable in the expression that forces the synchronization!
We now have a 20 second gap after sleeping for 10 seconds (because the exception we throw still doesn’t interrupt us).
Experiment Three:
We would like to use a given instance of a .NET object without imposing the use of synchronized scopes. As it happens, if we have a .NET object that is pointed at by two references (i.e. someVar and someVar2 - where someVar2 was set equal to someVar with a simple assignment), the requirement that the orchestration compiler normally imposes regarding the use of a synchronized scope goes away.
In the orchestration below, the left branch is using someVar, and the right branch uses someVar2. The trace indicates that the sleep operations happen at same time (though once again the exception doesn’t interrupt the right-hand side and is caught only when the right-hand completes.)
Lesson: If you have a .NET component (that you know to be thread-safe) you wish to use in an orchestration - and you wish to use a single instance of it - you will need to have multiple references to that same instance to use in each branch of a parallel shape. (The synchronized-scope alternative is likely unacceptable!) The first variable declaration of your component might use the default constructor, while the others will have “Use Default Constructor” set to false and will be assigned to the first.
Experiment Four:
Now, there is more to be said regarding exceptions and what they will interrupt in parallel branches. If instead of using a Thread.Sleep we use a Delay shape (or a Receive shape, etc.) we find that throwing an exception in one branch of the parallel shape will indeed interrupt the other branch(es). Notice in the timings below that the Delay 30sec shape does not complete once the exception in the left-hand branch in thrown. Particularly when you are structuring real-world business flows, this is an important and quite useful behavior.
Experiment Five:
The behavior of exceptions in parallel flows is closely related to another behavior in BizTalk 2004 - that of what BizTalk is prepared to "interrupt" when a long-running scope has exceeded the timeout value that has been configured for the scope. A Delay shape (or Receive, etc.) will indeed be interrupted when the timeout expires (and a TimeoutException will be thrown). (Note that for an atomic scope, the timeout governs the maximum time to allow prior to aborting the transaction.) See the timings below and note that the Delay 30sec shape does not complete.
On the other hand, as you might expect at this point, a blocking call in an expression shape (like a Thread.Sleep or a DCOM call, etc.) will not be interrupted. However, the exception will be raised when the blocking call eventually returns:
Summary:
someVar2 = someVar1;
someVar3 = someVar1;
someVar4 = someVar1, etc.
Comments [4] -
Tuesday, December 07, 2004
I had the pleasure of presenting on BizTalk 2004 at the Heartland Developer’s Conference 2004. The whole conference was a lot of fun, and as Kent Tegels has noted, Joe Olsen did a great job in organizing the logistics for this event – you wouldn’t have known it was the first-of-a-kind (save for Joe’s mea culpa on caffeine…)
I had a chance to talk to Sam Gentile about the work he has been doing at Addesso. This was enlightening, since in my non-BizTalk-work-life I do quite a bit mobile work in the field-force-automation space (using ruggedized Pocket PC devices – great fun!)
As to the presentation itself – one of the topics I discussed was BizTalk 2004’s scaling model, and the power of having being able to create multiple host instances for a single logical host definition that contains your orchestrations. This gives you a “competing consumer” effect – multiple processes on multiple servers, all pulling from a common work queue. In my talk, I suggested that this idea wasn’t new and related it back to a 1998 article in Enterprise Developer’s magazine. In that article, an architect from Merrill Lynch was discussing the scaling difficulties in using NT4/MTS – the load balancing options weren’t pretty. He proposed a pattern he dubbed “Auctioning” where clients would submit work to an MSMQ queue, and multiple servers (all running common server-side components) would pull work from that queue. The benefits he articulated relate quite closely to what you will find in the parallel aspects of BizTalk 2004.
Check out this graphic from the article – it should look a little familiar to BizTalk folks.
Below is my paraphrase of the article’s main points, recast with BizTalk terms (and exactly relevant to BizTalk, I believe…)
You can find the whole presentation (which also discussed BizTalk compared with traditional application servers) right here.
Comments [0] -
Monday, October 11, 2004
I recently went through a really nast bout of troubleshooting with the client I currently work with, related to MSMQT. Hopefully, my tale can save you similar pain.
The core issues was this: The BizTalk MSMQT adapter can be configured during installation to integrate with Active Directory. The default is that it will not operate in this fashion, but rather in "workgroup" mode. There are (at least) two reasons why you might want to have MSMQT integrate with Active Directory: 1) you want to make use of an MSMQ router in your environment or 2) you want to use certificate-based authentication at a protocol level (where the public certificate is managed by AD.) (Note: I know this now; I didn't know it a couple weeks ago...)
We have been installing our servers in "workgroup" mode. To install in Active Directory mode requires a special permission granted by the domain administrator.
Now, when you a configure a Send Port within BizTalk and select MSMQT as the transport, the property pages in the BizTalk Explorer offer a checkbox that is labeled "Use MSMQ Authentication". If you hit the "Help" button on this dialog, the explanation that is provided is this: "Identify whether BizTalk Message Queuing uses protocol authentication every time it sends a message on this port."
As it turns out, although it isn't documented as such, a Send Port with this option checked can only work if MSMQT has been installed in Active Directory-integrated mode. If you have the "Use MSMQ Authentication" option checked on a Send Port and you are not in Active Directory-integrated mode, then messages will not flow. When we eventually discovered this discrepency and fixed our bindings files, the problem was resolved. (Note: there is a similar option when configuring Receive Locations.)
This checkbox had been checked at the point our initial binding files were exported, and became a part of our scripted deployment. What was worse, when we encountered this problem a few weeks ago in QA, we began troubleshooting the BizTalk configuration on the server directly and wound up "fixing" the problem by creating an additional Send Port (subscribing to the same traffic as the original) that simply had the MSMQ Auth checkbox off. But we didn't realize that discrepancy at the time, so we had to troubleshoot the same problem all over again a few weeks later. We definitely got ourselves into the wrong troubleshooting mindset by assuming that Biztalk was flaky in some way.
Key lesson: If you don't get into a given environment (QA, production, whatever) with your scripted deployment, then you really didn't get there at all….
A few more notes. As I said above, if you have the "Use MSMQ Authentication" option checked on a Send Port and you are not in Active Directory-integrated mode, then messages will not flow. What you will see is:
IMHO, Biztalk 2004 should be more serviceable in this regard, and should give better error information. And of course, the documentation for MSMQ Send Port configuration should have mentioned that MSMQ Authentication would only work for Active Directory.
Microsoft Premier Support became involved, and after around 18 hours of analysis they said "We see some certificate-related errors in the traces. Do you use MSMQ authentication? Are you AD-integrated?"
We looked in our binding files (since the decision had long since been forgotten) and saw this snippet:
…
<TransportType Name="MSMQT" Capabilities="16495"
ConfigurationClsid="9a7b0162-2cd5-4f61-b7eb-c40a3442a5f8"/>
<TransportTypeData><CustomProps>&
lt;Authenticated
vt="11">-1</Authenticated></CustomProps>
</TransportTypeData>
<RetryCount>3</RetryCount>
<RetryInterval>5</RetryInterval>
…
See that in the escaped XML? Yup, that is a property called "Authenticated" that is an old-fashioned Variant of type bool, where "-1" means "true".
Leaps out at you, right? Determining if you are AD-integrated means looking at
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc.3.0\MessageQueuing\MsmqtWorkgroupMode.
From all of this I (gently) conclude that the product instrumentation/tracing should point out this condition more quickly to a support engineer. In addition, the MSMQT adapter should warn you of a mismatch during configuration with the Biztalk Explorer and, ideally, when you deploy/import bindings.
Hindsight being 20/20, the support engineer should have asked to see our binding file - and should have compared it with one exported from a server that was indeed sending messages (since we had one.) Of course, we should have made such a comparison, too! (and much earlier...) The engineer did look at the Biztalk Admin Console, but of course that doesn't give any of the detailed port configuration information - only Visual Studio/BT Explorer does.
Having said that, the support engineers were great to work with and were certainly dedicated to getting to the bottom of our issue.
Key lesson: Diffing binding files will prove to be a key troubleshooting technique with Biztalk...
Comments [0] -
Wednesday, August 25, 2004
A lot has been said regarding the MSMQT adapter for BizTalk 2004 already, but below are a few recent observations that may be of help to you.
When people ask me what MSMQT is, my short answer goes something like this: "MSMQT is the name of the BizTalk 2004 adapter that implements the MSMQ network protocol directly within BizTalk. It allows BizTalk 2004 to send/receive MSMQ messages directly, and move messages to/from the MessageBox very quickly - without external (DTC) transaction coordination. Only private queues are supported for receives."
Using the MSMQT adapter you can:
You cannot using the System.Messaging MSMQ APIs/COM MSMQ APIs/MSMQ C Libraries or the MSMQ MMC console on a BizTalk 2004 machine that is running the MSMQT adapter! This is because these all rely on the native MSMQ Service running.
You can use the aforementioned APIs (though not the MMC console) from remote machines (that are not running the MSMQT adapter) to put messages in MSMQT-defined queues.
To test MSMQT queues on the BizTalk 2004 server where they are defined, you can:
Now, when we you drop a file on your directory, it will be routed to your queue. The receive location could also use HTTP posts, in which case the WFETCH tool from the IIS Resource Kit (or a similar tool) will let you easily get test messages to the queue.
Note: If you have existing applications that used to send to public MSMQ queues, but will now be sending to MSMQT queues, the most common problem is that they are referencing those queues via the PathName property, rather than the FormatName property (with 'DIRECT=OS:machine\private$\qname' syntax.) This is only a problem for users of the COM/C-Library APIs - the System.Messaging.MessageQueue class actually allows format names to be used with the Path property.
See this Microsoft Support discussion for lots of additional insight into MSMQT.
Comments [0] -
Wednesday, July 07, 2004
There are several QFEs for BizTalk 2004 that you may want to know about, in case you encounter the situations described below.
Situation: You attempt to create or edit Receive or Send ports within the BizTalk Explorer in Visual Studio, and CPU consumption hits 100%. Memory consumption climbs until the IDE crashes with an OutOfMemoryException.
Fix: Ask PSS for the hotfix associated with KB870619 (also known as hotfix 1185)
Situation: You have a CDATA section in an inbound xml document within an orchestration. The CDATA section contains flat file data, with vital trailing (or leading) whitespace. After you execute a transform in your orchestration, the CDATA designation is stripped (although the flat data is still there) - and the leading/trailing whitespace is now lost. Curiously, using the "Test Map" feature (by right-clicking on a map in the solution explorer) doesn't exhibit this behavior.
Fix: Ask PSS for the hotfix associated with KB841563
Situation: You have a scope shape, and scope timeouts aren't working as expected. Specifically, for blocking calls in an expression shape within the scope (like a DCOM call, etc.) where the timeout is being exceeded, you see 100% CPU consumption in the BizTalk service and the orchestration never terminates.
Fix: Ask PSS for the hotfix associated with KB811250. (Note: this problem may have been addressed in the fix rollup released in April - I'm not sure what the ordering was here.)
Comments [0] -
Wednesday, April 21, 2004
In the latest round of documentation that was released for BizTalk 2004 there is an "orchestration operator" defined that was not previously documented: 'succeeded()'
The documentation states that this operator can be used to determine the outcome of a transactional scope or orchestration. When might this operator be needed?
Well, it turns out that the orchestration compiler has some interesting rules about what you can do in an exception handler that might not be entirely intuitive at first (though as you reflect on analogies to C# or other exception-enabled languages, it begins to make sense.)
Suppose that you have defined a Request/Response port as the means of interacting with an orchestration, and you want to ensure that some response is generated regardless of the failure conditions you encounter. Your first attempt might look like this (I know mine did…) Stretch this JPG out to full size to see it clearly (IE will shrink it.)
This will generate a compiler error: error X2162: must receive before sending a fault message on an implemented port
What is going on here? It sure seems as if we have received a message already - we did it in the Rcv_SomeDoc shape. However, we have the Snd_ResponseDoc shape inside of Scope_WorkThatMightFault, and the orchestration compiler is assuming that we might have already executed that shape prior to the catch block executing (i.e. prior to an exception being raised.) A Request/Response port must only have one response for any given request…and our Snd_FaultDoc shape has the potential to violate this rule. It sure would be nice if X2162 could be more explanatory in this regard…
How do we overcome this? It isn't terribly obvious…We must wrap the Snd_ResponseDoc shape in an (additional) transactional scope, and check in our catch block to ensure that the associated transaction did not succeed before performing Snd_FaultDoc. See this diagram (Acrobat required).
What we are doing here is structuring the flow such that exactly one response will be sent for the original Rcv_SomeDoc shape. The way we do this is to use a Decide shape, with an expression such as "!succeeded(Transaction_SndResponse)" in the rule branch. The Snd_FaultDoc will be in the 'true' side of the branch (i.e. we did not successfully perform Snd_ResponseDoc), while the 'false' side will likely be empty.
This is a pretty subtle bit of enforcement that the orchestration compiler is performing. It is somewhat analogous to a typical language compiler ensuring that all code paths have a return value for non-void functions or methods. And, of course, even though it is not enforced, it is certainly the case that 'catch' and 'finally' blocks in standard languages often have to be aware of what has or hasn't taken place in the associated 'try' block. The orchestration compiler (apparently) just has some well-defined & strict rules it wants orchestrations to adhere to (such as "exactly one response for each request emanating from a Request/Response port".)
There is a somewhat similar case that is described briefly in the BizTalk documentation. Imagine we wish to make reference to a message or variable in our 'catch' block that was initialized within the associated scope. In this case, the orchestration compiler will assume that we might not have gotten around to initializing that variable/message prior to the exception being thrown - and a compiler error will be generated as a result: error X2109: use of unassigned local variable 'Variable_Blah'
In this case, we can wrap the portion of the scope's work that is responsible for initializing the variable/message of interest in an (additional) transactional scope (i.e. "Scope_InitWork"), and we can use a Decide shape with an expression such as "succeeded(Transaction_InitWork)" in the rule branch. This will allow the orchestration to compile…
Wow! One might start to agree with Charles' boss...
Comments [2] -
Thursday, March 04, 2004
I intend to cover some more foundational material for BizTalk 2004 in the future, but today I wanted to cover an issue that at least some people will run into fairly quickly when beginning to use the product.
There are times when it is desirable to work with multiple XML schemas that specify the same target namespace, and which specify different definitions for the same element.
For instance, you may wish to have a lax schema when a document lands on your doorstep initially - but further into the processing of that document (along a particular path) you may wish to validate against a stricter schema. Or, you may have a situation where you have what is arguably an envelope structure which can't be cleanly stripped off (for a variety of reasons) - leaving you with documents which might look quite different, but have the same target namespace and element usage.
BizTalk 2004, in general, wants to see one schema deployed to the BizTalk management database for any given combination of target namespace and element declaration. If you deploy two schemas with target namespace http://MyNamespace and element declaration MyRoot, and then attempt to receive a MyRoot-rooted document through a receive port using the default Xml Receive pipeline, you will receive an error from BizTalk like this one:
There was a failure executing the receive pipeline…Source:"XML
Disassembler"…
Reason: The disassembler cannot retrieve the document specification using this
type: "http://MyNamespace#MyRoot". Either the schema is not deployed correctly,
or more than one schema is deployed for the same message type.
To overcome this, you can use custom BizTalk Pipelines on the send and receive ports that will be dealing with schemas that are subject to the ambiguity. Within a pipeline, you can restrict the set of available schemas from "everything that is deployed" down to the schema(s) that you are interested in.
Specifically, for receive ports, you can add a new item to your project (a "Receive Pipeline"), and add the default Disassembler and Party Resolution pipeline components. For the Disassembler component, edit the "Document Schemas" collection and add the particular schema you are interested in. See this picture for an illustration.
Likewise, for send ports, you can add a "Send Pipeline" item to your project, and add the default Assembler pipeline component. Again, specify the schema you are interested in with the "Document Schemas" collection of the Assembler.
For each of the Send or Receive ports that will be trafficking in these messages, specify your newly created pipelines - instead of the default Xml Receive/Send pipelines!
Now for the gotchya! (you knew there had to be one, right?)
BizTalk 2004 will require that the assembly containing your custom pipeline(s) is deployed to the GAC (along with every other BizTalk project assembly.)
When components loaded from the GAC wish to dynamically load other types & assemblies, they must do so with fully qualified assembly names. Applying this to our current discussion means this: BizTalk pipelines must have fully qualified information for the assembly that contains the schemas you configure within pipeline components.
If the pipeline component lives in the same BizTalk project as the schemas you are attempting to reference, the property designer (when editing the "Document Schemas" collection) will only be populated with a namespace-qualified type name - the fully qualified assembly name will be missing. At run time, the schemas will not be found…and the behavior at run time will appear completely unchanged from the case where no custom pipeline was specified at all.
To work around this, simply put your pipelines in a different project/assembly than the project containing the schemas you need to reference in the designer.
A bug you say? Certainly it would be nice if the designer warned you, and it deserves a KB article soon…But keep in mind what is happening: A GAC-destined component (a pipeline Disassembler) is providing designer support which allows you to select another component (a schema) which will be loaded dynamically at run time...It raises an interesing problem that goes beyond just BizTalk 2004.
Whenever a component that is destined for the GAC has IDE designer support which in turn allows you to select a type for a "plug in" component that will be loaded by the "host" component dynamically at run time (without using Assembly.LoadFrom semantics) - you will run into this issue. Why? Because if you select a type from the same project, the fully qualified name can't be reliably known. After all, the project might not have been compiled yet, or the fully-qualified name might be set up to change with each compilation (gasp!) via 1.0.*.* versioning policy. If you use such a designer to select a type in a distinct assembly, the fully qualified name can indeed be known -and shame on the component author if the versioning policy isn't sane.
Of course, being deployed into the GAC raises all kinds of thorny issues, but this one was a bit subtle...
Comments [1] -
Scott Colestock lives, writes, and works as an independent consultant in the Twin Cities (Minneapolis, Minnesota) area.
© Copyright 2010
