This great post from Tomas on receiving ack/nack notifications from the MSMQ[C] adapter prompted me to write up something I had researched a few weeks back...

But first, a bit of background...

When doing work with message queueing systems, a very common (and well supported) convention exists when two systems are exchanging messages: System 'A' sends a message to System 'B'.  When System 'B' wishes to reply, it uses the message identifier of the 'request' message as the correlation identifier of the 'reply' message.  The reply queue used is either well known by both parties, or is communicated as a property on the request message.  For purposes of discussion, let's call this the "queueing contract". 

Both MSMQ and MQSeries have MessageId and CorrelationId properties on messages.  When looking at messages in a reply queue, System 'A' can peek at the messages, looking for a particular CorrelationId prior to issuing a receive.  Alternatively, it might use the CorrelationId as a look-up into working state of some sort.  To facilitate request-response interactions, MQSeries has had a ReceiveByCorrelationId API for a long while, whereas MSMQ added this in 3.0 (that is, on Windows XP and Windows 2003.) (Actual property and method names may vary in real life...)

There are, of course, many deployed systems in the wild that implement the queueing contract - using both MSMQ and MQSeries.  You may find yourself needing to integrate with these.  At first blush, it seems that consuming a service such as this should be a breeze from within BizTalk orchestrations - after all, accomplishing correlation is a first-class-citizen feature for BizTalk, so how tough can this be?

Correlation in BizTalk is indeed a first-class citizen...for data within your documents (or promoted message context.)  The classic example is to define a property schema that embodies a business identifier like "PONumber", and then visit all schemas that will be used across a set of message interactions - promoting the appropriate field in each as a "PONumber".  Then, a correlation type and correlation set are defined which reference that property, and Send and Receive shapes are set to either initialize or follow the correlation as appropriate.

The problem with an orchestration consuming the queuing contract we described earlier is that this contract occurs at the transport layer.  To be a consumer of the queueing contract, an orchestration would like to send a message, and initialize a correlation based on the MessageId...but the MessageId is produced further downstream, at the point an actual MSMQ or MQSeries message is generated by an adapter.  Moreover, an orchestration would subsequently like to receive a message, following an initialized correlation - but the CorrelationId isn't promoted by the MSMQ adapter (though it is for the MQSeries adapter...)

Solving for the MSMQ Adapter

The MSMQ Adapter gives us a small amount of help to get past this...We can in fact do our initial send operation through a solicit-response port (rather than a one-way port) and get back the MessageId that was generated by the adapter -- it actually comes back in a single-element document with a tag named "MSMQMsgId".  Starting here, the approach I took to solving the whole of the problem looked like this:

1. Send a message through a MSMQ solicit-response port.  The response operation in the orchestration designer can be of type "String".
2. On the response half of the solicit-response port, execute a pipeline & pipeline-component that extracts the returned MessageId and promotes it as the CorrelationId (using the standard MSMQ adapter namespace for that property.)  (For tracing purposes & useability, the MessageId is also returned in a <string> wrapper by the pipeline component rather than the native <MSMQMsgId> wrapper.)
3. When the initial message (containing the correlation id in both content and context!) is received back into the orchestration, it is sent back out to initialize a correlation set (typed as MSMQ.CorrelationId.)  This Send shape is just a "dummy send"...it exists just to initialize the correlation, because that is the only means given to us in the orchestration world.  I used Patrick Wellink's "NOPE" Adapter for this purpose, but there would be other options as well (including Tomas' Null Send Adapter, which I just recently rediscovered and which apparently performs better under load.)
4. Next, we receive the "real" response message - following the correlation that was just initialized.  Because MSMQ.CorrelationId is not promoted by default, the response is brought through a second pipeline component that performs that service for us.

See the diagram below (and the downloadable sample) for more detail. 

(click)

What can go wrong with this approach?  Well...there is race condition that can occur if the "real" response message (step 4 above) arrives before steps 2 and 3 can execute and establish the correlation (and associated instance subscription.)  In other words, if the "real" response message comes back before the orchestration has had a chance to express interest in that particular message, a routing failure will occur.  This might be highly unlikely for your scenario (given response times of the service you are interacting with) but it is something to test for, nonetheless.  (With BizTalk 2006's ability to route failed messages, you could potentially catch and retry in this scenario...)

(Note - if you are going to rely on the technique in the sample, consider asking Microsoft support for BizTalk 2004 QFE 1647, which addresses an issue with truncated correlation identifiers being returned to BizTalk...)

Solving for the MQSeries Adapter

Consuming the queueing contract with the MQSeries adapter for BizTalk can be done with a technique quite close to the one just discussed - but with a little less work.  The response to the initial solicit-response interaction returns with MQMD_MsgID already copied to a  MQSeries.BizTalk_CorrelationId property.  Likewise, the when receiving the "real" response message, MQMD_CorrelationId is copied to the MQSeries.BizTalk_CorrelationId property.  This means all correlation can be done with this one property - no pipelines/pipeline-components are needed!

From the documentation:

(click)

(The "dummy send" - though not pictured - would be required here as well to initialize the correlation set.)

In addition, there is a much cleaner solution that the BizTalk 2004 MQSeries adapter provides.  It turns out that MQSeries allows the "caller" to assign the message ID (unlike MSMQ), so within BizTalk, you can in fact set MQMD_MsgID and MQMD_CorrelationId to the same value prior to doing your initial send.  Now, you can initialize a correlation set based on MQMD_CorrelationId with the first outgoing message, and follow the correlation for the "real" response message.  Slick!

From the documentation:

(click)

Solving for the MQSeries Adapter in BizTalk 2006

Ahhhh, here we have what looks like real simplicity...There is a new feature in the BizTalk 2006 MQSeries adapter termed "Dynamic Receive" (referenced in the BizTalk 2006 Adapter Enhancements document.)  For a solicit-response port, Dynamic Receive allows context properties on the outbound message to determine which server, queue manager, and queue should be "watched" for the response message.  This is used in conjunction with a "match" option - which allows you to wait for a particular message based on CorrelationID (or any of several other properties, such as MessageID, GroupID, SequenceNumber, Offset, or MsgToken...)

Not only does this give us quite a bit of flexiblity for where response messages should be found (we don't need a fixed receive location anymore) but it also elminates the whole of the problem of implementing the "queueing" contract within orchestrations.  With this feature in place, an orchestration simply uses one solicit-response port, and the work is done.

Needless to say, it would be fantastic to get similar support from the MSMQ Adapter (at least the ability to have a single solicit-response port with a similar "match" criteria option, even without the dynamic receive location.)  But I don't think that is in the cards for BizTalk 2006...

Notes on the download:

• See the BTMSMQCorrelation.PortBindings.xml file for the names of the two queues and two file directories that must be created for this sample.
• Install Patrick Wellink's "NOPE" Adapter or choose a different strategy for dummy sends (like Tomas' Null Send Adapter.)
• Use the included Deployment Framework-based deployment, or deploy manually.
• To run, launch the "TestQueueServer" console app that will implement the queueing contract (request/response)
• The sample will import and run just fine in BizTalk 2006, though you will have to deploy manually.  (Still working on the 2006 version of the deployment framework - more later...)

Hopefully this is helpful to those doing queueing work with BizTalk - enjoy!