trace of thought (Scott Colestock) - Deployment Framework for BizTalk 2006...

Friday, June 30, 2006

Deployment Framework for BizTalk 2006...

(Update: The original download files were missing the PDB-to-GAC functionality I've discussed before. Please download again if you have already...)

The BizTalk Deployment Framework has been updated to work with BizTalk 2006...It is hard to believe that this project has been going on since May of 2004!

The Deployment Framework for BizTalk 2006 still has the same goals as the 2004 version:

Streamline the deployment process for developers, ensuring repeatability
Make it easier for a team of BizTalk developers to stay in sync - not just with BizTalk artifacts, but with all the other infrastructure that surrounds the solution (virtual directories, queues, file folders, etc.)
Extremely close parity between server-side deployments and developer deployments - so you are always testing your server deployments as you go through a typical developer work day.

BizTalk 2006 itself introduced quite a few features to make deployment easier, and can work fine for small (or solo) projects. Here are a few limitations I've encountered:

Though much improved, it is still possible to get into "dependency chaos" - where you spend time manually undeploying/deploying individual artifacts.
Binding changes have to be communicated "manually" between developers on a team, since the import process is done through the Admin MMC or command-line. Just "getting latest" from version control isn't sufficient.
Binding files for your various environments have to be maintained as separate files (manually), rather than "merging in" environment-specific settings
Artifacts such as rules, virtual directories, queues, folders, additional dependent assemblies, etc. can be represented as "Resources" within the Admin MMC (and in exported MSIs), but not in a fashion that easily moves between developers on a team.

So! To get started with this version, download the Deployment Framework (Tools) zip and run the MakeBizTalkExternalTools_VS2005.vbs script. This will add entries to the Visual Studio tools menu for deploying and undeploying using the framework. You can download the full sample to see the framework in action (first build it, then do Tools-BizTalk Deploy. You'll see something like this.)

The high-level approach is the same as the 2004 framework - you supply a small project-specific NAnt script that indicates via properties what elements of a deployment your solution requires. You include BizTalkDeploymentInclude.nant to get all the core deployment functionality, and make sure the DeployTools directory is copied to your project. (Unzipping the Deployment Framework Core into your project is a good way to do this.) See the documentation for a more complete discussion.

The primary difference in the upgraded framework for BizTalk 2006 is that we now create a BizTalk Application definition, and use BTSTask to add all BizTalk artifacts as resources within that application. Starting and stopping the application is done at an application level rather than per port/per orchestration.

Packaging up your solution as an MSI can be done with the WiX-based scripts that have been in the framework for awhile, or by using the MSI export mechanism in BizTalk 2006, if you prefer. (The latter solution will require a few additional steps, and doesn't give you the parity described earlier. But it works if you need to go that route.)

Other Notes:

NAnt .85 RC4 (and NAntContrib ) is required. Be sure to copy the new BizTalk.NAnt.Tasks.dll to NAnt's bin directory. We need to call BizTalk's .net 2.0 assemblies, and NAnt wasn't built against 2.0 -- so change nant.exe.config to have only<supportedRuntime version="v2.0.50727" /> in the<startup> element.
Log4net usage in the sample (which isn't required of course for the framework) has been updated to log4net 1.2.10 , as has my serializable wrapper. You can find the new log4net.Ext.Serializable in the Tools download (which is useful all by itself, apart from the framework.)
You can actually use the BizTalkDeploymentInclude.nant file in this release with BizTalk 2004, if you like, to aid in your migration. There is NAnt property to indicate 2004 vs. 2006.
Scan all the previous release notes...

Leave comments with any questions/issues/etc. Enjoy!

Download: Full Sample, Framework Core, Tools Source , Docs

Friday, June 30, 2006 2:46:41 PM (Central Standard Time, UTC-06:00)

Comments [23] -
Deployment Framework

Wednesday, October 04, 2006 6:36:14 AM (Central Standard Time, UTC-06:00)

Working with BizTalk 2006. Receiving the following error in the deployAndStartPorts. Would appreciate some comments on how to proceed. BTW - BizTalk Deployment Framework is very cool -- thanks for bringing it to life.

deployAndStartPorts:

[exec] Starting 'BTSTask.exe (AddResource -Type:BizTalkBinding -Source:"837_Processing.PortBindings.xml" -ApplicationName:837_Processing)' in 'C:\Documents and Settings\Administrator\My Documents\Visual Studio 2005\Projects\837_Processing'
[exec] Microsoft (R) BizTalk Application Deployment Utility Version 3.5.1602.0
[exec] Copyright (c) 2006 Microsoft Corporation. All rights reserved.
[exec] Information: Adding resource (-Type="System.BizTalk:BizTalkBinding" -Luid="837_Processing.PortBindings.xml") to application "837_Processing"...
[exec] Information: Validating resources (count=1)...
[exec] * Validating resource (-Type="System.BizTalk:BizTalkBinding" -Luid="837_Processing.PortBindings.xml")...
[exec] Information: Performing change requests...
[exec] Information: Calling BeginTypeChangeRequest for all selected resource types...
[exec] PerformingBeginChangeRequest
[exec] Adding resource (-Type="System.BizTalk:BizTalkBinding" -Luid="837_Processing.PortBindings.xml") to store.
[exec] Information: * Performing change request on type "System.BizTalk:BizTalkBinding" (count=1)...
[exec] Information: Commit the change requests...
[exec] * Performing EndTypeChangeRequest for resource type "System.BizTalk:BizTalkBinding".
[exec] Command succeeded with 0 errors, 0 warnings.
[exec] Starting 'BTSTask.exe (ImportBindings -Source:"837_Processing.PortBindings.xml" -ApplicationName:837_Processing)' in 'C:\Documents and Settings\Administrator\My Documents\Visual Studio 2005\Projects\837_Processing'
[exec] Microsoft (R) BizTalk Application Deployment Utility Version 3.5.1602.0
[exec] Copyright (c) 2006 Microsoft Corporation. All rights reserved.
[exec] Information: Importing bindings "837_Processing.PortBindings.xml" into application "837_Processing" in BizTalk configuration database (server="SOGETIDEV1", database="BizTalkMgmtDb")...
[exec] Error: Failed to update binding information.
[exec] Invalid or malformed XML data.
[exec] Command failed with 1 errors, 0 warnings.

David Wanner

Friday, October 06, 2006 12:42:22 PM (Central Standard Time, UTC-06:00)

Hey Dave

I just had that problem yesterday. If you don't need it, try setting the applyXmlEscape to false.

Also Scott - I noticed that the updateOrchestrations task doesn't check the boolean values of includeOrchestrations, schemas, etc...

Paul Baker

Monday, October 09, 2006 7:33:44 AM (Central Standard Time, UTC-06:00)

As I mentioned to Dave, to understand the applyXmlEscape switch, you need to read http://www.traceofthought.net/PermaLink,guid,640d3834-9574-4eb1-bca2-e38b78284e63.aspx

This switch is about being able to store your binding files - in particular the TransportTypeData element - in an unescaped form for easier readability and easier use with XmlPreProcess.

Paul - absolutely right on updateOrchestrations. I'll add that to the list.

Scott Colestock

Monday, October 16, 2006 10:15:07 AM (Central Standard Time, UTC-06:00)

We are currently having an issue with a BizTalk 2004 production deployment that we are moving to BizTalk 2006. This application uses Log4Net and the Serializable Log4Net wrapper. The application has been running in production for a year with no issues. When we were testing the 2006 port we noticed a strange problem. After sending a couple messages through the system, the BizTalk process starts gradually consuming more and more Threads. This contiues even when no messages are being processed until the BizTalk process is restarted.

I have narrowed it down to the Log4Net logging. The problem doesn't happen when I remove the logging. I have alredy tested with the latest Log4Net and Serializable wrapper assemblies. Any idea on what might be causing this. We are also using the registry configuration watcher.

Thanks,
Charles

Charles Kastner

Monday, October 16, 2006 10:37:47 AM (Central Standard Time, UTC-06:00)

What are you using for your log4net appender? Have you tried changing it to something simpler to see if that isolates the problem?

(I've seen these components run under stress with 2006 without issue, fwiw)

Scott Colestock

Friday, October 27, 2006 12:54:26 AM (Central Standard Time, UTC-06:00)

I also have a problem with deploying the BizTalkSample on Biztalk 2006, also in the deployAndStartPorts , but a different error message : Failed to update binding information.
Does it look familiar to anybody?
I attach the entire text for this part of the script.

deployAndStartPorts:

[exec] Starting 'BTSTask.exe (AddResource -Type:BizTalkBinding -Source:"BizTalkSample.PortBindings.xml" -ApplicationName:BizTalkSample)' in 'C:\Work\BizTalk\BizTalkSample'
[exec] Microsoft (R) BizTalk Application Deployment Utility Version 3.5.1602.0
[exec] Copyright (c) 2006 Microsoft Corporation. All rights reserved.
[exec] Information: Adding resource (-Type="System.BizTalk:BizTalkBinding" -Luid="BizTalkSample.PortBindings.xml") to application "BizTalkSample"...
[exec] Information: Validating resources (count=1)...
[exec] * Validating resource (-Type="System.BizTalk:BizTalkBinding" -Luid="BizTalkSample.PortBindings.xml")...
[exec] Information: Performing change requests...
[exec] Information: Calling BeginTypeChangeRequest for all selected resource types...
[exec] PerformingBeginChangeRequest
[exec] Adding resource (-Type="System.BizTalk:BizTalkBinding" -Luid="BizTalkSample.PortBindings.xml") to store.
[exec] Information: * Performing change request on type "System.BizTalk:BizTalkBinding" (count=1)...
[exec] Information: Commit the change requests...
[exec] * Performing EndTypeChangeRequest for resource type "System.BizTalk:BizTalkBinding".
[exec] Command succeeded with 0 errors, 0 warnings.
[exec] Starting 'BTSTask.exe (ImportBindings -Source:"BizTalkSample.PortBindings.xml" -ApplicationName:BizTalkSample)' in 'C:\Work\BizTalk\BizTalkSample'
[exec] Microsoft (R) BizTalk Application Deployment Utility Version 3.5.1602.0
[exec] Copyright (c) 2006 Microsoft Corporation. All rights reserved.
[exec] Information: Importing bindings "BizTalkSample.PortBindings.xml" into application "BizTalkSample" in BizTalk configuration database (server="MTEXBIZTALK2006", database="BizTalkMgmtDb")...
[exec] Error: Failed to update binding information.
[exec] Cannot update send port "BizTalkSample_SampleS2_File".
[exec] Cannot update transport information (address "C:\temp\BizTalkSample_OutDir\%MessageID%.xml").
[exec] The following items could not be matched up to hosts due to name and/or trust level mismatches:
[exec] Item: 'FILE' Host: 'BizTalkServerApplication' Trust level: 'Untrusted'
[exec] You must do one of the following:
[exec] 1) Create hosts with these names and trust levels and try again
[exec] 2) Re-export the MSI without the binding files and have a post import script apply a suitable binding file.
[exec] Command failed with 1 errors, 0 warnings.

thanks!

Simona Achim

Friday, October 27, 2006 3:03:45 AM (Central Standard Time, UTC-06:00)

I finally solved the problem!
It was not such big deal , but it gave me some headache (I am a Biztalk greenhorn)
I had to change the property "trusted" of BiztakApplicationServer to the value "true" in the files "BizTalkSample.PortBindings.xml" and "BizTalkSample.PortBindingsMaster.xml"
tnx

Simona Achim

Tuesday, June 05, 2007 2:35:27 PM (Central Standard Time, UTC-06:00)

I have a DLL that has several orchestrations. However, when I deploy, I want to pick and choose which Orchestrations actually get started.

Do you have any suggestions on how to accomplish this? I looked at the "startApplication" target but the Mode is set to "StartAll" and the other choices for Mode are only DeployAllPolicies, EnableAllReceiveLocations, StartAll, StartAllOrchestrations, StartAllSendPortGroups, StartAllSendPorts and StartReferencedApplications.

Thanks for your help.

Rodney

Tuesday, June 05, 2007 2:47:29 PM (Central Standard Time, UTC-06:00)

Rodney - you could consider going back to the biztalk2004 style of deployment, and use the controlOrchestrations task in the biztalk.nant.tasks.dll - you would be looking at the OrchestrationsToIgnore property on that task.

or, you could implement a customPostDeployTarget and turn back off the orchestrations you wanted to (potenitally using controlorchestrations task)

Scott Colestock

Wednesday, June 06, 2007 2:32:52 PM (Central Standard Time, UTC-06:00)

Here is our situation and I'm looking for suggestions regarding deployment with your tools. We have had a lot of success w/ your BT Tools and Biz 2004. Now we are using 2006 but our deployment got a little more complicated. But, without further due, let me get to the point.

We have anywhere between 5 to 8 BizTalk applications running on the server. Standalone BizTalk application (i.e. it’s not and/or does not reference other BizTalk applications) with its pipelines, orchestrations, maps, schemas etc. is very well documented. Things change when we try to deploy hierarchy of BizTalk apps. App_A runs in its own Host_A and is referenced by App_B running in Host_B, which is then in turn referenced by App_C running in Host_C.

Of course, if you change something in App_C then you’d have to undeploy. App_B, App_A, first, do your work, recompile App_C,App_B,App_A and redeploy in the same sequence. We would like to streamline development/deployment cycle to the point that its deployment would be completely transparent do the developer. If he/she modifies either of A, B or C then the framework would take care of what needs to be undeployed/recompiled etc.

What would happen if there was another application, App_D, which was referenced by App_B but not by App_A or App_C? Is it possible to put App_A, App_B and App_C in a sigle solution? Where would App_D go? Of course, they would further be subdivided into orchs, schemas etc.

Or is it better to have all the Apps in their own solutions, with their own .build files and that is where one would implement undeploy/recompile/deploy logic to take care of any dependencies?

Can this be done and if so then how? I don’t necessarily need hand holding and detailed instructions. Just suggestions or an algorithm for solving this would suffice. For example; start with this Nant task, then use Biztalk tools for this, etc. I can follow standalone deployments documentation very well and I’m sure I could implement nested scenario but I don’t know enough about Bttools and/or Nant to comfortably start with this.

Art

Thursday, June 07, 2007 10:29:52 AM (Central Standard Time, UTC-06:00)

Art - before I reply in detail, can you tell me what kinds of sharing you are doing between apps? (What causes you to have app-level references?)

Scott Colestock

Friday, June 08, 2007 9:33:41 AM (Central Standard Time, UTC-06:00)

Scott,
We have defined separate HOST for hosting our individual BT Apps. Some BT apps are standalone [which are not our concern] but others do reference one another [these are topic of this discussion]. Most, if not all, orchestrations w/in their individual Apps are exposed as WS
but they also can, and are, called by other orchestrations via Call Orchestration shape and that’s what I mean when I say “referenced”; just like the “old” COM world where UI.dll would call BS.dll, which would call DA.dll, etc.

Here is typical usage scenario [out or order for clarity’s sake]:

App1.Orch1 exposed as Ws to our external clients
App1.Orch1.SomeMsgType = App2.Orch1 [via Call Orchestration shape]
App2.Orch1.SomeOtherMsgType = App3.Orch2 [via WS]

There are others but I’m sure you already see what I’m taking about. Now, if you would want to deploy something like this then, and we are not even talking about maps, pipelines, schemas, etc. w/in individual Apps as we are only concerned with larger context of App1 thru AppX, you’d have to undeploy App1, App2, App3, do your work and deploy App3, App2, App1. All this on the top of regular BT tasks w/in individual Apps.

Some decisions were made prior to my arrival on this project [I hear for performance reasons] to partition BT solution into individual Apps and use Call Orchestration shape to gain OO functionality so to speak. It doesn’t matter, I have to work with what I have and I’d like to help them with their, and my as well, deployments. I’m pretty sure that thy have done it for a reason. Win and Web based clients make calls to these orchestrations on as needed basis. It’s quite common for either one of them to go directly to, say, App3.Orch4.MsgType for results.

Our current situation is that App is a BT solution but without any regard for these references. We have to remember what references what and automate deployments w/in App but not among Apps. This, in theory, can grow to arbitrary complexity.

Art

Tuesday, June 12, 2007 8:24:45 AM (Central Standard Time, UTC-06:00)

Art - What sort of framework support would you like to see for dealing with application references?

Other users of the framework have added a task to BizTalk.NAnt.Tasks.dll to deal with application references. (After the deployAppDefinition target, they call this new task - that simply uses a BtsCatalogExplorer instance to look up the application and call the AddReference method using the name of the app they need to reference.)

Would this give you what you are after ?

Scott Colestock

Tuesday, June 12, 2007 8:51:52 AM (Central Standard Time, UTC-06:00)

Scott,
This could quite possibly work for me. Right now, for example, when I deploy App1 (which is on the top of the food chain. i.e. one, which is not referenced) which references App2 AND App2 is not deployed then the deployment will fail complaining that App2, which is referenced by App1, is not deployed, etc., etc.
If the dependency tree goes App3->App2->App1 and if you change something in orchestration App3.Orch1 then App2 and App1 will also have to recomipled and redeployed. Our main goal here is to help developers, use something like updateOrchestrations task, which is the framework already. It takes currently about 15 min. to build and deploy our BizTalk Apps to any of the servers. Not only we would like to streamline the deployment on development machines to bare minimmum (most of the time developers are working on orchestrations, then schemas, maps etc) but also be able to automate deployment of these dreaded dependencies somehow.
I'm spending more time with NAnt and I think that's where I may try to implement my solution. How about this:
1. Deployment framework is wrapped around each application (App3, App2, App1)
2. These apps have no knowledge of one another and "think" they are deployed as standalone,
3. I create wrapper targes in NAnt to link the deployment logic accounting for dependencies.

If Deploying.App3 then
Wrapper(Deploy.app3, deploy.app2, deploy.app1).deploy
elseif Deploying.App2 then
Wrapper(deploy.app2, deploy.app1).deploy
else
deploy.app1 //no need for wrapper here

Can I something like this in NAnt? Thanks for your help!

Art

Wednesday, June 13, 2007 8:18:23 AM (Central Standard Time, UTC-06:00)

Art -

If I'm tracking you correctly, then I think your plan could work out...

I guess I would also say to minimize dependencies between apps whenever possible. Give some thought to this article prior to sharing schemas: http://msdn2.microsoft.com/en-us/arcjournal/bb245674.aspx

Also, give some thought to whether you can create targets that simply re-gac assemblies in the case where the changes are minor (to avoid cascading redeploys) - see http://www.masteringbiztalk.com/blogs/jon/PermaLink,guid,628aecb9-4eaa-4044-88c9-92cb53f81bdf.aspx

Scott Colestock

Friday, July 06, 2007 1:51:00 PM (Central Standard Time, UTC-06:00)

Hi,
I am using your deployment framework for automating the deploy of my application.
I am not clear about some points:
1:What is the use of portbindingMaster.xml
2:I have an orchestration that is being called by other orchestration's and a Email send port(SMTP).Do I need to do any particular configuration changes for Call orch and SMTP port.
My Binding file shows all the details.

I am able to build my solution , deploy it,Add Resources,Add Receive and Send port.The framework is throwing an exception when it tries to start the Application.

the exact error message is:
devlSSO:

[exec] Starting 'C:\build_projects\Continuous\BizTalkPOC\BizTalk\DeployTools\SSOSettingsFileImport.exe (AvidXchange.AvidSuite.BizTalk /settingsFile:"C:\build_projects\Continuous\BizTalkPOC\BizTalk\EnvironmentSettings\Local_settings.xml" /userGroupName:"Development\BizTalk Application Users" /adminGroupName:"Development\BizTalk Server Administrators")' in 'C:\build_projects\Continuous\BizTalkPOC\BizTalk'
[exec] Application already exists.
[exec] Settings file was associated with application in SSO.

startApplication:

[controlbiztalkapp] Starting BizTalk application...
[controlbiztalkapp]

BUILD FAILED

INTERNAL ERROR

Microsoft.BizTalk.ExplorerOM.BtsException: Could not enlist orchestration Process.Exceptions.ExceptionHandler,Process.Exceptions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=012ec4ffc6d9b1f2'. Could not enlist orchestration 'Process.Exceptions.ExceptionHandler'. All orchestration ports must be bound and the host must be set.
at Microsoft.BizTalk.ExplorerOM.BtsCatalogExplorer.SaveChangesWithTransaction(Object transactionObj)
at Microsoft.BizTalk.ExplorerOM.BtsCatalogExplorer.SaveChanges()
at BizTalk.NAnt.Tasks.ControlBizTalkApp.ExecuteTask()
at NAnt.Core.Task.Execute()
at NAnt.Core.Target.Execute()
at NAnt.Core.Project.Execute(String targetName, Boolean forceDependencies)
at NAnt.Core.Tasks.CallTask.ExecuteTask()
at NAnt.Core.Task.Execute()
at NAnt.Core.Target.Execute()
at NAnt.Core.Project.Execute(String targetName, Boolean forceDependencies)
at NAnt.Core.Project.Execute()
at NAnt.Core.Project.Run()

The exceptionHandler orchestraion mentioned is the orch which is being called by others and also has the SMTP port.
Is this the reason why it is not getting enlisted and port's not getting binded to it.
Please respond as I am stuck as to how to proceed.

Regards
RD

Rachana

Monday, July 09, 2007 5:33:50 AM (Central Standard Time, UTC-06:00)

Hi ,

I need to deploy BAM solution.
Can I use NAnt for the same.
If yes, could you please provide some input.

Regards
Vishnu

Vishnu Kumar Tiwari

Tuesday, July 10, 2007 7:35:14 AM (Central Standard Time, UTC-06:00)

Hi,
I tried to ImportBindings in exec task using BTSTask ImportBindings command .. my script uses four binding files to be imported. After importing 2 binding files the BTSTask hangs and never returns and so NAnt script. However I can see the success message of second import. After successful import cpu usage and memory uses of BTSTask stabilizes.

Atin Jain

Tuesday, July 10, 2007 3:14:30 PM (Central Standard Time, UTC-06:00)

Atin - I've seen that behavior (the hang) when there is something not quite right about your bindings file. Unfortunately, btstask should probably do more validation prior to starting the import. Take a look at any hand-edits you might have made.

Scott Colestock

Wednesday, July 11, 2007 3:12:56 AM (Central Standard Time, UTC-06:00)

Hi Scott,

Thanks for your instant reply.
Theres no hand-edits in binding files. All files are generated by export bindings wizard. Before using BTSTask we used to import bindings using BTSDeploy. But BTSDeploy used to hang on adding assemblies because size of assembly was large(wmiprvse memory limitation) but never gave problem in importing bindings. If I use BTS Admin Console, it successfully imports binding. If I restart biztalk server and sqlserver then sometimes it imports correctly using script. Is there a limitation of binding file size?

Regards
Atin

Atin Jain

Wednesday, July 11, 2007 7:38:36 AM (Central Standard Time, UTC-06:00)

Atin - I'm not sure what to say, except that the symptoms you're talking about are somewhat consistent with a malformed binding file. If your binding file is good, but very large, perhaps you want to consider increasing your DTC timeout? (just a thought...)

Scott Colestock

Thursday, July 26, 2007 8:50:30 PM (Central Standard Time, UTC-06:00)

Scott -- Thanks for sharing a wonderful set of tools. We've found them quite useful. I've noted that there's no licensing information on this site (only a copyright notice) or with the downloaded copy of the tools. Without a license, I'm not quite sure in what situations I can use these tools. Can you share your intentions with respect to licensing these tools? I'm sure our legal team would appreciate a formal license included with the binary distribution, but I'd be happy with your statement of intended use.

(Also, FYI, I think you're obligated to include a copy of NAnt license with the binaries you're redistributing from your site).

Thanks,
Pat.

Patrick Conant

Thursday, August 09, 2007 4:00:31 PM (Central Standard Time, UTC-06:00)

I'm using deployment framework for BizTalk 2006. When I create MSI file, it shows BUILD SUCCEEDED. The redist folder also contains everything I'm looking for. But when I install msi, it does not install everying.
I actually created a TestFolder and added TestProject.dll. I can see them in redist but they didn't get install when I install the msi

Consultant

Comments are closed.