Friday, January 16, 2009

Custom CRM Workflow Activities/Steps And Project Versioning

By now, many of you who have tried to make use of structured development methodologies in the development of CRM 4.0 custom workflow activities, have discovered that CRM 4.0's implementation of the Windows Workflow Foundation as its workflow system is a hindrance to you. You' aren't alone.

It appears that midstream, Microsoft wanted to turn the CRM product line into one of their "Live" applications. That would have been a nice touch, but the existing design does not adequately support that implementation unless you are happy with limiting yourself to the very small number and featureless out-of-the-box workflow steps available to you. Many of us are not, and that's where CRM's lack of staged development support can grind your efforts to a halt if you are not careful.


Contributing Factors

If you have taken it upon yourself to modify or extend the base functionality of your CRM 4.0 implementation, version each build of your project, are working within an environment which makes use of staged development, and are developing custom workflow activities, you need to prepare yourself for what lies ahead. Staged development of applications is a good practice, and in many large organizations with a strict quality program, its required. Versioning each build is also a good practice and required by many organizations. So why is it that Microsoft thinks we should develop directly on the production servers?


The Cause

Custom workflow activities have their version number embedded within the definition of the step when added to a workflow. The system makes use of this version number and the type name to locate the custom step at runtime. By adding custom activities to a workflow, you are in effect restricting your workflow to use only specific versions of activities. When you consider that differing versions of custom activities could cause you problems, this seems like a reasonable restriction. However, if the version number of your custom activities are changing with each build, even when there are no changes to the custom activities themselves, you end up invalidating all workflows you created previously. The only way to get such workflows working under the new version, is to remove and replace all custom activities with the newer versions.

If you fail to replace these activities, you will receive one of CRM's useless errors that an fault occurred while attempting to publish the workflow, and that you should attempt the publish action again. Do so only if you have time to waste. It's not like the server is going to change its mind, decide to stop playing foolish games with you, and suddenly allow you to publish the workflow. Also, don't waste your time attempting to debug your code, because the issue isn't there. In your trace logs, you will find an exception which states something like "Crm Exception: Message: Could not find a plugin using assembly name and type, ErrorCode: -2147200995". The exception leaves out a very vital condition of the type resolution process. The true cause of the error, which is that the plugin/activity could not be found using the given name, type and version.


What Microsoft Should Have Done To Avoid Workflow Development Issues

Okay, so take all of this with a grain of salt. How I feel Microsoft should have built the workflow system is based on MY many years of experience with a dedication to strong 0bject-oriented methodologies, a high level of efficiency, ease of use, reliability, and dynamic/flexible operation. You may share some of my views with the same enthusiasm while considering me "totally off my rocker" on others. My reason for this posting is to provide you (and Microsoft) with thoughts to ponder in overcoming any workflow system issues you may be having.

Is the Windows Workflow Foundation built in an object-oriented (OO) manner? Yes. Is CRM's implementation of an OO nature? Yes. The problems I see are not that they avoided OO practices, but that their implementation is on a very weak and problematic OO design. I do not feel that the manner in which CRM 4.0 implements the WWF is worth the time Microsoft has committed to its development, nor do I feel that the overall design of the WWF is what it needs to be in order to support the widest possible (and reasonable) range of applications, environments and implementations.

Is the execution of CRM's workflow system efficient? No, not at all. In fact, on a scale of 1 to 10, I personally give it a -10 -- but then I can be pretty picky here. Even with the use of reflection in some areas of my own workflow system designs (past and present) to facilitate a wide level of flexibility, I am able to execute workflow processes with a far greater number of features, and with way more error-handling and loggging support, in a fraction of the time that CRM currently requires. In some cases, I've found workflows within my current system to execute more than 2000 times faster than a closely similar implementation in CRM. In systems with a high transaction count, processes such as this need to be as efficient as possible, and I find that CRM 4.0 falls short.

Another issue regarding the manner in which workflows are managed, is that Microsoft has chosen to compile and cache them to disk.  The claim is that it is more efficient to do this.  I have found that this only adds further problems, because the management of those cached resources is managed poorly.  You might imagine as I did, that upon publishing a workflow, the cached copy would be invalidated so as to begin making use of the new version.  This is not the case.  I have seen cases in which an edited workflow is not put into use by the system for almost an hour!  Is it efficient to leave workflows in place for up to an hour which are known by the developer to be faulty?

Is the CRM workflow system easy to use? If you are using the out-of-the-box UI and provided components, it does seem to be fairly easy to develop simple workflows. Unfortunately, all of that is lost the moment that your developers must begin developing custom activities of a moderate to complex nature. This is due primarily to the rigid nature in which activities handle input parameters, the limited support of the CRM service instance provided within the workflow context object, and the manner in which the system locates and executes activity objects. Sadly, to overcome these issues, the system needs to be revamped from the ground up to support a more flexible model of workflow definition and execution. For example, instead of representing activities as strongly typed objects during the design and development phases, they should be represented by a base object which supports dynamic management of parameters, conditions, and child steps, each which support expression-based values. Such a design would allow for workflow development on systems without the installation of CRM, the Windows Workflow Foundations system, or associated activity libraries. It would also allow for a more efficient process of workflow modification in systems requiring constant availability.

Consider how activity data is recorded to disk via CRM's export process. Although the workflow header (if you will) is represented by XML, the activities (aka "steps") are represented by unreadable serialized binary data. In my design, the entire content of a workflow is in human-readable XML format, with a step being nothing more than an element describing the type of the object to instantiate at runtime, and child collection elements housing the parameters, conditions and child steps. By supporting a dynamic format, we greatly improve ease of use at all levels (design, development, operation/execution, etc.) by providing additional avenues through which to edit and build upon existing structures.

In the sample below, you can see how a step specifies the assembly and type of an object to create. Don't want to point to a specific version? Simply remove the portions of the assembly reference which are too rigid. I even go to the point of supporting file locations from which to load an assembly and its debug symbols if provided instead of an assembly name. Conditions are simple statements which support expression-based values through the use of an inline data element "slug" resolver. Parameters are simple elements which also support expression-based values. Want to add parameters? Simply add them to the XML-based workflow definition file. Because they are read in and handled dynamically, there's no need to worry with delegates which allow typographical errors, or activity upgrades invalidating your workflow. In fact, such a flexible design actually allows your activities to support internally managed versioning (determination of intended operation based on a supplied version number of format of input parameters) if you wish. (The full design of my workflow definition file format supports far more features than shown here, but the following sample should be sufficient to illustrate the simplicity which one might expect to be present in CRM's workflow export.)

<Workflow Description="A sample workflow" Name="Sample Workflow">
    <Steps>
        <Step Type="ConditionStep" Description="Determine if we should do anything.">
            <Conditions>
                <Condition LeftArgument="{#PrimaryEntity.Amount}" Operator="GreaterThanOrEqual" RightArgument="100.00" />
            </Conditions>
            <Steps>
                <Step Type="EmailStep">
                    <Parameters>
                        <Parameter Name="From">{#PrimaryEntity.createdby}</Parameter>
                        <Parameter Name="To">{#PrimaryEntity.Assignee</Parameter>
                        <Parameter Name="Template">Sample email template</Parameter>
                    </Parameters>
                </Step>
            </Steps>
        </Step>
    </Steps>
</Workflow>

Now let's pull all of these points together by speaking to the reliability of the CRM workflow system. The more you deal with the CRM workflow system and its associated components, the more you may begin to wonder if any testing was performed on the product prior to its release. At the base of any good workflow system, you should find a powerful resolver object/process which provides support for values based upon data expressions. These are commonly referred to as inline data element resolvers. CRM has one, which they call the "data slug resolver", and is responsible for resolving the content of templates (such as email templates) and workflow activity forms in a dynamic manner. I've spoken to Microsoft about the majority of the issues I've found which relate to CRM's resolver. These include the fact that it does not appear to follow any standard convention in describing values, is extrememly limiting in access to objects and properties, injects spacing which was not present in the templates, and is frankly too riddled with bugs to be relied upon. Their response was that they have seen and agree to all of those points, and that development continues to move forward. Quite a few times now, comments have been made which cause me to believe that we should just wait for CRM 5.0. Not having the time to wait, I built one to replace it (hence the difference of "{#" instead of "{!" in data elements within my sample) in three weeks which is very stable, makes use of standard dot notation, and is far more robust than theirs. If you are making use of a large number of templates and/or need them to be more precise in the generation of their content, you'll probably need to take on the similar task of building a replacement resolver. In short, the reliability is just not there, nor do I believe it wise to expect it to appear anytime soon.


Possible Workarounds

As with most problems encountered there are multiple ways to get around them, but before settling on one, take time to consider the requirements of your organization/department.

1. Turn off versioning for plugins. When I brought the issue of activity versioning to Microsoft's attention, they agreed that they had never really considered how build versioning might affect the process. Their own developers stated that they would suggest we create a proxy class whose version never changes. You could do that, and it would work if all your changes were limited to the logic within the Execute method, but what if you need to modify the parameter list or add additional custom activities? Well, now you are changing the proxy, and since its not versioned, you open yourself up to other issues (including possible dismissal from strict contracts which require total software auditing capabilities).

2. Late-bind your objects via a proxy. This makes use of Microsoft's idea of building a proxy project through which your plugins are called, yet applies a level of slightly complex intelligence which could possibly make the process work a little better. In this concept, your proxy calls upon a configuration file or configuration settings to locate, instantiate and return activities based on full names of types and a reference to their file location if not currently loaded into the application domain. The problem with this concept, is that it still does not overcome the blocking issue regarding changing parameters or the addition of new custom activities which would change the version of the proxy. It will cut down on the frequency in which you have to update workflows due to version number differences alone, but will not prevent you from having to make modifications after adding/modifying custom steps.

3. Replace the entire workflow execution process with your own. Honestly, this is the only solution that works to resolve the issue in its entirety, but unless you have experience in developing such as system, I wouldn't suggest taking on the task. I've built such systems, which are truly dynamic in nature and support build versioning and such, so we opted to go this route and commit about 4 weeks of my time to the task, but consider #4 before committing resources to workflow system design and development.

4. Build a process to simplify workflow reconstruction. This concept will allow you the flexibility of continuing workflow development, continuing activity development, and rebuilding workflows within an instant. Our problems required a more robust workflow system overall, which is why we decided to take on efforts of building a replacement workflow system, but in most cases, an automated workflow reconstruction process will be sufficient. I'll explain what is needed.


Automated Workflow Reconstruction

The idea of automated workflow reconstruction came to me quite some time ago, but since it would not resolve ALL of our roadblocks, it was dismissed for a complete workflow system replacement. If the versioning issue is the only one you have, and you cannot get around the versioning problem any other way, then by all means consider this idea. In general, you will need to build a process which reads in an existing workflow definition from the CRM system, replaces steps with your most recent versions, saves the changes, and publishes the workflow back to the system. Here's some things you will need to consider in your implementation:

1. Workflow versioning. It would not be cool to entirely lose your existing workflows due to a bug in your reconstruction process. I would suggest that you work in a version number at the end of your workflow titles, and create modified copies of workflows instead of editing and attempting to replace the current one in operation.

2. Workflow deconstruction. This processs will need to read in a workflow from the CRM system and deserialize the step data. Although a workflow is primarily a logical representation of subprocesses to execute, Microsoft chose to "compile" and hide the design of the steps by serializing the step collection and storing it in a property called uidata (user interface data). The more reasonable handling of this (given my above design for flexibility) would be to store the steps as XML.

3. Step update/replacements. After deconstructing/deserializing the activities of the workflow, you will need to go through each custom step (skipping the base steps) and ensure that it is making use of the version you will have present on your server. I would suggest a comparison of the type, name and version values, since this is how CRM looks for activity objects as well. If the step differs, your update efforts will need to check for editing/replacing the parameter definitions of the step.

4. Workflow construction. Like the deconstruction process which deserialized the steps, this step of your workflow reconstruction process will need to serialize the new step collection and place it back into the uidata property.

5. Save and publish. Once you have completed the reconstruction of the workflow and its steps, you need to save the workflow back to the system. The best manner of doing so (in light of item #1), would be to create an all new workflow with a versioned name, save the workflow definition/design to CRM, and publish the new version. If successful, you will want to immediately unpublish the previous version. Unpublishing the workflow will put it into a "Draft" state, leaving it available for you to roll back to if possible, yet causing the system to ignore its presence.


In closing...

I realize that this post does not provide you with a quick fix to your problems regarding the versioning of workflow activities. If I provided the code for all of my ideas, and explained the reasons for each action, this post would become a novel. My goal is to explain issues regarding custom activity versioning issues, the design flaws of CRM's workflow system, and some of the options you may wish to consider in overcoming them. Take time to consider which actions are most beneficial for your organization and the limitations and best practice requirements applicable to you, before setting off on your chosen course. If you are facing multiple roadblocks as I was, you may want to consider a third-party replacement product or developing your own replacement workflow process. How you go about developing or implementing such a replacement process will certainly depend on the issues you need to overcome. Good luck in your endeavors!

NOTE: The USDA currently owns the rights to my most recent workflow system. Please understand and honor the fact that unless I receive documentation from them permitting me to share its coding with others, I cannot do so.

Eric Bewley, Sr. Software Consultant
USDA/NRCS, Vistronix