Friday, June 19, 2020

OIC: Identity Propagation In Structured Process

When a process calls a service you sometimes have a requirement that some user identity needs to be propagated to the service call. This article describes how you can propagate the identity (but alas not the principle) of a user on behalf of whom a service call is executed.

Updated on June 22 to add a missing, last step.

When calling a service in a structured process you sometimes must pass on the identity of the user that called the service. This could be the case when that service call is done to a SaaS application and it is required to track on behalf of whom that service is called. The identity (user name) only is not enough when authentication must happen using the principle (security token) of the user, but there are applications that can handle this using some combination of a system user (or client id plus secret) with an on behalf of user. And there are situations where having an on behalf of user only is enough, like when storing data in a database table with audit columns (you don’t want all the end users also to be database users so passing on the user’s principle to the DB would not make sense).

It is not always trivial who that on behalf of user should be. Take for example the following process model:


  • On the top a flow implementing the 4-eye principle. The first user does something after which the second user reviews it. Which identity should be passed on, that of the first or the second user?
  • Below that a flow where the user activity is optional. When that user activity is not executed, what identity should be passed on?
  • After that an exception flow to handle a fault with calling the service. This is handled by some Applications Administrator that may choose to retry. Which identity should be passed on, that of the Process Owner or the administrator?
There is no good or bad answer, it depends. You can decide to keep it simple though and choose for always using the identity of the last user that executed a Human Task no matter where in the process that is. And if there is none, then use the identity of the “system user” that started the process instance.

Now how to achieve this? In short:
  • In the Start event of the process set some “user” payload element to the creator of the process.
  • Whenever a Human Task is finished, replace that with the id of the user that did that.
  • When the service is called, pass it on as a custom header element.
Putting the id as plain text in the header is not exactly 100% hack-proof but spoofing it is also not trivial. Considering that calls are done over SSL it just might be secure enough.

In Detail

I work with one single process payload data object as much as possible. Let’s call that processPayload. You can add an element to that called onBehalfOfuser. In the Start event you can instantiate that with the creator predefined variable:



In the output mapping of every Human Task map the execData.systemAttributes.updatedBy.id to the processPayload.onBehalfOfUser:



In case of an OIC Integration (or ORDS REST Handler) you can pass that on as a custom header element. In case of an Integration you configure that in the trigger activity.


In case of an ORDS REST Handler you can configure that as an HTTP header parameter:



When calling the service, you map the payload.onBehalfOfUser to the header parameter, which either will have the value of the creator of the process, or that user id of the last Human Task that has been executed.


Finally, to make the Integration backward compatible you can use an choose-when-otherwise construction to default the identity with the invokedBy meta data element:





Friday, May 29, 2020

OIC: How to Find Human Task by Correlation and How to Abort a Parallel Task


This article explains how you can find an instance of a Human Task of process instance in the Oracle Integration Cloud (OIC) without knowing its task number, and how you can use that for example to withdraw a parallel task. You can use the same mechanism for other use cases as well, like to get a specific task instance for a specific process instance in a custom Workspace, etc.

When a Human Task is scheduled in a process there is no out-of-the-box way for the process instance to “know” its task number, because scheduling a task concerns an asynchronous call (so you don’t get an immediate response with the task number). So, although the task number is visible in the process flow trace (as shown below), the process instance itself does not know it.


The sample process below has two parallel tasks. The outcome of Parallel Task 2 is either CONTINUE or DONE. When DONE, Parallel Task 1 must be withdrawn. I will use this as a use case to illustrate how setting some “correlation id” on a task can be used to achieve that. 


For those of you who remember the on-premise BPM Suite may be aware of the Update Task activity you could use to do just that, but such an activity does not exist in OIC. Instead you will have to use the PUT operation of the /ic/api/process/v1/tasks/{id} API. But as you can see this requires an id, which is the task number that you don’t have. To get that you can use the GET operation on /ic/api/process/v1/tasks first. However, there can be many instances of the same process, implying many instances of the Parallel Task 1, so how to find that one you are looking for?

The clue is that you must map some unique key to the task when scheduling Parallel Task 1 so that you can use the GET tasks operation to query tasks on that key. The GET supports a keyword query parameter to do this. You can include the unique key in the task title (which is indexed and used in the search by keyword), but that clutters the task title with some technical id.

Fortunately, Human Tasks have a specific request element you can use to do just that, which is the identificationKey. In the example below I a have mapped some idPrefix that is unique for the process instance, and post-fixed that with “-1” to make the identificationKey unique over all Human Tasks instances.


Now in the Get Task activity I use the GET operation to search tasks by keyword equal to my unique key:


The response is a list of tasks with an “items” element that will contain the (one) task instance I’m interested in. I created the Tasks Business Type for the response of the API. To get the instance of the task I will have to map the “items” of that response to an array of items first, for which I created the TaskItems Business Type, and then map the first item in the array to a Task Business Type I created. So, Tasks -> Task Items -> Task as shown below:


The mapping from the response of the API to my TaskItems is done using a Transformation. As I’m only interested in the task number I have left all other elements out of the TaskItems, except for the state:


To map first element of the TaskItems to a variable of my Task Business Types I also use a Transformation:



Now I have a hold on the task number so I can call the PUT operation to withdraw the task in the Withdraw Task activity:


When I start an instance of this process, and go the Workspace I will see both tasks:


I can then submit the instance of Parallel Task 2 by choosing DONE.



And as a result, the instance of Parallel Task 1 is withdrawn and the process ends:


In a real process application, I would have made this a bit more robust and would check if the Get Tasks returns a task instance and that the state of that task is ASSIGNED. I probably would even create a (reusable) Integration to withdraw any task from any process by identificationKey. But you get the draft.

Sunday, February 09, 2020

OCI: When and How to Create an Integration to Call a Service from a Process?

With the Oracle Integration Cloud, when you have to call a service from a Process you can choose to call an external service directly or you can put an Integration in between. This article gives some directives why you may want to do the latter, and how to prevent a pitfall that is easy to step in to.

To call a service you have to import the WSDL with it's XSD's. With that Business Types are auto-generated for all complexTypes in that XSD. Recently I was refactoring a case where this resulted in some 220 (!) Business Types being generated from 1 single external service, of which only a few were actually used. Granted, it concerned a service with a very complex interface, but for some reason all the external SOAP services we have to consume are moderate to very complex and easily generate 50+ Business Types. Not only that, they also use relatively long namespaces. Can you imagine what will happen when you have to call 5 of these services from the same Process application! You barely can see the forest from the trees, and you may find it pretty difficult to identify the correct Business Type to use for your request.

The following example shows the selection list showing the types to chose from when creating a data object for one of the most simple cases we have.

Another issue might be that your Process is tightly coupled to the external service. When it's interface changes you will have to re-import the WSDL. There is a risk that it will not be back-ward compatible which may result in a situation that you first have to completely remove the existing WSDL with all its resources, but can't do so until you have to removed all references to it from data objects (variables) that were defined using those Business Types. That can become quite a challenge.

All this can to a great length be prevented by creating an Integration as a proxy between the external service and the Process application. But you will solve nothing when you use the same WSDL for both the Invoke as well as the Trigger Connection. You must take the extra step to create another WSDL with an XSD for the trigger connection, which offers an as much simplified interface as possible to be consumed by the process. This is nothing less than an example of applying the basic principle of decoupling.

What I personally do is, starting from the parent element of the request and response type, copying the complex types I need from the XSD of the target service, snip all elements I don't need from it, simplify the data types (e.g. change normalizedString to string) and remove all attributes I don't need, or replace those that I do by an element instead. And so work my way down to the child elements as far as required. And then create a Trigger Connection based on the simplified WSDL to be used for the Integration that will be consumed by the Process, as shown in the following picture.


In the example of the 220 Business Types I mentioned before, I was able to reduce that to just 22 Business Types.

Sunday, December 01, 2019

OIC Integration: defining and using constants

Oracle Integration Cloud does not have native support for constants, but it is easy to set this up. This post discusses how.

For integrations there are two ways to define constants in the Oracle Integration Cloud:
  • Lookups
  • Variables

Lookups are initially meant to support mapping of values from one domain to another. For example, one domain has country code using two letters ("NL") whereas another domain uses three letters ("NLD"). The lookup can then be used to "translate" the value from one to the other ("NL" <-> "NLD"). The same feature can also be used to support configurable constants by providing a list of name-value pairs. For example, in the following SMColor lookup two different name-value pairs have been stored, "YELLOW" with value "yellow", and "BLUE" with value "blue":






In an integration you can use this lookup to get the value by name using an XPath lookup function. As I will show hereafter, there are two different XPath expressions, each being used in a different context.

Variables are set in an Integration using the Assign activity. You can define a variable with a specific name and a value. You can also use a combination of the two. In the following example you see the variable "red" being defined with value "red" and the variable "blue" is initiated using an XPath expression that on its turn gets the value from the lookup name-value pair with name "BLUE":




Now let's have a look at how this can be used in a Mapper activity. In the following picture you see three different elements being mapped from respectively the lookup, the variable and the combination of both:




The response of my service looks as follows:

{
  "statusFromLookup": "yellow",
  "statusFromVariable": "red",
  "statusFromBoth": "blue"
}

Note that the XPath expression used to initiate the variable, is different from the Mapper activity. The difference of expression is whether the lookup function is used from XSLT as the Mapper activity does, or not.

When used in a Mapper activity use an expression like the following:

dvm:lookupValue ("tenant/resources/dvms/SMColors", "Name", "YELLOW", "Value", "not found") 

Otherwise use an expression like this:

dvm:lookupValue('oramds:/apps/ICS/DVM/SMColors.dvm','Name','BLUE','Value','not found')

Now when to use what? Some pointers:
  • To change a variable in an integration, you will have to modify the integration and reactivate it. You can do that as a new, minor version to prevent downtime. Not a task for any type of OIC user.
  • The threshold for someone to change a Lookup is lower. More suitable for values that need to be changed run-time (you don't have to re-activate the using integrations) for example by an Application Administrator.
  • It is easy to make a mistake in the XPath expression. So when you have to use it in an integration multiple times, consider using the combination as mapping a variable is simple.
  • Execution of an XPath expression has a small, but still a performance penalty.

For Structured and Dynamic Process there also is more than one way, which I will also discuss in some next blog topic. None of the above solutions support "versioned" parameters. I will discuss how to do that as well.

Sunday, November 17, 2019

OIC Process: Auto-Mapping Elements in the Data Mapper

Hereafter I describe a 'hidden feature' regarding data mapping in Oracle Integration Cloud - Process.

When mapping data in the Oracle Integration Cloud (or OIC for short) you sometimes discover that elements you want to map from are not always available as a source on the left-hand side. As I recently found out (thank you Eduardo Chiocconi!) that does not necessarily mean that they are not available for mapping.

An example might be including some elements of the request in the title of the process instance. Until now I always did this by including a Data Mapper right after the Start Event. However, the same I could have achieved in the Start Activity itself.

The following figure shows how I map the value of the "title" predefined variable to itself, concatenated with some values from the request (customer name and id):



As you can see in the Processes tab of the Workspace both the title and the concatenated values are visible. Saves you Data Mapper activity :-)


Monday, October 21, 2019

Microprocess Considerations

In this article I discuss some considerations when applying the Microprocess Architecture, and how those can impact the design of the process.

This article has been updated on November 11 2019 after feedback from Luc Gorissen, and on December 28 after feedback from Sushil Shukla.

As pointed out in the article about the Microprocess Architecture, one should carefully consider if this is the right architecture for the process application to create. Considering the implications (for example one single business process can end up comprising multiple process applications) it should not be considered to be a "one-size fits all" kind of architecture.

Guidelines that can help you to determine if and where is a fit, are the following:
  1. Do process instances have a longer time span during which one must be able to incorporate changes to the process (in one way or another)?
  2. Is the process expected to change often?
  3. Does it concern a complex business process, where business functions can be executed isolated from each other (and with that potentially can be reused)?
  4. Are multiple business units involved in the flow?

If you answered one or more of these questions with yes, it probably is a good candidate. If not, probably not. As I will discuss hereafter you do not have to implement the Microprocess Architecture on all parts of the application. There are also alternative solutions like abandoning a running instance and handle it manually, or restarting a process instance. Such alternatives are out-of-scope of this article though. Maybe I will discuss them at some point in the future.

Before explaining the rationale behind these criteria, let me first explain that instance migration refers to moving a running stance of a process from one version of the application to the next. For this to be possible, the next version needs to be backward compatible with the one in which the instance runs. At the time of writing of this article, The Oracle Integration Cloud (OIC) does not yet support instance migration, but will soon. But even when it does, there will be limitations. As it is yet to be seen what those are, I can say not much about them but you can imagine that an instance which is in a Receive activity (waiting to be called) cannot be migrated to a version from which that activity has been removed.

Now lets discuss the criteria that makes (part of) your application a candidate for the Microprocess Architecture in more detail.

Point 1 is a clear indicator, as you cannot assume instances can always be migrated to the new version that has the changes incorporated. An example is a long-running legal processes that has to cater for changing procedures and laws. Or a move house process initiated by the customer some time before the move will actually happen, and in the meantime the organization or customer situation may have change changed, requiring the move house to be handled differently. As long as the top-level process is not changed in a non-backward compatible way, the applying the Microprocess Architecture may support this at a great length.

Point 2 might be less obvious unless you start thinking about what it means when you have changed the process and there are instances running in previous version that cannot be migrated. You should try to avoid having multiple versions (or revisions as you would call them) of the same process application being deployed, but may be forced so. This will have impact on the process engine, not only from a performance but also operations perspective. Someone who has to analyze the flow of the process will have to be aware there can be many revisions for (part of the application) that all work differently.

Point 3 addresses the level of functional modularization that can be achieved. Often it is already a pretty natural way of development to implement isolated business functions in modules, or in the context of this article, microprocesses of their own that also can be maintained and deployed isolated from each other. An example is some generic omni-channel notification feature to inform customer about the status of something like an order, service request, or complaint. Another example is a reusable process to handle technical faults. The microprocesses can be reused, but there still is a top-level process that determines the orchestration or choreography of the microprocesses. In case of a Dynamic Process, the business rules determining the choreography can be also be implemented as modules of their own that (currently) only can be changed dynamically as long as the rules are data-driven, and the interface of those rules do not change. All in all the Microprocess Architecture mainly addresses flexibility to the microprocesses, and less to the top-level process. One therefore should strive to have as little business logic in the (long-running) top-level process as possible, and delegate this to the microprocesses and rules.

Point 4 is a very clear indicator. Whenever parts of a business process are executed by different business units it always will be a good idea to group business functions by business units in such a way that they are isolated from each other, resulting in microprocesses of their own. Obvious rationale that a business unit can then execute its own roadmap for changes of the process without unnecessary interference of the roadmaps of other business units.

Now let's discuss how this can be applied to a process application or parts of it.

When a process starts there may be quite a few activities that are executed before it has to pause for a longer period of time. Or said differently, before it reaches a human activity that may take days or weeks before it will be executed, or a point where it has to wait for a message from some external application, organization of organizational unit (in BPM-speak these activities would be in a different pool). The implementation of the activities up to that point may not require to be microprocesses of their own. After all, once started any change to the process cannot be applied to those activities anymore, they will already be done. In contrast, for all activities coming after that you still have an opportunity to execute those differently.  In other words, any point where the process can be paused for a longer period of time should be considered to be the end of a microprocess, and the first activity after that to be the start of a new one.

An activity may represent a business capability, that may consist of several smaller but strongly related steps. It would be wise to isolate these from the rest of the process, so that this set can be maintained and and operated separately. This then will be a microprocess of its own, or even a microservice.

Finally, as argued, changes in the top-level process will impose a challenge at some point. To some extend this can be addressed by letting the cross-over from one phase to the other be the start of a new microprocess. For example, the first phase may concern the sales cycle to a customer. The customer may need some time to consider the offer. Once the product has been sold the delivery process can start. This can be a good opportunity to start a new microprocess, implying a split of the main process into two separate ones, a "Sales" and a "Delivery" process.

Sunday, September 15, 2019

The Process Group Pattern


This article describes the Process Group Pattern, which can be used to correlate process or integration instances that all support the same business process. It is also one of patterns supporting the Microprocess Architecture.

Updated on 2019-09-16 to include screenshot of processes in Workspace.

For a somewhat more complex process, and especially when applying the MicroprocessArchitecture, you may have more than one process and probably several integration applications that make up the implementation of one single business process. This implies that when executing a business process there will be 2 or more instances of the process, and integration applications. Now how can a business user or Applications Administrator correlate all these instances to monitor the flow of the business process?



The on-premise Oracle BPM Suite (and SOA Suite) has the concept of "flowId" which is an id that is generated by the BPM engine at the start of the first instance and then "passed on" from one instance to the other. Using Enterprise Manager, by means of the flowId one can easily follow how one process or integration calls the other, and by putting it in the process or integration instance title, also in the Workspace. The Oracle Integration Cloud (OIC) does not have the concept of flowId, as least not yet. Now what to do? Here comes the Process Group Instance to the rescue.

The Process Group Instance Pattern is relatively simple. It includes a unique "processGroupId" that, like the flowId, is generated at the start of a process flow and then passed on from one process or integration to the other. A robust way to get a processGroupId is by using the GUID you can get from OIC (or the BPM Suite).

However, unlike the flowId, the processGroupId is unique over all engine instances you may have. When using a GUID, it is even unique over the world. Uniqueness over engine instances becomes important when at any point in time you have two or more of those and when some of the components of the process application are deployed on a different instance than the one starting it. Also unlike the flowId, the processGroupId is persisted in a custom database and can be kept persisted beyond the life cycle of the business process (which after purging will have disappeared from the engine's database). Finally, you can return the processGroupId as a response to the start operation of the process, allowing the starting application storing a cross-reference to the Process Group instance.


To support that instances can be queried by processGroupId in the Workspace, you can set the title of the instance as a first activity in every individual process application. For OIC integrations you can make it one of the tracking variables. The below shows how in OIC this would look for process instances:


Next to the processGroupId, you can also persist more meta data information about the business process, ending up with a business object looking like the following:
 

The combination processGroupType and businessId should be unique, at least for running instances. By storing this combination together with the processGroupId you have a mechanism to prevent duplicate instances of the same business process to be started.

For an even more complex business proceess consisting of a main and a few functional subprocesses, you can introduce an extra layer by adding a Process Instance business object. This might come handy for example when you have a stand-alone, reusable business process (like a Signing process) that is called by the main process.


Assuming that a functional process application is deployed on only one process engine at the same time, the processInstanceId might be filled out with the technical id of the first instance as generated by the engine.

More formally:

Context:
The implementation of a business process is made up of several components (process or integration applications) and there is no out-of-the-box way to relate the instances of these components to each other in a (custom) Workspace. One might also want to have a recording persisted of meta data of the business process after its instance(s) have been purged from the engine's database. Or one might need a means to prevent duplicate instances of the same business process to be started.

Solution:
A Process Group is the collection of instances of components that make up the flow of one single business process. It includes a unique processGroupId that is generated at the very beginning of the business process, which is persisted in a custom database, together with a combination of a businessId and processGroupType. There can only be one combination of businessId and processGroupType for any running Process Group instance at any time. The processGroupId can be returned by the start operation to support cross-reference from the starting application. 

For more complex process applications an extra Process Instance layer can be added as a child of the Process Group, to support business process applications made up of two or more functional process applications. A Process Instance is the collection of one or more instances of tightly coupled components that together make up one single process application, which in principle is reusable.

Implication:
A custom database is required for storing the Process Group and Process Instance. A function is required that returns a processGroupId that is guaranteed unique over process engine instances when (at some point in the future) components of the business process need to be deployed on two or more engines.