9 Ways To Bulletproof Your Boomi Processes

A Bulletproof Process
I have worked with hundreds of Dell Boomi Atomsphere processes since 2010,
and prior to that worked with a number of other data integration tools and ETL
processes. Over that time, my understanding of what causes errors in integration
processes and how to prevent them has grown. In this white paper, Ill talk about the
tools that are available to the Boomi developer to prevent and detect process errors
- the tools you need to make your Boomi Processes Bulletproof.
What are the Bullets we must proof our integration processes against? In my
experience, nine times out of ten, the Bullets are Data. Data that is wrong, missing,
new, out of range, not found in the destination system, or in some other way
unexpected. After all, the movement of data from one system to another as part of
an overall business process is why Integrations exist. And it is the variations in the
source data encountered that are the Bullets which threaten to cause the Integration
to Fail.
A Bullet-proof Integration process should handle as wide a variation in expected
source data as possible, and should fail gracefully when presented with Data that
cannot be handled. The Bullet-proof process should handle expected or anticipated
variation in data without Process-level failure. The Bullet-proof process should notify
a User who can fix the data problem when the problem is not systemic. The Bulletproof process should notify the User so that the error message is closely associated
with the record or object that failed, ideally right on the failing record itself. The
Bullet-proof process should leave a trail of breadcrumbs to allow diagnosis and
debugging, and not fail quietly or in a way that does not alert responsible users.

Techniques & Tools

The techniques and suggestions described here are tools that you can use in Boomi
integrations to:

Prevent normal variations in source data from causing the

integration to fail. An example would be when multiple addresses
in a Netsuite Contact cause multiple output records.

Notify Users about non-fatal data problems, and support the

timely correction of the source data. An example would be a
missing required field.

Support IT staff in diagnosing operational problems. An example

would be when a User calls, and asks if you can confirm that
order 123 was pushed over the integration yesterday. If the right
tracking fields are in place, you can search all of yesterdays
activity for order 123, find the execution that passed the order, and
dig into the details of the source record, or even re-run the order in
debugging mode.

These techniques make the difference between a merely adequate integration, and
a robust, maintainable part of the business process.

Error Detection
After Updates
This is the basic bulletproofing technique - your
process should look at the return codes from
any update step, and notify as appropriate. In
many connectors, you have an option to have the
connector return an error, rather than failing the
process. This is most applicable for API updates
into complex cloud applications.
For instance, Netsuite has a sophisticated
connector in Boomi that gives the process access
to standard and custom objects and fields. With
the Setting return all errors = yes, the Netsuite
connector will return a status document from any
call, with success = false when an error occurs.
You can make an xml profile to capture the return
codes from your Netsuite update call, and test the
return code in a decision step.
Then use the error messages provided in the
return document as part of your notification
message. Your process will then detect errors in
an API call, and provide useful error messages to the right people so that the error
can be corrected. It is good practice to post the error back to the source object, if
possible, to alert the End User to the problem. As most integration run-time errors
are data problems, the End User can often fix the problem if notified promptly and
clearly.

Try Catch to Trap Errors

This is a relatively new and welcome error detection and trapping component. Try-Catch
lets you go down an error branch if any subsequent process or process step fails. Before
this feature, a process error would just stop the process, and you would need
to go in and diagnose from process logs. Many update
types including the much-used Process step do not fail
gracefully, and dont allow error trapping. The Process
step is the most effective way to update database tables
by running SQL fragments that update specific fields.
Try Catch lets you detect process errors at any
downstream step, and send the failing document down
an error branch, where notifications and updates can
occur.
Try-Catch provides a document property that allows you
to capture and report the error that caused the failure.
The only downside to Try-Catch is that you lose the
details of the failing step. This is because the logs after
the Try-Catch has switched the document to Catch reflect the processing that happened
in the Catch branch, and the document details of the error branch are lost.

Smart XML Profiles

With cloud application process
endpoints like Netsuite and
Salesforce, the record layouts, or
profiles, are often complex XML
documents. Many errors occur due
to mapping issues between complex
XML documents with multiple subrecords. An example is the Netsuite
Contact profile, which can generate
multiple output documents from
a map if the contact has multiple
occurrences of the Address Book
sub- component. Use Qualifiers and
Identifiers within your XML profiles to
make sure that you map to only the
occurrence of a sub-object that you
want.
Setting up a Qualifier on the default flag of the Address Book object will insure
that you only map to a single, preferred Address in the Contact object. See the
Atomsphere reference articles about Identifiers and Qualifiers to learn about these
powerful profile tools.
Then notify the appropriate parties with information that will help them diagnose
the error situation and recover the processing. There are a few ways to do this - the
Notify step allows you to create Boomi platform alerts, warnings, and errors, and
use the Boomi platform to send these alerts.

Boomi Suggest
Many errors have their source in mapping problems. Should I map to the Status,
the Category or the Type field? Which of the several description or memo fields
should I use? Does the current date/time go into that field, or the date/time from the
incoming record?
If you are working with common profile layouts in your map, Boomi Suggest can
be very helpful in identifying and mapping common fields. It is both a time saver in
suggesting and mapping common field connections, and a quality aid in that Boomi
Suggest will identify common mappings that you might otherwise get wrong.
Boomi Suggest is statistically driven from hundreds of thousands of maps that
have been created on the Boomi Atomsphere platform. For that reason, it is
most useful working with common layouts like Salesforce object layouts. I have
found it essential in EDI mappings, where many layouts are standardized, but
documentation can be sparse and confusing. In the right situation, Boomi Suggest
can save significant time in mapping, debugging, and error handling.

Business Rules Step

Based on my informal surveys of Boomi Atomsphere processes in production, the
Business Rules step is under-utilized as an error detection approach. Which is a
shame, as it is specifically designed to make error detection easy.
The Business Rules step has several features that make it a powerful error detection
asset. You can set up multiple business rules, each with its own source data, logic,
and error message. If any one of the rules fails, the document is routed down the
failure branch, otherwise down the success branch.

The logic for each individual rule can be quite complex. Input parameters to the logic
can come from almost anywhere - the source profile, properties of all types, and
even connector calls. A special document property, found under Base, is created for
failing records. This property holds all of the error messages for each of the rules
defined in the step.
There are a few challenges in using Business rules: The dedicated UI is complex,
and not intuitive. The controls used to set up the logic are also a bit confusing. And
some gyrations are needed to pull out the error message for failing records.
But the Business Rules step does allow
you to test for multiple complex rules
at one time, use a map to extract the
errors, and alert with a customized error
message for each error. Well worth
learning to use.

Notifications

he techniques discussed above are ways of detecting errors when they

occur in your Boomi process. When an error does occur, the first thing you
want to do as part of error handling is to notify the appropriate parties with
information that will help them diagnose the error situation and recover
the processing. There are a few ways to do this - the Notify step allows you to
create Boomi platform alerts, warnings, and errors, and use the Boomi platform to
send these alerts.
More typically, your error branch creates an error notification message and uses
an email connector to email it to the appropriate party. Your email should contain
several important pieces of information: the process that failed, the ID of the record
that failed, and the nature of the failure. Use a message component to assemble the
error message, a set properties component to set addresses and subject , and an
email connector to send it out.
You can include properties
and profile fields in the
message body, just note that
the syntax used to construct
the message is a bit different
from the syntax used to
include parameters in a SQL
statement or a parameter
build.

Error Post Back

To Source
Many lower volume integrations that are part of bigger business processes use
an error post-back technique. In this approach, when an error is detected in a
document being processed, an error message is formatted with useful information
about the error and the data. Then, rather than sending an email, the error message
is posted back to the originating record or object, to a custom Integration Error
field.

The benefit here is that the error is directly associated with the source record, and
can be dealt with immediately. Notifications can also occur via email, but often a
scheduled report that lists objects with error messages in their Integration Error field
is more useful.
This technique is effective with lower volume transactions that require some human
intervention as part of the larger business process. A good example is Approved
Expenses - a manager must approve the expense report manually, and is the logical
person to notify if there is an integration error.

Enviroment Properties
The use of these properties is an advanced, but important, part of error
management. The management of parameters across environments is a
fundamental part of the Boomi architecture. Environments are part of what make
Boomi Atomsphere an enterprise-class integration development and deployment
environment.
The Dell Boomi team has given lots of thought to the development process, and it
shows - the seamless flow from build to test to production, component versioning
logs, and environment properties are three examples of this. Everyone uses
environment properties in the connectors to route transactions between test and
production data sources. But you can also use environment properties for error
handling.
A simple example is routing notification emails. You probably want to have these
routed to you during build and test, but to operational contacts during production.
In a properly structured Boomi integration process, the majority of the production
errors are data problems in the source or destination system, and these are
generally resolved by the appropriate operations people that is, Users not IT staff.
These Users are probably not getting alerts from the Boomi platform, so your
process should give them the information that they need, via email, to diagnose and
resolve the problem.
Create a meaningful notification message as described above, and route it to them
via setting an environment property in the production environment that contains the
notification list for the To: property of the email error message.
This can be a distribution group in your email environment, and by setting it up as
an Environment Property you can route error emails to you during development and
test, and to operations contacts in Production.

Tracked Fields
An advanced technique in Boomi integration is the use of tracked fields in your
processes. Most connectors have a tracking tab, which allow you to push a unique
ID into the process logs for that process. The Process Monitoring interface in the
Boomi Atmosphere environment lets you look for processes that are in error, but
also lets you search for individual documents by a tracking number, or ID. Pull
down the triangle next to Executions on the Process Reporting tab to switch over to
Documents mode, and enable search by tracked fields.
This is useful in lots of
situations - often a process
will throw an error, but it was
the prior execution that caused
the problem. Or hundreds of
documents were processed,
and you need to find a specific
one. Setting a tracking number
in the process logs lets you use
a logical or business ID to look
up information in the Boomi
logs. This is so important in
diagnosing run-time problems,
that no process is ready for
production until tracked fields
been properly set and tested.

Conclusion
Ive covered some of the most important techniques and tools.The Dell-Boomi
Atomsphere Cloud Integration Platform is deep and rich in functionality, and there
are many other tools, techniques and tips for the advanced developer.
Id like to hear about your favorite tips, as well. Feel free to send an email to
content@kitepipe.com with your techniques, and Ill include them in version two. Or
post your tip as a comment in the Blog at kitepipe.com.
As more and more enterprises move business processes to specialist cloudbased application platforms, the role of integrations will increase in volume and
importance. Its my goal to provide you with the tools to build and maintain those
integrations in robust and professional manner.

About The Author

Larry is founder and Chief Solutions Architect at KitePipe,
a specialist cloud integration services firm that helps
customers improve operations.We do this by streamlining
functions through integration and workflows, and deploying
dashboards that track the improvement in operational
metrics.

Larry is an expert Dell-Boomi Atomsphere architect, consultant, developer, and

trainer. Larry writes and speaks about Cloud Applications and Integration topics,
and leads the services team at Kitepipe. His consulting career includes work with
numerous Fortune 100 companies, and top-tier global systems integrators. He
consults with software vendors on cloud architecture and integration product and
services strategy.

About Kitepipe
At Kitepipe,We assist clients with seamless business solutions across multiple
cloud-based or on-premise applications to achieve operational gain. Services
include design, configuration and testing of integration components to achieve
seamless business coordination across applications. Our specialty is Boomi, and we
offer a range of consulting and structured services offerings for Customers who use
the Dell-Boomi Atomsphere platform.
Check out our Offers section for information about a services offering that can help
you:

Jump start your Boomi Platform implementation, eliminate costly delays, and

reduce time to integration project success;

Refactor and reformat your existing Boomi Processes to reduce maintenance

time, improve component re-use, improve performance, and minimize your

purchased connector usage;

Multiply the value of your Netsuite investment with rich integrations to

fulfillment, sales, webstore, manufacturing, or partner/customer systems.