Enterprise Application

Integration
Systems Integration
Methodologies
Nov/2009

Paulo Marques
Informatics Engineering Department
University of Coimbra
pmarques@dei.uc.pt
What’s this about?

Presentation Data Tier

Web Server Database Server

Network

Business Logic
App Server

2
Motivating Example
Warehouse
2 (Amazon)

7 5
1

3
How the Software Engineering world should look like…

4
The ugly reality…

Just-in-time Ad Hoc Construction

Spaghetti Code

5
Ball-of-Mud Enterprise Application Integration

6
Why Structured Enterprise Application Integration?

 Information systems are getting more and more complex!

Interconnected
Nowadays systems are not isolated: networks play a vital role! Also, new
communication channels complicate issues (e.g. WAP, SMS, iPhone)

Constantly Evolving
Today’s systems are stressed beyond their limits. People want more and
more functionality of the infrastructure in place. Thus, the size of systems
continues to increase. Centralized monolithic solutions are no longer viable:
cooperation among systems is paramount. Nevertheless, heterogeneity
complicates things (e.g. J2EE, .NET, SAP)

Adhoc IS NOT AN OPTION

- 70 to 90% of the software costs are maintenance
- 40% of the code is within the databases
- 1.000.000 lines of code is now “low medium scale”
- How to create systems so that they can be easily integrated?
- How to create software that is reusable? And at what granularity?
7
Enterprise Application Integration in Context
 Databases
 Networks How to build
 Software Engineering Applications
 Programming Languages

 Enterprise Application Integration

 Service Oriented Architecture How to integrate applications
 Enterprise Service Bus

8
Issues that need to be addressed
 Any means used for Enterprise Application Integration must address,
at least, the following issues:
 How is Data Represented – Each application has its own data model
and information schema. There may be shared concepts between
applications but, typically, they are represented in different ways. It’s
essential to have ways of mapping and transforming data.
 Localization of Services – Each application was developed as a
“software silo”. Thus, it can unexpectedly change localization. It’s
essential to have ways of localizing and remotely accessing applications
that are executing autonomously.
 Time and Synchronicity – The RPC model is synchronous. This means
that both parties have to be present when a call is made. Such constraint
is not viable when communicating among autonomous independent
systems. Typically, it’s necessary to use an asynchronous and robust
communication model between systems.
 Fault Tolerance – Networks are inherently fallible. Simplistic
approaches like re-trying operations do not work. It’s necessary to have
robust mechanisms for tolerating faults and, in many cases, distributed
transactions (ACID or not).
9
SOA

 Service Oriented Architecture

 A system is made up of autonomous entities (sub-systems)
which act independently of each other. All systems have
interfaces which were thought explicitly for system integration.

 Sub-systems have a course granularity (e.g. billing, warehouse,

purchasing, etc.), typically being capable of executing
independently of each other. These subsystems, and in particular
their integration interfaces, are called services.

 A possible way of implementing a SOA is using Web

Services and related technologies.
«An SOA is an orchestrated sequence of
messaging, transformation, routing and
processing events in which XML technologies
expose the message content and the
components that operate on the messages.»
10
SOA in a nutshell

System System System System

A B C D

Service Service Service Service

Interface Interface Interface Interface

Enterprise
Service Bus

Process Service
Orchestrator Directory

A Service Oriented
Architecture
11
Process Orchestrator?

12
Why use a SOA?

 Future, Scalability and Evolution!

 An organization is made up of many different systems.
Sooner or later they will have to talk to each other.
(typically, sooner than latter!)

 If systems are not thought from the beginning for integration, the
solutions found for doing so will be ad-hoc, without possibility
of evolution, and with high development and maintenance
costs

 Using a Service Bus and a Service Oriented Architecture allows for

future evolution, both by easily adding new services and, if
necessary, replacing existing service implementations
 A SOA allows to do so with minimal disruption of the business
processes of an organization.

13
IMPORTANT CONCEPT: Evolving Systems!

14
Progressive Deployment

Enterprise
Service Bus

Phase 1 Phase 2 Phase 3

15
The importance of Open Standards

 It’s absolutely important that a SOA is based on Open

Standards
 It allows the architecture to stay in place on the long-run
 Allows external services to be easily integrated into the
organization’s infrastructure
 Allows for the substitution of existing systems by better ones (e.g.
better technology, with a lower cost of ownership, or that are
more adequate to the organization’s strategy)

 Technologies like XML, HTTP, SOAP play a vital role

«It allows applications to bind to services

that evolve and improve over time without
requiring modification to the applications
that consume them»
16
Some Challenges

Naïve Implementations use Web

Services for Everything!

Web Services are Sexy!

17
Problems associated to the Client/Server model

 Synchronous interaction which forces both parties to be

present at the same time
 Normally blocking: who calls is forced to wait for the
server processing to be over
 Maintaining sessions is “expensive” in terms of resources,
limiting the number of simultaneous clients
 Use of connection-pools
 Use of thread-pools
 Use of Asynchronous I/O

18
Faults affecting the synchronous model (Client/Server)
 If a fault occurs, the session context
is lost
 Re-synchronizing systems can be
EXTREMELY complicated!
 Especially so when nested calls are
performed.
 Some examples of faults:
 If a fault occurs before (1), “having
has actually happen”.
 If the fault takes place between (1)
and (2) – server fault – the request
is lost
 If the fault takes place between (2)
Possible Solution:
and (3) – subsystems fault – data 2PC
inconsistencies may occur
 If the fault happens after (3),
corresponding to a lost response, But it’s so
the client will retry the operation SLOW
19
Motivating Example

widraw(5000, #4598)

(Application Server) (DB Server)

“Transfer 5000€ from the

CGD account to the MG
account”

(Application Server) (BD Server)

(DB

20
How to address this?

The server cashes after having

taken the money out, but before
the money has been deposited.
What now?

(DB Server)

deposit(5000, #7653)
“Transfer 5000€ from the
CGD account to the MG
account”

(Application Server) (BD Server)

(DB

21
Bank Transfer using a Two-Phase Commit

Bank A Bank B

Start
Transaction

Withdraw from
acct. A

Transfer

Credit on
account B

Two-Phase Commit

22
A more realistic solution
Bank A Bank B

First transaction

Start
Transaction

Withdraw from Transactional Message

acct. A Queue

Write Msg
Start
Commit Transaction

Read Msg

Credit on
account B

Commit
Second Transaction
23
SOA Revisited

System System System System

A B C D

Service Service Service Service

Interface Interface Interface Interface

Enterprise
Service Bus

Process Service
Orchestrator Directory

24
Things we haven’t talked about

 Data representation and transformation

 How to design business processes

 How to design for security in large scale systems

 How to design for fault tolerance and

how to deploy systems in practice

 What middleware options are available

 How to integrate with business partners

(and related standards)
25
Conclusion

 Large Scale Enterprise Application Integration is much

more than simply writing network code

 Service Oriented Architecture (SOA) is critical for

designing systems that:
 Are maintainable in the future with low costs
 Can adapt to the new business needs of organizations
 Are reliable and easy to use

 Creating an SOA is much more than simply calling web

services scattered throughout the organization (or inter-
organizations)
 Beware of the hype!

26
Questions?

27