Architect Data Services
for ESB
If your services share interdependent information, you need both a DSA and an ESB to ensure consistent data.
by Christopher Keene

Service-oriented architecture (SOA) provides a way for companies to assemble applications out of free-standing units of functional code. As long as these services each have their own, nonoverlapping data, a messaging layer such as an enterprise service bus (ESB) is sufficient. However, when services share interdependent underlying data—such as inventory or customer data—a data services architecture (DSA) is required in addition to the ESB to ensure that each service is working with consistent information.

For example, a company might have a Web portal that integrates services to perform order entry, check order status, and update order status. The implementation of these individual services might be provided by a packaged application such as SAP or by custom applications written in Java or C#. To function properly, however, each of these three services must agree on inventory availability and customer account information.

The role of a DSA is to guarantee that data used by custom-written application services is always up-to-date with the information managed by packaged applications such as SAP. This capability is complementary to the ESB—an ESB is concerned with application integration and workflows between services, whereas a DSA is concerned with data integration and transactions occurring within a service.

This article describes best-practice data architectures that enable companies to extend the ESB architecture beyond application integration and provide a framework for developing new custom applications. A DSA abstracts data from both traditional databases and existing packaged or legacy applications and uses data synchronization to ensure that data used by custom applications is always up-to-date (see Figure 1).

Operational Data Consistency
Gartner defines an ESB as a messaging infrastructure that connects and unifies interactions between services in an SOA. In other words, the ESB enables "plug and play" communications between SOA services. Yet an ESB by itself does nothing to ensure that data interdependencies between services are handled properly.

A traditional application is usually deployed as a "silo," meaning the application has its own database that contains a copy of operational business data needed by that application such as customers, products, and inventory levels. Typically, each data silo gets synchronized only periodically, so between synchronizations each silo has slightly different data. These data inconsistencies can cause business errors when silo applications are integrated using an ESB.

When silo applications are turned into services, each with different inventory data, you encounter problems (see Figure 2). In this example, the "show status" service thinks that the inventory level is 27, while the "check_avail" service thinks that the inventory level is 0.

Given the business risks associated with inconsistent data across multiple data silos, the first step for an IT department implementing an SOA is to ensure data consistency. Developers can’t build custom, service-oriented applications without first implementing a DSA that guarantees data consistency across all services.

Ad Hoc Data Consistency
The classic approach to distributed data management in the world of silo architectures is to extract reference data into a dedicated operational data store for each business application. This approach depends on little interaction between silos, so out-of-date data within a particular silo is unlikely to cause problems. You need a DSA to ensure that each of the services being integrated into a common workflow by the ESB is working with up-to-date information (see Tables A, 1, and 2).

You must specifically design the DSA to provide consistent, high-performing, and highly available data across an SOA. The ESB provides a functional abstraction layer for applications and data, while the DSA provides a data abstraction for both applications and data (see Figure 3). So, an application such as SAP looks like a set of function calls to the ESB, and like a set of data definitions (transformed according to a metamodel) to the DSA.

The key capabilities of the DSA include:

• Graphical metamodeling tools for defining the object-relational mapping and model-driven, iterative code generation to accelerate development.
• Built-in intelligent cache that understands the object model and schema for high performance.
• Transparent database optimizations for all major databases for native performance and vendor independence.
• Cache clustering for scalability and high availability.
• Ability to synchronize changes occurring within an underlying database or application up into the DSA caching layer.
• Ability to create custom data abstraction adaptors for package and legacy applications that allows them to be accessed and updated as though they were databases.
• Support for major development platforms, such as BEA WebLogic, IBM WebSphere, and Microsoft .NET application servers.
• Support for major deployment platforms, such as Windows, Unix, and Linux.

A best practices example is NetJets, a provider of leased jet services. NetJets owners include both companies and private individuals who are willing to pay a premium for the convenience of personal jet service. Well-known NetJets customers include Tiger Woods and Arnold Schwarzenegger.

NetJets must deliver an aircraft almost anywhere in the world within a few hours of receiving an owner’s call. If a NetJets aircraft is not available, the company must charter a jet starting at around $6,000 per hour. This creates a tight linkage between the scheduling efficiency of NetJets’ IT systems and the profitability of the business.

To accommodate its rapid growth, NetJets undertook a large-scale SOA effort, called IntelliJet 2. The NetJets IntelliJet 2 is a comprehensive flight-management system that monitors and tracks every flight while also maintaining owner profile data for all NetJets owners. Its functional scope includes every area of the business.

In aviation, a single data consistency error can have enormous safety and financial consequences. To ensure rock-solid data consistency, NetJets invested heavily in a DSA that delivers immediate response to the end user, scales to meet the needs of the business, and helps ensure the system is available at all times.

The DSA provides caching, optimized updates, distributed cache synchronization, load balancing, failover, and client notification. The flexibility of this architecture has allowed NetJets to introduce new services more quickly than its competitors, further increasing its competitive advantage in the marketplace.

ESB + DSA = Successful SOA
The ESB connects and coordinates interactions between application services. Yet, for an ESB to be effective, the common operational business data underlying each service must also be coordinated for consistency. For example, if the order entry and warehouse shipping services have out-of-date inventory information, using an ESB to connect the two services could lead to intermittent business errors.

A DSA provides a highly flexible framework for building new custom services you can connect through the ESB. Here are the areas of complementary functionality:

• Key components of an ESB include messaging, message transformation, and intelligent routing. The key components of a DSA include data modeling, data transformation, and transactional data synchronization.
• Important benefits of an ESB include integrating existing applications, enabling seamless introduction of new applications, and allowing IT to respond quickly to changing business needs. The DSA supports all of these benefits by ensuring the consistency and availability of data underlying the services integrated by an ESB.
• An ESB typically provides XML transformation of documents, enabling the output of one service, such as a PeopleSoft purchase order, to be transformed into the appropriate input format for another service, such as an SAP purchase request. The DSA provides data transformation at the SQL level, ensuring that custom applications can access consistent shared information.

An ESB provides a way for companies to assemble applications out of free-standing units of functional code and assume that each service has its own nonoverlapping data. However, when services share interdependent underlying data, a DSA is required in addition to the ESB to ensure that each service is working with consistent information.

About the Author
Christopher Keene is vice president of data caching for ObjectStore. He has appeared as a featured expert on realtime data convergence in numerous publications ranging from The Wall Street Journal to InformationWeek, and at conferences ranging from Comdex to the Enterprise Architect Summit.