Make Application Integration Easy
Limited reusability, multiple data formats, tight coupling, and many other problems have hampered easy integration, but SOA can change all that
by Rag Ramanathan

August 20, 2004

Application integration is one of the most critical issues faced by enterprise information technology managers. Enterprises use many custom-built and off-the-shelf packaged applications to run their business processes. These applications must be integrated to share information and to create new applications.

Traditional application development and integration approaches have been inflexible and have not been standards-based; therefore, they have not facilitated an agile enterprise IT environment that can support the changing needs of a dynamic organization.

Development and Integration
In large enterprises, applications must interact with business data from one or more sources, some of which may be other applications. In other words, applications cannot be developed without integration.

On the other hand, application integration requires certain application development tasks, such as developing and assembling components, connecting them to back-end systems, implementing process flows and work flows, developing user interfaces, and testing and debugging. It is no longer useful to consider application development and application integration separately.

Application development and integration methodologies have evolved to allow IT to keep pace with the rapid growth and change of businesses. However, the rate of technology change lags behind the rate of business change. In most enterprises, the IT department's lack of agility impedes the enterprise from changing according to customers' needs.

Three of the most common application integration methodologies were point-to-point integration, enterprise message bus or middleware integration (known as EAI), and business process-based integration.

In point-to-point (or peer-to-peer) integration methodology, applications integrate with other applications as needed (see Figure 1). The interconnections shown could also be built with Web services. That would not make this an SOA-based system, because it lacks other characteristics such as loose coupling, an intermediary base, and shared infrastructure.

Because point-to-point integration was complex, costly, and difficult to maintain, another approach was introduced. This approach was called enterprise application integration (EAI), which was based on a message bus/broker or middleware. In this case, connectivity between the applications and the message bus was developed using proprietary bus APIs and application APIs (see Figure 2).

The business process-based approach improves on EAI. In this case, a business process flow engine is used in conjunction with the message bus or middleware to run process flows that execute multistep transactions with various systems.

Disadvantages
None of these traditional approaches is ideal (see Figure 3). Some of the problems include:

• Custom or proprietary integration between message bus and applications. Both EAI and business process-based integration reduce the number of integration points compared with the peer-to-peer approach. However, all three approaches require custom or proprietary integration between the message bus and each application, and different and proprietary data formats were involved at each integration point.
• Tight coupling between message bus and applications. All the applications need to know the inner workings of the other applications with which they are integrated. The integration between systems is granular and tightly coupled by message type. The business process management tools used in traditional EAI implementations were proprietary. This prevented the use of best-of-breed products.
• Programmatic rather than abstracted data access. The challenge of accessing, integrating, and transforming data (enterprise information integration) has largely been left to developers to perform using manual coding. The presence of disparate data sources is a fact of life in enterprise IT environments. Developers have adapters for accessing data sources, transformation engines for reformatting data, and replication for physically consolidating the data. To integrate data sources, developers use these tools to program the integration requirements into applications. Although this approach works, it is neither efficient nor flexible, for these reasons:
  • Lack of abstraction. Programmers must use low-level APIs to implement integration requirements, which greatly increases development and maintenance costs.
  • Multiple data formats. Each data source has its own API and data format. The developer must understand each data source in sufficient detail to manage the data integration. This often requires multiple specialists to implement and maintain the integration, driving up complexity and cost.
  • Custom information integration framework. The developer has to manage all the integration issues such as the relationships between data sources and data formats. This leads to one-off solutions and inconsistencies that are difficult to maintain.
  • Tight coupling. Programming creates hardcoded dependencies between applications and data sources. Updating a data source can break many applications, which hampers maintenance.
  • Limited reusability. The integration code tends to be specific to each application and data source, making it difficult to reuse.

Why SOA?
All three traditional application integration approaches are complex, costly, and inflexible. Moreover, they don't support the needs of business changes. Service-oriented architecture (SOA)-based application development and integration, however, solves many of these problems.

A set of standards-based interaction patterns defined by SOA enable a consumer application to utilize services. These approaches define how services are described, advertised, discovered, and communicate with services.

Unlike earlier approaches, in SOA, all patterns around a service are implemented on standards-based technologies. Most communication middleware systems, such as remote procedure call (RPC), common object request broker architecture (CORBA), distributed component object model (DCOM), Enterprise JavaBeans (EJBs), and remote method invocation (RMI), rely on similar patterns. No implementation is perfect; issues exist with interoperability as well as defining acceptable standards.

SOA attempts to eliminate these weaknesses. For example, each middleware system has a fixed granularity: functions for RPC, objects for CORBA, and so on. Services, however, can be defined as functions, objects, applications, or other things. This makes SOA adaptable to any existing system and does not force the system to conform to any particular level of granularity.

SOA helps information systems move toward a "leave-and-layer" architecture, meaning that instead of altering existing systems to provide a standards-based services interface, they are wrapped with a layer that provides the services interface. Web services is one way of implementing this layer. Instead of replacing existing architectures, SOA transforms systems and applications into agile services. SOA covers not only the information from packaged applications, custom applications, and legacy systems, but also the functionality and data from IT infrastructure such as security, content management, and search. SOA-based applications are faster to build because they can easily add functionality from these infrastructure services.

The high-level view of enterprise applications using service-based integration shown in Figure 4 bears a strong resemblance to the structure shown in Figure 2. However, the key differences are that this system uses standards-based services and includes process/data services, orchestration, and composition. The standards-based services become the integration points between applications. Orchestration and composition of services add flexibility, reuse, and integration of services.

Now, that you've had a brief overview of SOA, let's take a detailed look at SOA to see how it makes integration easier.

SOA Design
SOA allows enterprises to focus on business processes in their application development, rather than focus at a lower level on integration or application issues. An SOA is a collection of reusable networked services, communicating through well-defined, platform-independent interfaces. These services provide access to data, business processes, and IT infrastructure, and they allow for service provision, consumption, and life cycle management.

Table 1 highlights the differences between distributed component architectures and SOA. SOA provides many advantages over traditional approaches; it is:

• Standards-based
• Loosely coupled
• Geared toward shared services
• Coarse grained
• Oriented toward federated control

An SOA-based solution requires a certain set of characteristics from an SOA infrastructure; these are listed in Table 2, along with attributes required to manage and support an SOA infrastructure.

SOA Services
Figure 5 shows the three fundamental roles in an SOA—service providers, consumers, and brokers—and some of their artifacts and operations.

Services are software resources available on the network. Service providers expose services via standard mechanisms, and service consumers consume services programmatically over a network. Service brokers publish the whereabouts of services, and locate services when requested by a consumer. The consumer and provider roles are not exclusive; service providers can also be consumers, and vice versa.

Providers describe their services in standard language in a service contract and publish it with a broker. Clients query service brokers (or registries) for the services they want, and receive the contract and information on accessing the service. The client or consumer then binds to the service and calls the provider directly.

A service has two parts: the interface and the implementation. The interface defines the programmatic access contract between the consumer and the provider. A service interface must contain the identity of the service, input and output data details of the service, and metadata on the service's functionality and purpose.

The service implementation contains the functional or business logic of the service. The implementation should be a "black box" to the service consumer; the consumer doesn't need to know the functional implementation of the service.

There are five types of service:

• Data Access: allows uniform access to different data sources
• Component: provides access to the services of packaged applications such as enterprise resource planning (ERP)
• Business: provides complex services that use more than one function of packaged or custom applications
• Composite: uses the three previous types to create a new service that includes both new and existing functionality
• Shared or enterprise infrastructure services: lower-level services, such as message logging, whose reuse allows rapid creation of new higher-level services.

Data access services. Data access services allow users to access, integrate, translate and transform data from various relational and non-relational data resources across the enterprise. These services typically hide the direct access to the resource, the complexity of the underlying formats, and the direct transformation and manipulation of data. They provide a uniform API, loose coupling, a common data model, and reuse of consistent information across applications.

Services for data access are the most common, widely used, and easily implemented services in an SOA architecture; separating the data layer and the application layer is generally straightforward. As data resources are widely accessed and shared, they become the first target for implementation as services.

XML is widely used for application data exchange. An infrastructure that provides abstract, uniform data access, regardless of the source, is valuable in SOA implementations. XML data services (XDS) supplies access to multiple types of data resources, data modeling capabilities, translation and transformation of physical data to logical data, and XML-based access to logical data.

Component services. Component services are coarse-grained services that are published from a single enterprise resource, whether it is a packaged application (for example, ERP, customer relationship management [CRM], or supply chain management [SCM]). An example of a component service might be "Add customer in ERP." These services are considered valuable enough to be published directly.

Component services are implemented by using individual application APIs to expose reusable functionality. These services can be implemented using distributed computing technologies such as J2EE EJB, COM/DCOM, and CORBA.

Business services. Business services are functionality, exposed from business applications, which perform one or more business operations. Business services are usually composed of multiple business transactions across multiple applications. They may be end-to-end business processes, for example, Process New Hire, or they may need to be part of a larger business context, as in Provision New DSL Line or Add New Employee. Provision New DSL Line needs to be part of a larger context such as DSL Order Processing, requiring information from that process to complete its business function.

Composite applications. Composite applications are created by combining new logic with transactions that have been exposed by existing applications as, for example, business or component services. Application integration technologies and business process management tools play key parts in creating composite applications.

Function-specific portals such as Sales Portal and Employee Portal are examples of composite applications; these would require business, component, and data services.

Shared or enterprise infrastructure services. The greatest value provided by an SOA is the ability to compose applications rapidly and consistently by reusing existing services. These can be called shared services, common services, or enterprise infrastructure services.

Infrastructure services can be categorized into four types:

• Shared application services
• Messaging and brokering services
• Portal services
• Shared business services

Some examples of infrastructure services are service repository, service finder and broker, single sign-on, content brokering, logging service, monitoring, unique ID generator, data services, and security services.

Service granularity. Services can be defined, based on their functionality and the amount of data they send and receive, as fine-grained, coarse-grained, or composite.

Granularity of service has two related meanings in SOA: how the services are implemented, and how much data, or how many messages, are consumed and returned by the service. Fine-grained services implement minimum functionality and send and receive small amounts of data. Coarse-grained services implement larger business functions and exchange larger amounts of data.

Fine-grained services are best consumed by coarse-grained or composite services rather than directly by end applications. If an application is built using fine-grained services, the application will have to invoke multiple services over the network, each of which exchanges small amounts of data. Consumers of coarse-grained services are not exposed to the fine-grained services that they use. However, coarse-grained services can't provide granular level security and access control, because they may use multiple fine-grained services.

Composite services can be assembled from coarse-grained and fine-grained services. Between coarse-grained services and composite services, the amount of data is not the differentiator. A coarse-grained service might be Create New Customer, which would need to validate the customer with some external service and create the customer record in a CRM application.

A composite service might be Provision New DSL Line, requiring a service call to validate the order, creating or verifying a customer, checking for product availability, and allocating the resources for the line.

High-Level SOA Solution Model
Enterprise infrastructure or shared services add great value in an SOA. Shared services are typically implemented in logical layers based on their functionality and locality. Figure 6 shows a high-level view of an SOA solution model applying the SOA characteristics shown in Table 2. The characteristics are implemented by a shared services layer, which is logically grouped into several service layers. The figure uses some of the applications and services developed by the BEA Information Technology group as an example.

Figure 7 expands each services layer and shows some of the common services for each; obviously these will differ for each enterprise. These are examples of services implemented and envisioned by the BEA Information Technology group.

SOA and Web Services
SOA does not require Web services for implementation, and Web services deployment does not result in an SOA. Web services, however, are a good example of a service, and are a component (not a necessary component) of an enterprise SOA.

SOA provides conceptual design patterns for service-based distributed systems. Web services provide standards-based technology to implement an SOA in a cost-effective manner using communication methods available everywhere today. When the Web Services Definition Language (WSDL) is used to define the service interface, then the service is a Web service.

CORBA IDL, Tuxedo FML (more of a data format definition than an interface definition language), COM/DCOM Microsoft IDL, and CICS common area (COMMAREA) were the other service definition languages prior to the wide acceptance of WSDL.

In the past, CORBA, EJB, COM/DCOM, CICS, and Tuxedo provided service-based application development and integration technologies for implementing an SOA. CORBA, EJB, and COM/DCOM provided object-based distributed systems, in which the objects provided functions or methods that performed certain services.

SOA provides an architecture to manage the many loosely coupled services developed using Web services or other similar technologies. Web services can be implemented using J2EE-based application servers, an enterprise service bus (ESB), or the .Net framework.

J2EE application server-based Web service implementations use components and applications developed using J2EE specifications, such as EJB, servlets, and Java database connectivity (JDBC), and expose (JAX-RPC) them as Web services using SOAP over HTTP or over Java message service (JMS). Asynchronous, coarse-grained Web services are implemented using JMS services provided by the application server.

In addition to pure J2EE-based Web service implementations, the BEA platform provides value-added and abstracted, controls-based, process flow-based Web service implementations with WS-acknowledgement specification support.

SOA and Enterprise Service Bus (ESB)
An ESB combines publish/subscribe messaging, synchronous (RPC) and asynchronous messaging, basic transformation, content-based routing, and native Web services support. In addition, ESBs support a range of application servers. An ESB acts as a shared messaging layer for connecting applications and other services throughout an enterprise. Originally defined by analysts at Gartner, ESB is increasingly seen as a core component in a service-oriented infrastructure.

J2EE connector architecture (J2EE CA)-based adapters are usually used to integrate ESBs with enterprise systems. The ESB supplements its core asynchronous messaging backbone with intelligent transformation and routing to ensure that messages are passed reliably. Services participate in the ESB using either Web services messaging standards or the JMS.

An important distinction between JMS and Web services is that JMS is a Java programming API, whereas Web services provide a standard wire-level protocol (SOAP over HTTP). This means that JMS does not remove the interoperability problems associated with proprietary messaging solutions. Every JMS provider (for example, WebLogic, Sonic, and Fiorano) implements a proprietary protocol between the JMS client and the JMS server. Therefore, there is no interoperability between JMS clients and servers; for example, a Sonic JMS server can be accessed only by a Sonic JMS client.

An ESB by itself can't provide the complete infrastructure for SOA. For example, it is difficult or impossible to implement application logic, components, process flow logic, and workflows using an ESB, but those can be implemented and service-enabled rapidly by the BEA platform. For an enterprise that is not geographically distributed, an ESB is not required to implement SOA.

SOA and Event Driven Architecture (EDA)
In enterprises, systems go through state changes. An event is triggered by a state change and carries data about that change to other systems or people.

Businesses can benefit from handling events caused by real-time system state changes. For example, instead of periodically checking to reorder items for a warehouse, the warehouse management system can trigger an event when a threshold for low item count is reached. That event can then kick off alerts to reordering systems.

According to Gartner, the fundamental architectural pattern of modern, agile enterprise IT is service-oriented and event-driven.

Event-driven architecture is an approach for designing and building applications in which events trigger messages to be sent between independent, decoupled modules that are unaware of each other. An event source typically sends messages to a middleware or message broker that has subscribers registered for that message. Since the event messages are delivered through a message broker using the publish/subscribe approach, it is possible for one event to be delivered to multiple consumers. This is the main difference between EDA and SOA: in an SOA the producer and the consumer have a one-to-one relationship, and in an EDA an event producer can eventually deliver one message to any number of consumers based on the subscription rules registered with the message broker.

Advanced EDA implementations must support complex event processing, in which multiple related events are aggregated using policy-based rules and a single business event or service is triggered. iSpheres, a BEA partner, provides an example implementation of this feature. It is also possible to custom build this feature using the BEA platform.

EDA and SOA are complementary architectures. An EDA source can be an SOA consumer or server application, and an EDA consumer can be an SOA client program. BEA WebLogic Integration's message broker, various event generators, and adapters, along with BPM, are fully capable of tying EDA-based and SOA-based implementations together.

SOA is used when a business problem requires a request/response or real-time solution, and the client knows the service provider ahead of time. EDA is suitable when business requires one-way messaging, involves long-running asynchronous process, and the event source doesn't need to know who the event receivers are.

Applicability and Disadvantages of SOA
Use of SOA requires additional design and development efforts and infrastructure, which translates into additional IT costs. Therefore SOA may not be a viable architecture for all cases.

For the following applications, SOA and Web Services may not be the recommended architecture:

• Stand-alone, non-distributed applications that do not require component or application integration; for example, a word processing application does not require request/response-based calls.
• Limited scope or short-lived applications; for example, an application that is built as an interim solution, not intended to provide functionality or reuse for future applications.
• Applications where one-way asynchronous communication is required, and where loose coupling is unnecessary and undesirable.
• A homogeneous application environment; for example, in an environment where all the applications are built using J2EE components. In this case it is not optimal to introduce XML over HTTP for inter-component communications instead of using Java remote method invocation (RMI).
• Applications that require rich, GUI-based functionality; for example, a map manipulation application with lots of geographical data manipulation is not suitable for service-based heavy data exchange.

Most of these scenarios do not apply to enterprise applications. SOA is network-centric, so it requires complex monitoring and auditing of services. Because service reuse and sharing are key characteristics of SOA, the number of consumers of services will be high, which raises versioning, change management, and metering issues. These issues require a management infrastructure that may be too expensive for some projects.

SOA is a strategic solution with significant tactical applications and benefits; however, there is a threshold to be passed to get the full benefit of SOA. SOA usually requires analysis of costs and benefits.

Can SOA make application integration easier? Just ask the IT managers who are using it to tie together their custom, commercial, and legacy apps to improve mission-critical business processes. They'll tell you.

Compared to the integration approaches of the past, an SOA can provide many benefits to an organization—reusable, platform-independent business services, standards-based development, and the ability to make changes quickly when necessary to keep up with the speed of business.

About the Author
Rag Ramanathan is currently an enterprise architect in the BEA Worldwide Enterprise Consulting group that focuses on developing SOA best practices guides for BEA consultants, partners, and customers, and delivering business value assessment for SOA. He has more than 12 years of industry experience with more than six years at BEA as a software engineer in R&D and professional services consultant. Contact him at soa@bea.com.