Integration at the Edge
The enterprise service bus erases the distinction among internal and external networks for supply-chain applications
by David Chappell

Posted May 26, 2004

For the past decade, throughout the era of enterprise application integration (EAI) and the evolution of the Internet and application server technology, a clear dividing line has developed. This line separates the communications and application integration infrastructure within the four walls of a corporation and external communications with business partners, vendors, and customers.

This separation has been driven largely by the capabilities and limitations of the technology. To date, technology such as application server infrastructure is specifically designed to make clear distinctions between what's inside the firewall versus what's outside the firewall. This distinction is evidenced by completely separate architectural approaches, with different programming models required for building applications. In the J2EE application server architecture, for example, this approach is manifested as a Web container versus an EJB container.

Hub-and-spoke EAI brokers could get as far as the corporate boundaries, but were not really built well for scaling beyond that. Various bridging technologies are designed to bridge the gap at the edge of the network. In many legacy cases, this technology is bolted on as an afterthought. The majority of the effort being done to date in the area of Web services has also been focused on this edge of the network.

But just exactly where is the edge of the network? Before we answer that, let's explore another forbear of the enterprise service bus (ESB), the e-marketplace, also known as the e-commerce trading hub (see the sidebar, "ESB").

e-Commerce Trading Hubs
In a trading network of business partners, companies desire to move away from expensive, EDI, value-added networks and use the public Internet as a means of communications among entities, wherever possible, and to lower the barrier to entry for participation such that small-to-medium enterprises can afford to participate. This desire was the impetus behind the creation of e-marketplaces and trading exchanges such as those powered by CommerceOne and GE Global Exchange Services (GXS).

A trading exchange acts as an intermediary, or semiprivate business portal, that facilitates electronic commerce between buyers and suppliers in a supply chain (see the sidebar, "Case Study: ESB in the Supply Chain"). The majority of the interactions with the portal are not browser based—the interactions are done directly among specialized applications that require little or no human interaction. The interactions happen between applications residing at the trading partner and back-end applications that reside in the trading hub. These back-end trading hub applications provide value-added functions such as dispersion of request for quotes (RFQs) between a buyer and multiple suppliers or availability to promise (ATP) data from suppliers to buyers (see Figure 1).

This environment introduced some rather challenging requirements, some of which seemed at odds with each other. For example, this scenario requires secure access between the applications residing at the partner sites, and the applications residing in the trading partner hub, using the public Internet as the vehicle for communication. This scenario also required reliable messaging, yet the traditional message-oriented middleware (MOM) vendors did not have a MOM infrastructure that was capable of spanning across the public Internet in a scalable and secure fashion.

The suppliers communicating through a trading hub are often fierce competitors with each other, so it is imperative that they not be able to see each other's data such as pricing information. This consideration requires full authentication and access control between the trading partners and the trading hub. Functionality in the trading hub must be exposed to trading partners through a service interface. A supplier must also be able to freely share its data with the trading hub, and the trading hub applications must selectively be able to pass that data along to buyers, but not to other suppliers competing for the same business. A supplier must never be able to masquerade as another supplier and get access to its sensitive data.

The size of an e-marketplace community could potentially consist of thousands of trading partners all communicating through the same trading hub. Trading partners needed to be able to communicate asynchronously with the trading hub in a "fire-and-forget" mode using reliable messaging.

Some very important ESB capabilities were born out of these requirements. For example:

• Strict authentication and access control among the entities connecting to each other
• Scalable clustering of message servers to be able to handle large volumes of message traffic from potentially thousands of concurrently connected trading partners
• Complete segregation of data channels
• Selective control over which channels can get opened up between application end points, across intermediary hubs
• Selective access to shared service interfaces and end-point destinations
• Coordination of the business-level message exchange between applications that are separated by physical network domains and geographical location
• Secure MOM communication through as many firewalls as there happen to be along the way between the trading hub and the suppliers and buyers

Another requirement of the trading hub that contributed to the vision of the ESB architecture was the ability for a trading hub to do business with other industry-related vertical trading hubs. This requirement means that segregated groups of applications (the trading hub back-end applications) need to selectively expose and share their interfaces and data with groups of applications residing in other trading hubs. Each trading hub and each trading partner needed to be able to maintain its own autonomy and local integration environment while still being able to communicate in a larger e-commerce network. This network of trading hubs and trading partners could potentially fan out ad infinitum.

The evolution of e-marketplace infrastructure has significantly contributed to the emergence of ESB providers offering the underpinnings to support these requirements. The vendors building trading exchanges looked at a variety of EAI brokers and application server technology, and turned to the messaging vendors for help. Some of the newer messaging vendors were already beginning to provide the required underpinnings for the routing, segregation, and fan-out deployment topology.

This process of defining requirements, talking to vendors, and designing this infrastructure took more than a year to happen. During this time, a number of messaging vendors and EAI broker vendors were putting a great deal of energy and effort into designing and building their next-generation products to fit the requirements to support the e-marketplace vendors, which helped to contribute to the emergence of the ESB concept.

The Extended Enterprise
Fast forward a couple of years, and it turns out that the same kind of technology requirements that are needed to be solved to support large-scale trading exchanges are also the same requirements to support the needs of corporations building out their own integration networks. While e-marketplaces as a business model never really took hold as well as their early proponents had hoped, the need to provide common shared services across departmental and corporate boundaries is still there. This need expands well beyond supply-chain scenarios.

Because of mergers and acquisitions, collaborating business partners, and globally distributed business units, there exist varying modes of communication based on the degree of trust between the business entities. This model represents the "extended enterprise." In the extended enterprise, the edge of the network is always changing, or perhaps there was never really a single outer edge to begin with. In a global organization of semi-independent business units there are many firewalls, yet there is a need to have a distributed integration backbone that transcends the underlying topology (see Figure 2).

There are different trust levels when dealing with business partners. Reliable messaging requires that a piece of MOM software be installed at either side of the communication link. Sometimes it is perfectly acceptable to be able to tell a business partner, "If you want to do business with me, you must install that software at your site." In this case it's appropriate to have an efficient reliable MOM link between the two organizations. New reliable SOAP protocols can help but not completely alleviate that requirement. If you don't have that close of a relationship, you may instead need to supply clear instructions on how to send business documents over a secure HTTP link, and manage business message exchanges using a layered service that specializes in managing the details of partner links.

The ESB provides the architecture that separates the higher-level SOA and integration fiber, such as the management of physical destinations as abstract end points, and the transformation and routing from the details of the underlying protocols. This means that the network boundaries can change over time, and an ESB can support the changing of the underlying protocol—that is, from MOM to SOAP-over-HTTP to WS-Rel*—without affecting the higher-level integration functions that sit above all of that.

In the trading hub example, SOA, data transformation, and routing of messages based on context was the job of the value-added services that reside inside the hub. As we take lessons learned and concepts of trading hubs and apply them to in-house IT integration, we can use those same concepts and make them available as independently deployed services.

About the Author
David Chappell is vice president and chief technology evangelist for Sonic Software, a provider of services-oriented integration technology. He is the author of several previous books for O'Reilly & Associates on enterprise messaging and Web services.