Saturday, January 24, 2009

SOA is much too young to be dead

Today I'll look into the SOA controversy that was generated by Anne Thomas Manes's SOA's obituary
This may look like a slow response (90% of the blogosphere has already published their comments) but I will take some length to explain the issue (and this required a little time for thinking over). Surprisingly, one still get a very fuzzy and abstract idea about what SOA from reading all the positive answers to Ann Thomas Manes.

What is much clearer are the benefits one should expect from SOA (and they have been clear from 2003). SOA brings increased agility and reduced costs. The increase agility comes from reusability an the cost reduction comes from sharing services. Reusability comes in two flavors : direct (reuse of an existing service) and indirect=mash-up (quickly mixing existing services to produce a new one).
There are additional benefits (such as the ease to mix internal and outsourced services) but these two are enough to get anyone's attention. The question is: how to get there ?

1. SOA is a 3 step process

First, we decompose SOA into three steps ("doing SOA" means doing all three in sequence):
  1. Service Definition
  2. Service Architecture
  3. Service Integration

Service Definition simply means to come up with the catalog of services. It is a "business oriented" step: the business processes are analyzed and the relevant business services are identified. At first it looks like the "simple" (nothing but simple actually) first part of a functional analysis of a new system. There is (much) more to it, because the role/user analysis is an integral part. The focus on reusability and agility starts here. Hence it is necessary to play with business scenarios, which further means that it is not an information system play. In the "acronym soup", the relevant terms are EA (Enterprise Architecture) and SOBA (Service Oriented Business Architecture) which have been introduced later on to insist on the "business/enterprise" dimension.

Service Architecture is the technical step that takes a large catalog as an input and produces a structured hierarchy as an output. A list of thousands of equally visible services is in no way a feasible approach to manage a large information system. Classical issues such as modularity, encapsulation, visibility, etc. must be taken into consideration. I write "classical" because this is the heart of software engineering and software architecture. Re-factoring techniques (the same that are used when writing a class or a component library) must be applied to produce services that have a better chance of being reused or being combined. I won't dwell into it today, but the role of data & models architecture is key (as I said, this step is the IS step).

Service Integration is what most people think about when they say SOA: linking all the services into an integration infrastructure. This is where all the technologic  issues live (ESB or not, SOAP or REST, etc.). The term SOI (Service Oriented Integration - for instance, see this definition) was coined to denote this step.

These three steps help to grasp the huge difference between "SOA in the large" (with a large enterprise as its scope) and "SOA in the small" (SOA for one information system, either for a small company or one department). The effort repartition obviously varies, but here is the pattern from my own experience:
  • In the large: 40/30/30  (i.e., 40% of the effort is the definition part)
  • In the small: 20/10/70
It is interesting to notice that there is (almost) nothing new with "SOA in the small": service definition + architecture + flexible implementation is what has been making a good piece of software for 30 years. 

2. What is easy/hard ?

Service Definition is hard when the scale gets big. Although producing the service catalog is straightforward for a department-size project, there are mutiple issues of heterogeneity and multiplicity of stakeholders that makes it much more complex to manage at the (large) entreprise level. Not only the management of the catalog gets harder as the size grows, but there is a subtle balance to be found between "common sense" and "formal methods". There are indeed methods/pattern/frameworks available for this first step but:
  • if this step is too formal, too many stakeholders are discouraged
  • if it is informal, the process crumble under its own weight.

The same is even truer with Service Architecture. It is quite simple when doing "SOA in the small" but gets harder as the size grows. Contrary to Service Definition, there is little help available as a methodology: hard learned IT experience is necessary. Unfortunately, this is not something that one learns through writing Powerpoints. If that step is poorly executed, the integration is more difficult, but - and this is the key point - little reusability or agility is observed later on. This is something that I am trying to teach during my "Information System Course" at Polytechnique, but this is a hard, technical topic which is better acquired through practice than theory.

Service Integration is actually not so difficult nowadays, because the technology is quite mature. I am not saying that it is easy, but it would be sad to ignore the progress that has been delivered by technology providers. There are still a few difficult issues (cf. next section) and, more generally, a large-scale integration project still exhibits all the possible pitfalls of a complex IT project, but there is nothing new here. Actually, when done "in the small", this is straightforward and there are so many success stories to prove it.

3. What is the maturity state of SOA ?

Using the same decomposition, here is my own assessment: 
  1. 50% for Service Definition. This step is rather well explained, many good books are available (such as "SOA for profit"). It is still a difficult job at the enterprise level. There is indeed a governance issue (those who mock the "governance issue" have missed what Enterprise Architectuire is, and are still trying to accomplish "small scale SOI"). There is also a "pedagogy issue": an entreprise-scale effort cannot be undertaken if its meaning is not understood, by all stakeholders.

  2. 20% for Service Architecture. There are very few books that can help, and even fewer are directly relevant for SOA. The only one that I recommend to my students is "SOA, Le guide de l'architecte." (in French, sorry for my English speaking readers). I have tens of books about IT/software architecture in my private library, including great pieces from Fred Cummins, Len Bass, Richard Shuey, Paul Clements, Jean-Paul Meinadier, Jacques Printz, Robert Seacord, Roel Wieringa - to name a few -, but none of them (including my own two books :)) adress the issue of Service Architecture "in the large" thoroughly.

  3. 70% for Service Integration. The technology is there, it is scalable and proven (WebSphere or Weblogic being two classical commercial examples). Obviously, there are still technical issues. For instance,
  • distributed transaction & synchronisation is still a hard problem.
  • performance overheads (cf. SOA is not scale-free) still exist and makes the deployment tricky when response time is an issue.
  • monitoring & autonomic load balancing is difficult and/or not sufficiently developed.
  • service discovery and pub/sub architecture (EOA) are not as straightforward to implement as one might wish
But those are hard issues, nothing to be ashamed of :) 


4. SOA is a practice, not a target or a technology

As David Linthicum said: "SOA is something you do, not something you buy". It is not something that you do once (a project) but that you do all the time, continuously. 
This is what brought the creation of the sustainable IT architecture approach. Done right, SOA is a practice that transforms information systems for the better. According to the money that you invest, to the legacy and to the business constraints, the transformation is slower or faster, but it takes time and is never ended. The sustainable IT approach comes from a change of perspective: instead of always focusing on what is to come, on the new projects, one should search for more value from the "existing assets". Another "sustainable" principle is to save your strengths to last longer (hence the continuous improvement approach as oppose to the big bang).

It's a "culture thing". Part of it (cf. Service definition) is everybody's concern, hence it must be simple and must be pushed at the CxO level. Therefore, IT governance is a key component for changing the culture.

I will conclude by stating that SOA represents a "Chinese strategy" for IT management, in sense of François Julien. I could not urge you enough to read his book, but if you have not, a key principle is to distinguish between planned strategy (top-down, in the Greek fashion) and "building up your situation potential". A Chinese proverb says than one does not grow a plant faster by pulling its stem.
Similarly, instead of pushing "SOA technology projects", a better approach is to build up your "situation potential" : build a common data model, train everyone, build a shared consensus about what the service catalog should be, etc. Each project is then seen as an opportunity to move one step in the right direction. 






3 comments:

Dimitre said...

Yves, thanks for the great article and suggested readings. I agree with your view of SOA and believe that it has a bright future.
In my opinion one of the things that are holding back the wide adoption of SOA is the abstract concept of service. It conveys really vague meaning that you are asking someone to do something for you. But, what is that? Most of the people interpret the service as Remote Procedure Call (RPC) or sending a message.
I believe that we are going to make a breakthrough in SOA adoption if replace the term service with concrete service types that developers and business people can relate to. For example we build applications:
- data;
- operations – programming logic;
- presentation component;
We can define standard interfaces for data service, operation service and presentation component service.

It is much easier to approach the business saying:
- Which data do you want to standardize? Customer, Order, etc. We do not need to discuss specific methods (like how to search for customers). It is handled by data service interface. At some point we need to define the access control to data for different user roles.
- Which business operations do you to standardize? Check customer credit, address validation, etc.
- Which presentation functionality do you want to standardize? Those are coarse-grain presentation components that implement certain workflow like: Make payment, Select cell phone plan, etc.

Conceptually every enterprise application can be a client or server of such services, but it also may make sense to deploy the services in specialized enterprise repositories.

The benefits are really obvious when coupled with an application-level architecture that is based on the same service types. For example it gives developers the flexibility to get a data object from locally implemented data service (using access to DB), another application or Enterprise Data Repository.

The idea is described in more details in my blog – http://shipka.wordpress.com/articles/bottom-up-approach-to-soa/
Implementations of this approach to SOA are available at: http://shipka.com.

What are your thoughts on this?

Yves Caseau said...

Thanks Dimitre for a very interesting comment. I have started to browse through your site ... it will take a litle time before I can really give you a feedback:)
Very generally speaking, I both agree and dissent with what you are saying:
(1) You are totally right with your proposal to replace the abstract term "service" with more pratical/precise concepts. A taxonomy such as the one you propose (data/operation - logic / presentation) makes perfect sense.
(2) on the other hand, part of what I am saying "lies on a different "abstraction space", that is, at the business level. So "bottom-up" cannot do 100% of the job, since a litle "strategic/future" thinking is required.
Those two don't really oppose with one another, but I don't think that a company can get 100% of the "SOA benefits" without making a managerial commitment to "Enterprise Architecture" (which is more than developpers using a proper methodology).
Cheers,
- Yves

Dimitre said...

Thank you for the feedback. I agree with your statements about the need of strategic/future thinking and active involvement of the “Enterprise Architecture Group”. They are critical for the success of SOA regardless of the approach “top-down” or “bottom-up”.
My message was not clear, since it tried to communicate two things at the same time:
1. There are technological limitations for implementing successful SOA strategy. The prevailing SOA technologies SOAP and REST force companies to use Remote Procedure Calls (RPC) as the only tool for service implementation. The service concept in this context is synonymous with RPC. The adoption of a new technology that enables applications to share objects, operations and presentation components is a major advancement and creates new opportunities. First it is better aligned with the architecture of individual applications. Second, it enables sharing at a higher level that is more intuitive and easier to communicate to the business. The sharing of presentation functionality will provide the biggest cost saving opportunities for the companies.

2. The “bottom-up” approach coupled with the new application integration technology enables light-weight SOA implementations that are more acceptable for the business. The “top-down approach” has three phases: 1) Service Definition, 2) Service Architecture and 3) Service Integration until we start making ROI. When applying the “bottom-up” approach we begin identifying enterprise services (data, operations, presentation components) within the context of every new application. Those services may be implemented locally and exposed to other applications when reuse opportunities are identified. Shared resources (services) may be moved to specialized repository later. One of the key features of the new application integration technology is the ability to change resource implementations from local to remote and vice versa without affecting the application functionality. The approach enables companies to create only the resources they need. It does not have upfront cost and generate ROI as reuse opportunities are realized.

 
Technorati Profile