Top 10 Reasons SOA Projects Fail

Last month an interesting study about SOA implementation success rates was released:

The Burton Group SOA Study

It states that fully 50% of the projects assessed were considered a complete failure and 30% more seriously underperformed. That leaves only a 20% success rate - this track record is nothing less than dismal. Why is this still happening? Why is SOA so difficult to realize?

Well, rather than writing a dissertation, let's address the root causes with a top ten list:

1. There is still no widely agreed upon set of best practices for SOA implementation (or for that matter even an industry standard definition for what SOA is and isn't).
   
2. The confusion regarding SOA and web services is still pervasive; technically you can have SOA without any web services being developed per if the architectural principles are met.
   
3. SOA is not application-centric, despite appearances to the contrary. SOA is the first and only comprehensive development paradigm that is wholly integration-centric. (it combines the previous middle-ware integration capabilities with OO and web-centric logic).
   
4. Most SOA architects don't have a clue about 75% of what a true enterprise integration requires. Solution integrators tend to hire former J2EE / web services developers and assume that they will magically grasp the intricacies of System of Systems enterprises magically from the giant wisdom cloud in the middle of so many diagrams. NOT...
5. SOA without the enterprise data layer isn't going anywhere, fast.
6. COTs driven SOA is inherently risky, as the recent merger between two of the four leading vendors has proven (Oracle buying BEA). How many people are now having to reassess their infrastructure as a result?
7. Deploying a SOA stack in itself that doesn't actually do anything is an intellectual exercise to be sure, but also a complete waste of time and money.
8. SOA is dynamic, or at least it's supposed to be and more importantly it is not our first foray into managing dynamic environments, so why does the industry seem to have collective amnesia in this regard? Systems engineers have been doing what SOA developers are failing to do now for twenty years, yet you'll never see a member of that rare species on most SOA projects.
9. Developers rule - NOT. Design also drives development not the other way around, we wouldn't want the drywall guy designing a skyscraper, yet that happens in SOA implementations (figuratively speaking). Some people call this model-driven SOA, however it was the way SOA was always supposed to be conducted. You can't really do "Agile SOA" in the strictest sense of the Agile philosophy given SOA's integration implications - the big picture has to be understood up front.
10. The standards and hype don't drive the solutions, the business does - if we remember that we can avoid some obvious traps. Hype doesn't solve problems or deliver capability and the standards that are applied are the ones that make sense, not merely the ones everyone says ought to be applied in a perfect world.
    

copyright 2008, Semantech Inc.

What is a SOA Architect? (part 1)

Of all the topics surrounding Services Oriented Architecture, none is more perplexing than this seemingly simply job description exercise - What indeed constitutes a SOA architect? Moreover how does that role compare or contrast with the following roles:

1. Enterprise Architect
2. Solutions Architect
3. Business Analyst
   
4. Middleware Architect
   
5. Application (or J2EE) Architect
6. Web Services Developer
7. Lead Application Developer
8. Systems Engineer
   

Are these roles interchangeable or do they overlap? For those us who work in the business, we certainly see many variations or combinations of these roles and skills sets in the expectations of organizations managing SOA implementations.



The most common combination of skills centers around these combinations:

• roles 2, 3 & 4
• roles 2 & 3
• roles 2, 3, 4 & 5
• roles 5, 6 & 7
• roles 4, 5, 6 & 7
  

Very seldom do I run across role 1 being merged into the others. Let's look a little close at the implications of these wide-ranging expectations though.

1. The role of SOA Architect does not yet have an industry standard definition
2. The role of SOA Architect often includes many diverse skill sets
3. The role of SOA Architect usually does not encompass systems engineering - this is interesting considering that systems engineering is the one discipline that deliberately combines all of the others.
   

copyright 2008, Semantech Inc.

What is Integration?

I read an article the other day that got me thinking about the general misunderstanding that often surrounds the term "integration. " The article was describing a recent study on SOA project success and equated some less successful projects with an increased level of integration.

It then occured to me that the enterprise view of integration is radically different from the application developer's view of the term integration. Understanding the difference between the two is critical. So let's look at them...

Application Integration - This refers to the specific point to point development of interfaces or necessary coding in order to ensure interoperability within a limited scope (this is often extended to the ability to connect data sources point to point as well).

Enterprise Integration - This refers to the full spectrum of connectivity and interoperability issues across all layers of an enterprise architecture (and across all instantiations of that architecture - regionally or globally). Thus, Enterprise Integration concerns itself with application dependencies and redundancies, potential deconfliction and consolidation, code modularization and reuse, system of system dynamics etc. If this is starting to sound familiar it should, much of what is emerging as SOA principles have in fact been borrowed from Enterprise Integration.

And even though most SOA projects have only deployed over the past 5 years, complex system of systems environments have been around for two decades. Why people are thinking that the type of dynamics being managed by SOA is new is surprising - but perhaps not so surprising if you consider that many "SOA architects" are in fact application architects and never had the type of systems of system experience that traditional enterprise architects and systems engineers have had.

In other words, if one is used to building stand-alone solutions stacks, then dealing with the dynamics of dozens or hundreds of interactive elements represents a radical paradigm shift - and how these people view 'integration' comes from the previous narrow-scope perspective.

This all relates back to SOI in that Service Oriented Integration recognizes that much of what we're doing with SOA Should Not Be Viewed as New or substantially different than what we've done previously with Enterprise Integration. Sure the standards and tools have improved and changed but Not the Practice Itself. This is perhaps the single biggest potential area of improvement for SOA implementations - the realization systems engineering skills and knowledge can radically reduce solution risks.


Copyright 2008, Semantech Inc.

Why SOI Matters

The promise of new technology is often tempered by the trials of its adoption – Services Oriented Architecture (SOA) is an excellent example of this. Billions of dollars are being spent by thousands of organizations worldwide as they attempt to transform themselves with SOA as the major change agent. Yet despite this overwhelming endorsement of the concept of SOA, the reality of its adoption has been much more difficult to quantify.

It has taken nearly four years for SOA-related solutions and products to mature enough to even begin to take stock of what their true impact on the enterprise has been to date. There have been mixed signals so far, but we’ve seen indications that a significant percentage of those projects have run into troubles. Part of the problem of course is determining exactly where SOA stops and other enterprise IT considerations begin. Or perhaps another way of looking it is that SOA is part of a larger picture but until now the nature of that relationship simply hasn’t been made clear. Once the full context of SOA-based implementations is better understood, there is a significant likelihood that SOA projects will enjoy greater degrees of success. That context might be described as Services Oriented Integration (SOI).

Services Oriented Integration (SOI) represents a new paradigm both in theory and practice, one that merges a number of existing IT domains. It is also a methodology, one borne from years of experience in enterprise integration but fully cognizant of the current generation of technologies now available to facilitate enterprise transformation. Like any good methodology, the foundation of SOI owes its inspiration to real-world experiences – experiences that required new types of problem solving and thinking as well as new tools. SOI does not represent anything radically new, it is not meant to contribute to the current escalating levels of complexity, on the contrary it has been created specifically to help manage that complexity by combining what might otherwise be considered separate disciplines.

Copyright 2008, Semantech Inc.

Service Determination

Two of the most complex problems facing all SOA implementation projects tend to be:

• The determination of what to do with existing capabilities
• The determination of how to define the specific nature of new or transformed capabilities.

Existing capabilities are usually systems but of course can include services or other code that could be readily included within some type of SOA development effort. We tend to refer to this as legacy capability, but the term legacy often has a negative connotation. Perhaps "realized" capabilities better describes the situation in that it reflects that these elements were at some point already successfully deployed and adopted.

Deciding upon the nature of the new capabilities is also extremely important. If one approaches the situation with the assumption that the development of standards-based services code will magically allow for a plug & play enterprise they are likely to be fairly disappointed. Building service applications does not represent a solutions architecture - with the advent of SOA we must now be more cognizant than ever before of the various inter-dependencies and relationships between infrastructure, application logic and data architecture. This then implies that we need to consider and plan how all of these will fit together or else will leave our fate in the hands of unforgiving chaos.

The fact that all of this involves SOA doesn't change the fact that one must apply a number of basic system engineering techniques and principles to help manage these issues. The diagram below illustrates one possible approach; it breaks the process into three parts:

Part 1 - Capability Taxonomy
Understanding our domain starts here - this ensures that everyone is on the same page from day one, skipping this step (especially in a large enterprise) will place most transformations at risk from the get go.

Part 2 - Capability Granularization
First of all, you need to decide what this is going to look like - are you going to manage like services within some sort of module framework (i.e. statically determined compositions), or are you going to make more discrete services available - ones that can be combined by the users at run-time or added to pages / portals. Or perhaps there will be a mixed approach - whatever that approach is though there will need to be rules guiding how certain capabilities are going to be developed.

Part 3 - Capabilities to Systems Allocation
Then, systems analysis needs to occur to determine if existing capabilities (within systems etc.) can be parsed or transformed into the various new forms defined in the previous step. Only then will the eventual reuse and migration paths become clear.



Copyright 2008, Semantech Inc.

SOI Patterns - part 1: Overview

Some of you might be familiar with the concept of SOA Patterns or patterns in Object Oriented (OO) development. This approach goes back to the beginning of the OO movement in the early 1990's but has been progressively expanding ever since.

Some of the classic design patterns include:

• Command
  
• Builder
  
• Abstractor Factory
• Mediator
  

The Mediator Pattern = "Definition of an object that encapsulates how a set of objects interact. Mediator promotes loose coupling by keeping objects from referring to each other explicitly, and it lets you vary their interaction independently."

You can see perhaps embedded within this definition some core design tenets often associated with SOA. So it's no wonder that the since the technology itself is mostly derived from OO that the notion of design patterns would be inherited as well. By far, the most comprehensive exploration of SOA Patterns has come from famed SOA author Thomas Erl. What I like about what he has put together is the recognition of a number of larger integration issues within those patterns. Some of the patterns he's developed include:

• Data model Transformation
• Domain Inventory
• Legacy Wrapper
  

As might be expected though, most of these patterns is fairly SOA-centric, in other words the viewpoint is still tilted to the application perspective as being predominant. This is perfectly understandable when considering that the main audience is developers. However there is the larger audience of enterprise stakeholders including the architects and engineers that need to be taken into consideration. While a SOA transformation might be the central organizing theme within an enterprise, carrying it out will almost certainly extend beyond the boundaries of a SOA-only focused viewpoint.

This is where SOI Patterns come in. Those patterns recognize the necessary synergy between all architectures and elements in order to achieve complex transformation through enterprise integration. For example, how do we determine what SAN design will be facilitate an application hosting environment deployed in our data-center to support both existing network management capabilities (such as email) as well as a SOA services enclave ? This implies design configuration choices to security solutions, network management, legacy systems, processes, governance etc. How would we for example reconcile ITIL processes at the data center hosting our SOA with the governance built for that SOA solution (using traditional SOA tools only)?

The core notion that we need to consider here is that even as the scope grows and the interactions become more complex, there are a number of generic similarities between enterprise transformations that could be considered 'patterns.' These are our pool of potential SOI patterns. We will begin to explore them one at a time here on our SOI Blog this month ...

copyright 2008, Semantech Inc.

Semantech Launches the SOA Glossary Blog

We're proud to announce the launch of the Semantech SOA Glossary Blog. The purpose for this Blog is twofold:

1. To provide a supplement to the SOI Blog in order to explore specific concepts in-depth and to provide a reference point to many of the discussions we will hold on the SOI Blog.
2. To provide the foundation for an eventual, wiki-style online reference guide. When the Blog has traversed through several hundred terms, we will transform it into a more permanent reference resource.
   

One of our goals with this Glossary though is to avoid focusing on pure 'theory;' part of the charter of the Semantech Inc. SOA Glossary will be to infuse each topic entry with as much practical (implementation-related) information as possible.

Copyright 2008, Semantech Inc.

Architecture Mapping part 1

One of the most frightening realizations for folks who get involved in large SOA initiatives is how little of the project often involves SOA. Few walk into such a project with a blank slate, there are existing processes, systems and most importantly - expectations. What soon becomes apparent is that the nature of major transformations touches the full range of architectures both present and desired.

How do we manage this? Can it really be managed?

Well - yes it can be managed, but the how does make all of the difference in terms of the eventual success. We must start with the admission of the full scope of what we're dealing with - don't let anyone tell you're "boiling the ocean" in one breath and then ask you to transform their entire IT paradigm in the next. Set the record straight up front - making transformation happen will require an expansive view and more coordination than they may have anticipated. The upside of course is that this recognition automatically reduces about half of the risk that might otherwise be associated with the initiative.

One of the first things you'll run into is that the fact that there are a lot of architectures out there - whether formally captured or not makes no difference. Much of this will likely have been considered outside the scope of SOA, especially aspects of the data architecture - but nothing could be further from the truth.

It's usually within the data architecture even before the business architecture that you begin to see the implications of a great many SOA Design possibilities. But even within these other architecture elements, there must be coordination. The following diagram represents an example of how you can follow data architecture elements from concept to data exchange - this would need to be placed side by side with business flows and application designs following similar stages of development...



Copyright 2008, Semantech Inc.

Middleware SOA

This was a linkedinc.com question - Middleware SOA. Interesting future technology or what?

response:

There isn't an industry standard taxonomy in this regard, however what I think your asking about is most likely associate with traditional middleware capabilities - this involves data management queues (used to be referred to also as EAI). Metamatrix is a good example of middleware but some might argue that ESBs and other ETLs could be viewed that way as well. Some elements that used to be thought of as middleware (like CORBA) have now migrated to the top of the SOA stacks (i.e. are built within the application servers themselves).”

Copyright 2008, Semantech Inc.

So, What is SOA Governance?

I was reading an article on this topic recently and was concerned about the answer that was provided - when the question was asked, the author immediately dived into an explanation of SOA governance from a product perspective. I'm not saying that this information is not without merit, however I felt as though the core answer had been sidestepped. So, I'm going to take a shot at answering it - the "G" word, one of the bigger buzzwords around, what does it really mean?

Since it is applied to a much wider context than just SOA, it is perhaps appropriate to start with the basic philosophical context of governance as it applies to Information Technology. Governance is management, management of all the diverse 'managements' that inhabit the typical enterprise, everything from storage management to asset management to portfolio management and so on. Governance in its widest sense encompasses all of this.

Governance is first and foremost, an expectation - the expectation roughly translates to the need to ensure that there is accountability within an entity. Secondly, it is the process or set of processes required to fulfill that expectation. As we de-construct it further then, governance gets more specific in terms of having the ability to impose rules (derived from oversight expectations) applied at the system or service level. Now this is where we start getting back to the discussion I was reading about earlier - the distinction of design-time versus runtime governance is a fancy way of breaking out development and operations lifecycles. In other words, we need to manage both the development of services (to ensure reuse etc) from the actual management of those deployed services.

The problem I see stemming from this discussion is the mistaken perception by those who start buying up SOA governance tools that these will answer all or most governance questions for their enterprise, it won't. And then of course there is likely to be consternation regarding the likelihood of having to retain an arsenal of various governance tools in order to have that comprehensive capability. That is a very legitimate concern, but then again lack of comprehensive Governance processes can be even more costly.




Copyright 2008, Semantech Inc.