The Elephant Eaters

New Year, new scapegoat?

2009-01-05T09:44:00.002Z

Dilbert being promoted from Legacy Systems Troll to official Scapegoat felt so familiar to those of us that work on complex systems integration projects that I just had to reproduce it here...

The cartoon may also provide us with a new slogan - "Brownfield Development - the premier approach for battling Trolls and Scapegoats". You never know, it might bring a few people over from World of Warcraft...

I hope the New Year is a happy one for all our readers!

The compost heap just below the Elephant...

2008-12-15T15:49:00.002Z

Brownfield is all about coping with complex existing environments, so today's Dilbert cartoon may feel relevant...

SOA and Brownfield Development

2008-12-05T11:53:00.002Z

Last week I had an interview on Brownfield with Rich Seeley of SearchSOA.com. IBM does get wonderfully paranoid about interviews, so I had to be on best behaviour and try and try and remember my media training...
As it turned out Rich asked lots of insightful questions and I managed not to destabilise the IBM share price, so on the whole things went very well. The article is here and if you register, you can download a Chapter of our book.

Introducing Brownfield - Part Two

2008-10-01T15:41:00.003+01:00

The underlying conceptual architecture of all Brownfield tooling is known as VITA. VITA stands for Views, Inventory, Transformations and Artifacts. In a VITA architecture, the problem definition of the target space can be maintained as separate (though related) native "headfulls" of knowledge known as Views. The core advantage of a View is that it can be based on pretty much any formal tool. Brownfield does not impose a single tool or language on a problem space – a core tenet is that the headfulls continue to be maintained in their native forms and tools. Views can currently be imported from a wide variety of sources including code, configuration files, UML, XML, DDL, databases and spreadsheets.
These native Views are then brought together and linked into a single Inventory underpinned by semantic technologies. The Inventory is analyzed via automated, semi-automated and manual techniques to create further viewpoints at varying levels of abstraction. Unlike Greenfield abstractions, these are built from the ground up, not top down. Extracts from this Inventory are then used with a series of Transformation tools which work at varying levels of abstraction to produce the Artifacts that the solution needs. Conventional model and pattern driven development techniques are used to complete the forward engineering of the new solution. In addition to code and models, this technique can also create abstract, but now precise documentation such as system contexts or component models. The technique has even enabled multi-layered, precise, yet easy to understand visualisations to be automatically created and shared in new real-time community environments like Second Life as shown in the video below:

The rapid cyclic nature of the discovery, re-engineer, generate and test cycle used in Brownfield Development means that solutions can be refined iteratively in terms of their logical and physical definitions as more of the constraints become known and the solution architecture is refined. This results in development acceleration, improved solution quality and cheaper defect removal. In addition, the ability to identify smaller coherent elements that are amenable to change means that wholesale replacement or transformation of systems is often not necessary, enabling incremental change and a gradual reduction in overall complexity.

Complex Landscapes

2008-09-15T10:03:00.001+01:00

I have spent the last few weeks reviewing some application management support deals. It seems to me that business is being driven by short term goals rather than long term – and that approach is costing them dearly in the long term, both in terms of flexibility and money.

When a customer buys a hardware solution, one of the major criteria for the choice is the total cost of ownership – it will be cheaper, more economical, or whatever to run it on the latest, fastest more efficient chipset. However it seems when it comes to applications, speed to deploy and cost to deploy are the main considerations. Businesses look at the total cost of acquisition rather than the total cost of ownership.

Surely the quicker and cheaper to deploy is the best in any case? Well normally not. Ill thought out solutions often increase the complexity of your estate rather than reducing it. Ultimately this will end up costing you more in terms of money and flexibility. Let’s take a simple example of a partial Service Orientated Architecture (SOA) implementation that would wrap, rather than re-engineer, the legacy system to expose it’s offerings as services to the rest of the estate. It certainly makes it easier to interface to the legacy system, but at what cost? The solution has solved something by adding another layer (or more) into your estate. You will still have to run the legacy system, still have to it to maintain, you will also have to maintain the new SOA services. Your estate has become more complex.

I am not stating that SOA is a bad thing, it is just an example. There might be good reasons for doing the example I gave above. Perhaps it allows you to isolate the legacy system while you develop a suitable replacement - so that it is easier to replace, in which case this will make that transition easier. This would be a good use of the service – as a means of transitioning. However this is not the normal case in our world. Solutions are performed piecemeal with little long term strategy – what will be introduced as a strategy will be overturned in coming years when that money is required on even more functionality.

So we continue to make our estates harder and more difficult to maintain we increase the complexity of our estates, and we do it often by adding layers of technology which it is ‘supposed’ to make it simpler. In fact it does the opposite.

So what is the solution? Well it is reasonably obvious, we need to stop trying to hide things that are difficult to understand behind a curtain and get to understand them. It is odd that in no other field of business would a businessman say that area is too complex to understand – I’ll just ignore it! Yet the business – IT gap makes that seem acceptable and normal.

The bottom line is simple, just like one of the Brownfield beliefs – embrace complexity rather than ignoring it.

Do this and your world will become simpler.

Introducing Brownfield - Part One

2008-09-08T23:44:00.004+01:00

Now that the blog is being relayed through to Amazon, it may be worth recapping what Brownfield and Elephant Eating is actually about. Brownfield came about because most organizations in the twenty-first century have an existing, very complex business and IT landscape. The introduction of 3GLs and families of computers such as the System/360 enabled the complexity of these landscapes to accumulate. In our fashion-driven industry, these programs have been augmented by the adoption of on-line processing, client server, object-oriented programming, the exploitation of the Internet, and so on. Each of these movements has had its own strategy for dealing with complexity, but in reality, each one added to it. Today’s IT systems are typically poorly documented and their maintenance is dependent clusters of knowledgeable ‘system experts’. Each new fashion has introduced its own experts, resulting in the costs of maintaining the status quo now absorbing 70%-90% of IT budgets. The budget for innovation is being squeezed between increased regulation and maintenance costs. As a result businesses are finding it harder to migrate to competitive, integrated architectures such as Web 2.0 and SOA.

To move to these new architectures and achieve demanding business targets this existing complexity often needs to be discovered, understood and harnessed. However no-one can understand the whole beast, so with traditional greenfield methods we rely upon the knowledge of many people to understand the whole. This means the environment for most big IT projects is an immense communications and co-ordination challenge because we are entangled in an almost uncountable number of poorly understood environmental constraints.

Not surprisingly we have observed that an increasing proportion of the effort of developing new business capabilities is spent on understanding and integrating with the existing complex system and business landscape rather than delivering value. It has been observed that up to 75% of overall project effort is now spent on software integration and migration rather than new function. Many of these large integration and migration projects fail expensively and publicly due to poorly understood complexity.

Engineering change in such environments has many parallels with the concerns of the construction industry in redeveloping industrial or contaminated sites. They are full of hazards, unexpected complexities and tend to be risky and expensive to redevelop. Today’s big IT projects are usually executed on such “contaminated” sites, where you need to be careful where and how you build; a change in one place can ripple through to other systems in unexpected ways. Such sites are more ‘brown’ than green. The accumulated complexity of IT environments has made them “Brownfield” sites.

Brownfield development recognises that any new software architecture must take into account and coexist with live software already in situ. It does this by making a number of improvements to conventional software engineering practices.

Current “Greenfield” tooling and methods use early, informal and often imprecise abstractions that essentially ignore complexity. However the devil is always in the detail. Early, poorly informed abstractions are usually wrong and are often detected late in construction, resulting in delays, expensive rework and even failed developments. A Brownfield oriented approach embraces existing complexity and focusses on understanding, communicating and harvesting it to reliably accelerate the overall solution engineering process. Such an approach can enable phased, incremental change where this was previously thought impossible.

Brownfield Development, starts by harvesting a detailed knowledge of the systems, services and data in the immediate vicinity of the solution under construction and then using that reverse engineered knowledge for forward engineering of the new solution. Rather than taking the conventional approach of starting with a Conceptual Model and driving down to Platform Specific Models and code generation, Brownfield starts by harvesting code and other existing artefacts and uses patterns to formally abstract upwards towards the Architecture and Business tier. It then uses all of these new layers of knowledge to perform the necessary transformation of the existing environment.

In the next post we’ll cover how we go about doing this with tooling using the VITA tooling architecture.

The Future of Application Development

2008-09-05T12:48:00.003+01:00

Apologies for the hiatus in the blog – we don’t get much summer in the UK and holidays and work commitments have prevented any recent updates. Anyway it’s been an interesting few months for Brownfield.
First of all I was the leader of a track for IBM’s Future of Application Development conference held in Somers (pictured above). Somers like the nearby IBM HQ Armonk is an amazingly large and attractive campus, completely unlike our locations in Europe (except IBM Hursley of course)!
IBM frequently has internal conferences run by its Academy of Technology to bring together technical leaders from across IBM to share thoughts and best practices. Here's me (third along, inspecting Peter Coldicott's shoes for some reason) on a panel with some very distinguished experts talking about what we should do about legacy applications.

This conference was unusual for two reasons:
First we were looking into the future rather than sharing best practice
Second we were joined by senior IT staff from HSBC, including its Head of Architecture, Peter Cook.
We also had some very august speakers including:

Grady Booch (co-creator of UML, OO and pattern visionary, who joined us from Second Life)
Martin Nally (CTO of Rational)
Linda Northrop (Director, Carnegie Mellon Software Engineering Institute)

Grady pictured below in Second Life (where he is nearly as handsome as real life) is part of the Brownfield Study and kindly name checked Brownfield again (thanks Grady!).

Unfortunately as the conference is still communicating its findings to the sponsoring Executives inside IBM, I can’t discuss them here. But I can tell you that the Brownfield presentation was (narrowly) the highest scored presentation of the conference!

Brownfield's everywhere

2008-06-16T07:25:00.003+01:00

Last week I was privileged to present to the UK’s National Research and Training Initiative’s Large Scale Complex IT Systems (LSCITS) project. LSCITS is a £14.6M programme looking addressing some of the problems of complex systems and creating a new breed of Engineering Doctorates (EngD) skilled in dealing with them.

To quote their own Initiative Rationale:

The motivation for the LSCITS Initiative is the on-going growth in the size and complexity of information technology (IT) systems. Our ability to develop, maintain and manage such systems is falling behind the growth in their complexity. There is a high risk that we will find ourselves reliant on IT systems that we don’t fully understand and that we cannot effectively manage.

The roots of complexity in IT systems are their increasing size; the increasing involvement of many different organisations in their construction and use; and the increasing rate of business and social change that they have to accommodate. To manage and control complexity, we need better technical tools and methods of system development.

People after my own heart!

Presenting alongside me were representatives from Rolls Royce and BT. Each company had been asked to provide its own perspective on the problems of systems complexity and where they thought LSCITS might be able to help.

Extraordinarily, despite the fact the companies were presenting from very different backgrounds (complex aero-engineering, national telecoms provision and commercial system development) we all had very similar things to say. In particular:

We all suffered from a lack of rigor caused by lack of formal tooling use. Even when tools were available, people were not happy using them. As a result the level of documentation was poor, reducing the opportunity for reuse of existing assets.
People were much happier using tools that they were familiar with (Word, Excel etc.) it was unlikely that we could move large numbers of people off them, so we need a better way to control and share information using such informal tooling (one representative lamented the failure to establish a single source of truth almost verbatim from the Eating the IT Elephant book!)
There was an acknowledgement that no-one can start from a Greenfield perspective. Incremental change and re-engineering is the only cost effective route going forward. No-one can afford a lengthy design or implementation period for a new system or product.
Not only that, but the self-contained design and engineering teams are a thing of the past too – no-one can afford to create everything in house, so components and other elements need to be sourced from outside the organisation. This causes all kinds of problems when the capabilities and limits of these sourced elements are not known.

All in all, we were all looking at new degrees of both organisational and technical complexity. Communication of that complexity was the root of each of our problems. Perhaps Brownfield is a pattern for success with all complex IT systems, not just for use by Systems Integrators?

Where are the requirements hiding?

2008-06-03T16:04:00.003+01:00

I've just been on a architecture Masterclass - which was nice. Unfortunately the class did little to dissuade the attendees that the IT industry has more than two types of requirement. It stuck to the line that we basically have functional requirements and non-functional requirements.
The Functional requirements are probably the best known. They tell the designer what the system has to do, usually from the perspective of the end user. Nearly everybody agrees these days that Use Cases are a pretty good way of capturing these requirements. I’ve used some alternative more technical approaches on some big projects, but we’ve nearly always had to create use case like perspectives anyway, just so we can communicate to back to the users. If you can get away with agile Joint Application Development techniques where the users and developers sit down together in small grounds then you’ll probably do even better, but often the high burden of detail, testing and sheer size of the projects we deal with means that such an approach gets ruled out.
The second form of requirement – the non-functional requirement are rather more poorly treated by tooling and there are few widely accepted good practices. The non-functionals tell the designer what the characteristics of the solution should be. Not what it has to do, but how it has to do it. The system must be always available. The system must provide sub-second response times 100% of the time. These are just two of the often impossible and conflicting sets of requirements that we deal with as lead architects.
NFRs are the requirements that make designers proud. You’ll rarely hear them boast that the system they created had the best implementation of a “Find Customer” Use Case, but they’ll never stop crowing that it could support 80,000 concurrent users whilst enabling a full recovery from a plane landing on the data centre within fifteen seconds without the loss of any data…
The odd thing is, the more projects I do, the more I am convinced that we miss most of the requirements that we actually need to deal with. They don’t fit into the functional or non-functional space. They are called constraints. Constraints outnumber the other requirements 10:1. They are all those Entreprise Architecture standards you need to conform to, they are the existing data you need to migrate, they are the interfaces to the existing systems, they are the budget, they are the timeframe. Basically they are the difference between the system working in its environment and context, or not. They are the Brownfield aspects of requirement. The NFRs and Use Cases talk about the system as if it lived on its own little planet. They define the boundaries and shape of an isolated bubble. What they conveniently ignore are all the provisos that actually make the system hard to deliver. It’s here that we see implementation projects fail: they adopt a domain model which is incompatible with the rest of the organisation and make integration impossible; they ignore the real complexity of the interfaces with other systems, assuming that an interface is just the definition of a data structure, they ignore the fact that the data used in the system is now duplicated in multiple places.
What’s worse is that these things are often found out late in the project lifecycle resulting in unexpected expense and delays. Why don’t we just accept the fact that systems have to be deployed into complex environments and do what any respecting building architect would do: commission a Site Survey and use its constraints to influence the design? It’s the Brownfield way, you know its right.

The elephant is in the room!

2008-05-19T15:43:00.004+01:00

It's a big week for Kevin and I. Our book Eating the It Elephant: Moving from Greenfield Development to Brownfield is finally available in the US and the UK. It's been eighteen months of hard work from the submission of the proposal of the book to it being real. For a book that talks about re-engineering IT and coping with complexity whilst taking in semantics and business attractors, we've been reassuringly told that its easy to read. My kids even think the cartoons are funny. So if you feel the need to buy a book with a particularly attractive elephant on the cover, ours is the one to go for. If you also happen to work in complex IT environments the contents might be worth reading too!

Is there any more room in the bathtub?

2008-05-19T15:26:00.002+01:00

I’ve been hearing some disturbing messages from my customers over the last few years. Some of them are beginning to regard IT as a burden and a blocker of change in their business. They complain that the systems they have are costing a lot to maintain and they are difficult to change. Rather than being the bright new thing enabling business change, IT is being seen as a dirty money grubbing blocker on their business. OK, I may be exaggerating somewhat, but if you talk to customer procurement people long enough you begin to believe them!
But they may have a point. I’ve been researching how much big companies spend on operating and maintaining their existing IT systems versus how much they spend on creating new business capabilities. A Forrester report I saw some time ago suggested that the maintenance budget was expanding and the amount of money being spent on regulatory compliance was also growing. The end result? A squeeze on the amount of IT spending available for business innovation. Now the annual squeeze was only a small incremental effect, but it was measurable.
The installed base of IT complexity in already complex environments is gradually but perceptibly growing. Costs of maintaining that complexity are increasing.
Adding a new channel to a system will add complexity. Layering SOA on top of existing systems will make them more useful and reusable, but adds to the overall complexity. Adding new function to an existing system without taking out any existing function will add to its complexity.
The IT industry stopped throwing the baby out with the bath water over 40 years ago when it became possible for business code to become portable without change between business systems. I suspect these days, our metaphorical industry bath tub is therefore overflowing with all ages from infant to middle-aged hippie. Figures I’ve seen suggest a bank or national government department will have 500,000 function points of installed complexity. That corresponds to 4,000 person years of delivery.
And that’s where the industry could hit a downward spiral. As IT environments grow and get more complex, we spend more time integrating and migrating from old to new. The number of potential (and actual) connections between systems gets larger. The testing burden grows and the reduced budget for innovation gets spent on integration and migration.
How bad is the problem? Well the very largest IT operations now spend 90% of their annual budget on the status quo. That’s only 10% left for innovation. Not only that, but we’re now beginning to see big projects where 60% of the overall costs are attributable to integration and migration, rather than the delivery of new function. With some basic high school maths that means only 4% of the IT budget is actually going to make a difference to the business. Ouch.
Now, not everywhere is as complex or hard as the kind of environments we’re talking about, but there is a danger that as time goes by the same increase in complexity will work its way further down the stack.
We need, as an industry, to do something about it and soon. We need to simplify, simplify, simplify – and that doesn’t mean sticking another layer of supposedly simplifying complexity on top of what we’ve got. If we do, the IT may make it back to being the shiny lithe hero of the story, rather than the obese unwashed villain.

Why do big IT projects fail?

2008-05-12T11:20:00.003+01:00

It’s over thirty years since The Mythical Man Month and we’ve made huge strides in the IT industry in formalising methods and tooling. Nevertheless looking around, we still see projects failing. Why is this?
If you listen to industry leaders like Grady Booch, during the 9th Annual Turing Lecture, they will remind you that building complex systems is intrinsically hard. You’re engineering with peoples’ ideas - fundamentally abstract concepts. You’re then turning those rather amorphous things into instructions that can tell millions of tiny super hot switches whether to turn on or off. The gap in perspective is immense. We have an ever increasing number of layers and technologies to help us make the translation, but each layer adds some complexity to the ultimate solution.
Business has got more complex too, of course. The ability to put complex decisions and processes into computer systems makes it possible, for example, to assign a mobile phone to an individual in minutes. Each mobile that’s commissioned takes thousands of individual steps which need to be co-ordinated. It’s not just the phone and the network; it’s the billing and the accounting, plus the customer support and service. If it was down to a human, it would never happen reliably – and certainly not in minutes. Business is complex and technology has enabled it to become ever more so.
Therefore what we do in engineering big systems is hard and in many was it’s a minor miracle that any very large system gets delivered.
The troubled projects that Kevin and I see are failing mostly because of the need to co-ordinate people and the intrinsic complexity of what those people are trying to achieve. So my hypothesis of the day is: projects don’t tend to fail because of technology flaws or poor solutions; they fail because we don’t contain the complexity or enable it to be communicated.

Drawing Parallels to Science

2008-05-08T10:17:00.003+01:00

Brownfield Development approaches the understanding of computers in a manner that is not all that new in terms of science but is extremely new in terms of computers. Brownfield Development looks at the smallest computational building block and develops a detailed understanding of the system by fitting each piece of the computational puzzle together. This is not all that different from Chemistry where atoms (which are made up of even smaller building blocks) are combined into elements and elements under very specific combinations and parameters make up molecules as well as compounds. In the same way, Brownfield Development, is based upon the smallest computational unit which is called a triple. Triples are made of 3 elementary components a Subject, Predicate and Object. When triples are combined they will eventually result in the representation of complex computer system.

Everything in nature can ultimately be broken down into atoms in the same way all computational systems can be broken down into the triple building blocks that make them up. What is interesting is that molecular diagrams look very similar in structure to a diagrammed ontology. Please see the 2 images below, one is a graphical representation of an ontology, the other is a graphical representation of a molecule...what is interesting to note is the similarity in structure between the two diagram forms. Granted these are mankind’s attempt at visualizing things that are so complex that we're still learning new things about them today but that is the whole point of Brownfield, embracing complexity such that we as human being can better utilize the systems that we work with and that operate almost all facets our society today.

Figure 1 - Ontology Visualisation

Figure 2 - Molecule Visualisation

Computing as a whole could benefit from looking at the other sciences and gleaning from the years of work that have already been done. The natural world is in my opinion the ultimate example of engineering at it's best. The following are links to wikipedia pages for a couple of other interesting areas of science that might have some interesting parallels with Brownfield development as well.

Brownfield Overview - in 2 minutes!

2008-05-06T00:11:00.002+01:00

The following YouTube video servers as a brief introduction to the Elephant Eaters way of doing things. Special thanks to Kevin MacLeod http://incompetech.com/m/c/royalty-free/ for the music.

Introducing Brownfield

2008-01-14T23:00:00.000Z

Brownfields are real property, the expansion, redevelopment, or reuse of which may be complicated by the presence or potential presence of a hazardous substance, pollutant, or contaminant. Cleaning up and reinvesting in these properties takes development pressures off of undeveloped, open land, and both improves and protects the environment."
Its a rare client these days that can offer an architect a greenfield site to build on. OK, so there are a few new startups, but most of our clients have a 20 year history of IT investment. That investment provides real business advantage, but increasingly the sheer bulk of IT that has built up over the years has evolved in such a way that business flexibility begins to be constrained.
This blog is all about changing the way in which we address these situations and begin to accept them as the norm, not the exception. The metaphor we use is brownfield, which is an apposite metaphor in more ways than one when we find ourselves knee deep in complexity...