There is No Process to Prevent Stupidity

IT development processes and governance can only minimize the risk of doing something really stupid and damaging to your business. But even the best development process is never a game-changer, and almost always results in lower net productivity, and can even contribute to greater alienation and misalignment of IT from business operations.

After a project train-wreck, after all the blame has been assigned, miscreants fired, and bills paid, executive management will naturally ask of survivors: "How will you assure me this never happens again"? And the answer almost certainly will be: "We will improve our processes and take fewer risks". Re-engineering development processes is a natural, human response to some act of profound stupidity.

To explore this topic a bit further, let's compare with current events in global finance. (There are many good analogies between the worlds of finance and IT, in part, I would hazard, because both deal in abstractions -- money and data, respectively -- that are the drivers of a modern economy. Like finance, IT deals in portfolios of projects and operations -- analogous to investments and accounting -- with varying risk profiles, based on a symbolic model of the world. To my thinking, comparisons between software and more physical disciplines, such as civil engineering, usually result in false analogies and misleading conclusions.)

• Many, many people have behaved stupidly over the past eight years, taking on excessive leverage in operations and paying dearly for risky investments. These behaviors culminated in an epic catastrophe in September of 2008, triggered by the collapse of Lehman and AIG. During the ensuing months, there has been a steady and growing apportionment of blame among CEOs, their supposed regulators, boards of directors and political supporters.
• Soon, the "fix", in the form of improved market regulations, will be proposed by the Obama administration.
• Free-market enthusiasts of the finance business worry (with some reason, yet too much hysteria) the new regulatory cure may be worse than the disease, reducing wealth-creation by inhibiting innovation in financial instruments. These observers also assert that there will be unanticipated side-effects (and here they are almost certainly correct.)
• At the same time, critics on the political left rails that the champions of unfettered markets will be the same folks re-engineering the new regulatory regime, endangering prosperity for the masses and continuing to benefit the wealthy few who can afford to take risks. The political left, however, includes fewer finance practitioners, and therefore tends to criticize from a position of domain-knowledge ignorance.
• President Obama has characterized the emerging regulation as follows: "The choice we face is not between some oppressive government-run economy or a chaotic and unforgiving capitalism," Obama said. "Rather, strong financial markets require clear rules of the road, not to hinder financial institutions, but to protect consumers and investors, and ultimately to keep those financial institutions strong."

When I assess an IT project disaster, I am firmly of the President Obama persuasion:

• Regulation, in the form of a defined development governance process that properly assesses & manages risks, is now a necessity. IT is now too essential to business, and, like a good insurance policy, is a necessary cost. The free-market approach of no-defined process which just leaves the practitioners "free to choose" is no longer a responsible decision.
• But like any business cost, process & governance needs to be light-weight and not impose excess burdens on practitioners. I would never recommend the programs pitched by the larger consultancies, to create a "culture of risk awareness", which will more likely shut down all innovation, unless like the political left-wing, you desire a total cessation of risk taking. Very few applications have the dependability requirements of a nuclear power plant or weapons system.
• Deeply complex processes can make it even more difficult for IT practitioners to relate to business operations, because it forces IT to adopt language and behaviors alien to the business per se.
• Lastly, like the global finance system, corrections in governance have to be performed by the experienced practitioners that understand the complexities of modern IT. There is no turning back the clock to a simpler time, and IT problems cannot be fixed by outsiders, only by the people within.

Almost from the dawn of computing, the challenges of IT abstractions, complexity, and growth has given birth to waves of engineering process enthusiasms and literature. I am of Fred Brooke's "no silver bullet" camp (c.f. http://www.lips.utexas.edu/ee382c-15005/Readings/Readings1/05-Broo87.pdf). My summary conclusion is: There can never be one right development process, standard model or methodology, for any & all businesses. Managers can only progressively improve the alignment of IT with a business through iterative projects, continuous learning, light-weight processes and above all, better hiring.

One last thought: When dealing with the aftermath of a project disaster, managers should remind themselves that the key determinant of technical outperformance is – and always has been -- the quality and talent of your employees. The relative productivity differentials between individuals in the technology industry remains as wide today as when it was first measured (see http://blogs.construx.com/blogs/stevemcc/default.aspx, and many others). Better hires, properly guided, will develop better processes to protect your business, resulting in more productive, less expensive and more responsive IT.

Comments

[bob the benevolent]

Be careful when you say 'stupid'. Some of your readers may think you're talking about them.

Also, be careful about conflating project failure and process structure and productivity variability and 'stupid'. Each of these topics could be a great blog post.