(Fr)agile Software Development

Much of our world is made possible by software. There are myriad software systems that manage and move our money, keep track of our health histories, light our homes and offices, and indeed even enable you to read this post. While the sheer scale of accomplishment from zeros and ones flitting about at the speed of light is astounding, the manner in which some of these systems are developed, tested, and delivered raises a few questions.

Over the falls in a barrel. The early years of evolution in software development owed much to needs of the defense and aerospace industries. These were highly mission-critical systems that had to work correctly almost ten times out of ten. A linear process that involved detailed specifications, technical designs, strict coding discipline, reviews, and rigorous testing ensured the delivery of many high performance systems.

A version of this made its way into the commercial marketplace under the broad "waterfall process" moniker. The series of hand-offs, from product management, to architecture, design, development and testing, with intermediate review cycles, hearkened a series of waterfalls as in a cataract. While the process worked well for the most part, it lacked speed. The many steps limited organizations to one or two releases to the marketplace a year. It was difficult to nimbly respond to competitive and regulatory changes. If changes were not included early enough in the cycle, it was tantamount to missing an exit on a tollway, and waiting for the next one.

Sprints around the racetrack. In the 1970's, the automotive industry introduced the concept of "simultaneous engineering", where design engineers, manufacturing engineers, and quality control worked together in teams. As opposed to the linear, "throw it over the transom" model, this engendered both speed and sharing of ideas. That germ of an idea made its way into software as Agile Development. While there are many agile methodologies, the general concept is that specifiers, programmers, and testers work together in short, iterative, "sprints" to produce executable software. Over multiple sprints, complete, ready-to-release applications can be built.

Lost in translation. While agile development has made it possible to release software more frequently, a few challenges have appeared on the way to nirvana. To the agile purists, I will grant that many of these have to do with incorrect interpretation and implementation, and perhaps not because of fundamental drawbacks in the methodologies. The challenges are amplified when you add offshore development where the advantage of co-located teams disappears. They are also most acute when software is developed for General Availability to a large and varied customer base, as opposed to internal use within an enterprise. Here are some of the pitfalls I have observed over the years:

What we have here is a failure to communicate. With apologies to "Cool Hand Luke", one of the main complaints I have seen is, "We don't know what is coming, and when!" We have moved from exhaustive, written requirements to writing nothing down. The refrain is that the sprint teams communicate with each other, and are on top of release content. Some will add that everything can be discerned from documentation within the code. The problem is that there are many stakeholders outside the sprint team, such as sales, marketing, professional services, and support. These people are not adept at reading code, and think in terms of functions and applications, as opposed to individual features. The result often is that market facing groups either oversell or undersell the product (more often the former!).

Who's on first? While sprint teams are cohesive and democratic, the flip side is that it can result in no one at the helm. While the methodologies call for a "function customer" who signs off on software content and quality, this role is often missing in action. Either the role is completely absent, or it is relegated to a Product Manager who is more of a Product Marketer than someone who can go head-to-head with a technician. In the absence of this key role, many cooks jump in to influence the software broth in one direction or the other, resulting in content churn. The process is agile yes, but highly unstable.

Tried and tested. Agile methodologies like test driven development put testing and quality at the center of the process. In practice, however, quality often ends up getting the short end of the stick. The very expectation of agility can compress timelines due to unrealistic promises made to customers. In the rush to "get it out of the door", thorough testing is skipped, and some vendors essentially do their quality assurance on the customer's dime, by continuously band-aiding software at the customer site until it works. In extreme cases, this becomes a license to hack with little regard to version control, belying the very concept of "General Availability". While poor quality is not limited to agile methods, the less rigid process restrictions can exacerbate the tendency in organizations that already have a culture of treating quality lightly.

Customs and traditions. In organizations that cater to customers of varied sizes, the concept of General Availability can be turned on its head. There is often the case of a large customer that wants software customized to meet a unique need. There are very few vendors that have the discipline to examine whether that particular capability warrants inclusion in the software delivered to the general marketplace. The path of least resistance is to include it as a base capability that is "configurable". Over time, the preponderance of configurable customizations makes the software incredibly difficult to implement and support. Again, the lack of a process to adjudicate the "base versus custom" question can result in a multi-headed Hydra, with hidden heads that can appear to bite you when you least expect it.

Distant shores. Every one of the problems discussed explode in complexity when offshore development is involved. The communication challenge now includes time zones, national cultures, and language. The concept of sprint teams working in iterations is predicated on the concept of co-located personnel who can discuss, white-board, and resolve questions face-to-face. Getting this done with people somewhere else on the planet is very difficult, and contributes to hidden costs in offshore development that can obliterate the wage differential in the early stages of the offshore journey. The challenge can be overcome, but it takes special focus and attention to drive out the inefficiencies.

Brave new world. The benefits of agile development have ensured that it is here to stay in most environments. The word to the wise is that getting it to work right involves recognizing the pitfalls, and addressing them involving the right stakeholders. I would not be surprised if many of you recognized your organizations in some of the challenges I have outlined. It is important to recognize that getting software development to work is not just the purview of the programmers alone. Someone said, "War is too important to be left to the generals". If you'll allow the stretch, let me end by saying, "Software is too important to be left to programmers, and methodologies".