Allow me to tell a bit of a story.
Back in the late 90s/early 2000s I was a regular visitor and sometime contributor to a website where XP and Agile were first being brought to the attention of the wider community outside the original team that developed the practices. At the time I was steeped in Structured, Top-Down Programming and Getting the Design Right before starting on coding. The practices of XP looked to me, in my inexperience, like cowboy coding. I believed that good programming required a capital-M Methodology and lots of UML with Rational Unified Process. I had previously worked in academia where there was no programming process and everything was slapdash. XP and agile seemed to be a similar “we don’t need no rules here” way of writing code.
By today’s standards of 3-5 years of experience, I was Senior, so I really thought I knew what I was doing. I couldn’t conceive of how software could get done if it wasn’t planned out in advance. Software development was supposed to look like this diagram:
I called XP a “pseudo-methodology”. It looked like chaos.
James Gleick wrote a book titled Chaos, published in 1987. I don’t recall when I read it but certainly by the the late 90s, before or alongside reading about XP.
Let’s stop here and talk a bit about that word, “chaos”, because it’s important, and I think a lot of where I’m going hinges on it. If you look in the dictionary you’ll see definitions of chaos like “complete disorder and confusion”, or “a state of utter confusion“. Gleick definitely did not write an entire book about utter confusion. The meaning of chaos in his book is an alternate definition at the dictionary citation previously linked: “the inherent unpredictability in the behavior of a complex natural system”. A very small change in conditions at one point in time could result in very large and unpredictable changes in behavior later.
You could not remove a single gram of sand from its place, without thereby, although perhaps imperceptibly to you, altering something throughout all parts of the immeasurable whole
The Vocation of Man, Johann Gottlieb Fichte, W. Smith tr.
A related concept is: emergence. Emergence, or more fully, emergent behavior, is like a flock of birds. Watching birds move in a flock might lead to the idea that there is a leader, but there is not. Because each individual acts according to the same rules, in the same environmental conditions, we see flocking. Schools of fish are similar. This is decentralized behavior.
I’ll have to go into chaos, emergent behavior, and complex systems in another post, later.
But sometime before March of 2001 I had an epiphany. I won’t go into the whole story but my realization was the when the work gets to the coding and testing phases all but the most trivial projects will discover things that hadn’t been accounted for in the requirements and analysis phases. It also became clear to me that in anything but the smallest projects, in the time between requirements and coding, something will change, invalidating at least some of the decisions made in the requirements phase. What I didn’t know before then was that Royce, in his paper that laid out what would be called the waterfall model of software development, including the graphic previously shown, rejected the concept. He wrote, “I believe in this concept, but the implementation described above is risky and invites failure.”
required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. … In effect the development process has returned to the origin and one can expect up to a 100-percent overrun in schedule and/or costs.
Winston W. Royce, 1970
What I think was happening when XP came along was that programmers realized that the systems they were working on had become too complex to reason about. The systems were, in Gleick’s terms, chaotic. The concepts of chaos, emergence, and decentralized control informed XP: The practices of XP are simple, and out of the simple practices arise complex behaviors. Simplicity, coupled with communications and feedback, make XP a programming discipline expressly, and perhaps uniquely, developed and suited for writing complex software of the sort that came about at the turn of the 21st century, in the kinds of organizations that became dominant in the software profession.
You might say I became an XP convert. I did have a tendency to get personally invested in my world view, and at my next job I became the Change Agent that was going to bring XP/agile to my employer. And you know what? I made a lot of mistakes and misunderstood a lot of things, but it sort of worked. My co-workers started thinking about things differently. They stopped being quite so annoyed when, halfway through the project, the businesspeople decided they wanted something kind of different from what they had said up front. Programmers and non-programmers started to look at the work-in-progress together and decide if things were on track, if the requirements were right, if everyone had come to a shared understanding of what the requirements really meant.
Mostly, I started to accept that users and stakeholders only sort of know what they want, that communicating complex ideas between people is hard, and that realizing ideas in software is harder still. The only way to get software that works as desired is to build incrementally, share progress, and iterate.
It’s been 20 years since the Agile Manifesto, and I have met two or three of the original signatories. In the interim, I’ve come to agree with observers who say agile, as done today, feels like chaos. Agile, done poorly, is chaos. And by that I mean “a state of utter confusion”.
The Wild Fox Den 