The devops handbook how to Create World-Class Agility, Reliability, & Security in Technology Organizations By Gene Kim, Jez Humble, Patrick Debois, and John Willis

• 
It prevents the work center from starting new work, which will 
likely introduce new errors into the system.
• 
If the problem is not addressed, the work center could potentially 
have the same problem in the next operation (e.g., fifty-five 
seconds later), requiring more fixes and work. (See Appendix 6.)
This practice of swarming seems contrary to common management practice, 
as we are deliberately allowing a local problem to disrupt operations globally. 
However, swarming enables learning. It prevents the loss of critical infor-
mation due to fading memories or changing circumstances. This is especially 
critical in complex systems, where many problems occur because of some 
unexpected, idiosyncratic interaction of people, processes, products, places, 
and circumstances—as time passes, it becomes impossible to reconstruct 
exactly what was going on when the problem occurred.
As Dr. Spear notes, swarming is part of the “disciplined cycle of real-time 
problem recognition, diagnosis,...and treatment (countermeasures or correc-
tive measures in manufacturing vernacular). It [is] 
t
he discipline of the 
† 
In some of its plants, Toyota has moved to using an Andon button.
Promo 
- Not 
for 
distribution 
or 
sale

---

32 • Part I
Shewhart cycle—plan, do, check, act—popularized by W. Edwards Deming, 
but accelerated to warp speed.”
It is only through the swarming of ever smaller problems discovered ever 
earlier in the life cycle that we can deflect problems before a catastrophe 
occurs. In other words, when the nuclear reactor melts down, it is already too 
late to avert worst outcomes.
To enable fast feedback in the technology value stream, we must create the 
equivalent of an Andon cord and the related swarming response. This requires 
that we also create the culture that makes it safe, and even encouraged, to 
pull the Andon cord when something goes wrong, whether it is when a pro-
duction incident occurs or when errors occur earlier in the value stream, such 
as when someone introduces a change that breaks our continuous build or 
test processes.
When conditions trigger an Andon cord pull, we swarm to solve the problem 
and prevent the introduction of new work until the issue has been resolved.
†
This provides fast feedback for everyone in the value stream (especially the 
person who caused the system to fail), enables us to quickly isolate and diag-
nose the problem, and prevents further complicating factors that can obscure 
cause and effect.
Preventing the introduction of new work enables continuous integration and 
deployment, which is single-piece flow in the technology value stream. All 
changes that pass our continuous build and integration tests are deployed 
into production, and any changes that cause any tests to fail trigger our Andon 
cord and are swarmed until resolved.
KEEP PUSHING QUALITY CLOSER TO THE SOURCE
We may inadvertently perpetuate unsafe systems of work due to the way we 
respond to accidents and incidents. In complex systems, adding more inspec-
tion steps and approval processes actually increases the likelihood of future 
failures. The effectiveness of approval processes decreases as we push decision- 
making further away from where the work is performed. Doing so not only 
lowers the quality of decisions but also increases our cycle time, thus decreasing 
† 
Astonishingly, when the number of Andon cord pulls drop, plant managers will actually decrease 
the tolerances to get an increase in the number of Andon cord pulls in order to continue to 
enable more learnings and improvements and to detect ever-weaker failure signals.
Promo 
- Not 
for 
distribution 
or 
sale

---

Chapter 3 • 33
the strength of the feedback between cause and effect, and reducing our ability 
to learn from successes and failures.
‡
This can be seen even in smaller and less complex systems. When top-down, 
bureaucratic command and control systems become ineffective, it is usually 
because the variance between “who should do something” and “who is actually 
doing something” is too large, due to insufficient clarity and timeliness. 
Examples of ineffective quality controls include: 
• 
Requiring another team to complete tedious, error-prone, and 
manual tasks that could be easily automated and run as needed 
by the team who needs the work performed
• 
Requiring approvals from busy people who are distant from the 
work, forcing them to make decisions without an adequate knowl-
edge of the work or the potential implications, or to merely rubber 
stamp their approvals