The greatest waste in software: trying to eliminate essential complexities

A large portion of the money wasted in software is born from an illusion: the belief that all complexity can be eliminated. The great software engineer Fred Brooks (winner of the Turing Award, the “Nobel” of software engineering) published, in 1986, an iconic paper titled “No Silver Bullet — Essence and Accidents of Software Engineering.” In it, Brooks made a brilliant distinction between two types of software complexity (essential and accidental), and understanding this distinction is not merely academic talk — it sits at the very foundation of how to invest correctly in Quality Assurance actions within software projects. Ignoring it leads teams to attack precisely the wrong kind of problem.

Every software solution addresses a real-world problem; that problem is inherently complex, and that complexity is not “reducible” through architectures or algorithms. Which is to say that everything related to the domain (industry, business rules, and functionality) has a baseline level of complexity that cannot be removed. These are essential complexities.

Trying to remove these complexities is a thankless task, because it is impossible. There is no shortcut, architecture, model, or process that can do it. Think, for example, of an application that supports a financial institution’s credit approval process: there is no “simple” version of this process that is still valid. There are minimum steps and characteristics inherent to regulatory and business requirements that must, necessarily, be present in the software implementation.

Accidental complexities, on the other hand, refer to complexities “created” by software engineers themselves. This includes things such as poor logical implementations, unintuitive designs, testing processes that do not prioritize functionality, lack of governance in the software process, and so on — in other words, everything that makes the software harder than the domain actually requires. These are the only complexities we can attack, whether through processes or through tools.

Keeping this distinction in mind is “essential” (pardon the pun) when designing a project’s software development process. It means understanding from the outset that certain complexities (attributes of the business domain) will necessarily have to be present in the solution, and that we have no alternative but to (1) try to understand these complexities as well as possible and (2) try to minimize the accidental complexities that arise when translating from domain to functionality, from functionality to software requirements, and from software requirements to artifacts.

Ironically, this context brought by Brooks means that there can be accidental complexities that derive directly from a lack of understanding of essential complexities. In other words, root causes of failures are introduced precisely because we try to create “shortcuts” or “band-aids” to “overcome” business problems. This is where the most serious technical debt is born — and the most expensive bugs.