2024-10-15
This post was significanly inspired by several talks and interactions at RailsWorld; most directly Eileen’s excellent talk. go check that out for another angle on this idea.
As software projects grow, they go through at least three distinct phases, and I mean “phase” here in vague analogy to phases of matter.
All projects begin in the “Human-driven” phase. This is where the whole system fits in a single person’s head, and often even in multiple heads concurrently. The day to day work of these systems involves individual people making sweeping changes across the codebase. As it grows, this gets harder and more painful to do, but the barrier is just inertia, not capability to understand the system.
Far at the other end, in the “Culture-driven” phase, a project has far exceeded the ability for any single human to hold the whole system in their head at one moment in time. Individuals may understand subsystems well enough, but no one can reason in any breadth and depth about the whole system. In the Culture-driven phase, a project is beyond the scope of a single human, and broad changes can only be effected by cultural changes; for example, by creating a “shitlist” of violations of some new rule, and socializing the responsibility to resolve them.
In between the Human-driven and Culture-driven phases is the “Zone of Heroism”. This is where, for the most part, the Culture-driven phase has started to crystallize, but there are still moments of brilliance where a highly-skilled human with the right context and a strong motivation to solve a problem can effect change across the entire project as a human-scale activity, not a cultural one. This is a gradient, of course: the crystallization into Culture-driven phase doesn’t happen all at once. Some subsystems or aspects of a project will remain more amenable to Heroism even as the rest has become an unknowable morass. But eventually, given enough time and effort, all systems enter the Culture-driven phase.
A person is smart. People are dumb, panicky, dangerous animals and you know it. –Men in Black
When a human can fit a whole domain in their head, they can turn it over, shape-rotate it, use insight to find new, better designs. Cultures aren’t good at this.
Cultures are networks of humans. Their “thought” is mediated by language, usually text; and the available ingress bandwidth of the individual nodes implementing the culture - humans - is pretty low. Cultures aren’t able to do high-bandwidth synthesis or insight, because they can’t propagate concepts around their network nearly as efficiently as can be done inside a single human mind.
What to do with this model? Given the above, it goes without saying, I hope, that remaining in the Zone of Heroism is preferable to fully sliding in the Culture-driven phase. So how can we do that?
Three major ways:
In his keynote, DHH invoked the idea of a “Brain Budget” for Rails; that Rails should limit the amount of mental space it consumes. This philosophy is quite omnipresent in Rails. While large apps do frequently push this past its breaking point, as Eileen pointed out that same day, Rails is intentionally designed to keep your application relatively free of incidental complexity.
Complexity is the ultimate (semi-)quantifiable that pushes a project around between these phases, and restricting it delays the onset of the later phases. Better abstractions delay this onset.
Around the boundary of the Zone of Heroism and the Culture-driven phase, a common experience is for a human to know exactly what they’d like to do but experience overwhelm at the amount of effort involved and abandon the effort. Usually the threshold where this happens could be pushed back with better tools. That may imply AI assistance, or something as mundane as a codemod-style tool. Whatever the tool, increasing familiarity with these across an organization effectively pushes out the frontier of the Zone of Heroism.
Large codebases frequently suffer from the problem of being somewhat illegible even to frequent contributors. Overwhelm can happen just because it’s difficult to chase down where the thing of interest is happening. Making a codebase easier to navigate makes it easier to contribute to in trivial cases, but also, a sensibly-structured codebase maps more readily into a mental model that can fit in a single head. At some point, though, the total number of nodes in the conceptual tree is just too high to fit in a head. I’m really curious about the potential for well-prompted or finetuned AI tools to push the frontier of wayfinding well past the current human limits.
Delaying the onset of the Culture-driven zone is good. The acts of Heroism in the intermediate phase are often driven by attempting to prevent the onset of the Culture-driven phase. Once that phase is entered, there’s less of a constant force toward maintaining tractability.
It may be possible to progressively de-crystallize a project that’s entered the Culture-driven zone through some combination of: