Maximizing Leverage in Software Systems

Published 2025 July 7

Introduction §

One of the great joys of childhood is having a convenient place to swim. In my youth, that place was a perennial irrigation ditch that flowed through my front yard. Unfortunately, the water level was rarely satisfactory, which was just the pits. It didn't take long for me to come up with a solution.

I would dig through the wood scraps in the garage and find a perfectly sized piece of plywood. This would be wedged against a narrow cement channel at the end of my yard, effectively damming the ditch. The water rose and so did my elation. In fact, my joy was doubled. Now I had a place to swim and play engineer!

Irrigation ditches are filthy and filled with hazards. I am confident that the folks who dug the ditch did not intend for it to be turned into a swimming hole; nor did the farmers whose expensive pumps and precious crops were downstream. I intervened in the system so that it worked for my purposes—consequences be damned!

Part of systems thinking ¹ is finding points of leverage so that you can introduce change. One challenge of systems thinking is knowing how to correctly introduce change. If you introduce imbalance to a system it will push back or adapt to achieve balance. This is called compensating feedback ². That's why change is so hard. If we fail to understand a system holistically, we run the risk of the system overthrowing any change we wish to impose.

In the programming world, we are surrounded by complex systems that we have little or no control over. One of the joys of programming is finding ways to work within or around the limitations of the current design of a system to produce a desired effect. This is so engrained into the culture of software development that I don't think many people realize that they are actively thinking in systems. I think it's worth highlighting a few instances where software developers were able to find leverage and produce meaningful results.

Unix Pipe §

Our first example of systems thinking takes us back to Bell Labs and the invention of the pipe | operator ³. Doug McIlroy had this idea of connecting programs together as if they were two ends of a garden hose. The data produced by one program would "flow" into the receiver of another. He kept floating the idea by Ken Thompson who eventually acquiesced and added the pipe system call to Unix and its subsequent shell syntax. In his book, Unix: A History and a Memoir, Brian Kernighan relates:

Ken and Dennis [Ritchie] upgraded every command on the system in a single night. The major change was to read data from the standard input stream when there were no input filename arguments. It was also necessary to invent stderr, the standard error stream. The standard error was a separate output stream; error messages sent to it were separated from the standard output and thus would not be sent down a pipeline. Overall, the job was not—most programs required nothing more than eliminating extraneous messages that would have cluttered a pipeline , and sending error reports to stderr.

The addition of pipes led to a frenzy of invention that I remember vividly...Everyone in the Unix room had a bright idea for combining programs to do some task with existing programs rather than by writing a new program.

As a daily user of pipes, I think I take for granted its utility. There are so many tasks it makes trivial that I can hardly imagine a world where it doesn't exist. For example, the other day I wanted to copy a username from a .env file to my system clipboard. The format of the file was similar to this:

USERNAME=foo
PASSWORD=bar

So I came up with a simple oneliner on the fly.

grep USERNAME .env | cut -d'=' -f2 | xclip -sel clip

Without the pipe, I would have had to redirect the output of each command to a temporary file and then redirect the temporary file to the next command. The ergonomics of this approach aren't great.

grep USERNAME .env >tmp1.txt
cut -d'=' -f2 <tmp1.txt >tmp2.txt
xclip -sel clip <tmp2.txt
rm tmp.txt tmp2.txt

The overhead of my oneliner just got a lot more complicated and messy!

Doug McIlroy was able to see that while each program provided a lot of utility on its own, it could be expanded in how it accepted it input. The beauty of this is that the underlying function of a program didn't change. cut still has the same effect whether it's input comes from a file or stdin. The modification worked with the grain of the system. By slightly reframing the context in which a program works, the designers of Unix maximized the leverage of their system.

Let's say that before you had pipes that for n programs in a data processing workflow it required n - 1 temporary files. With the introduction of pipes, all of your temporary files are eliminated in favor of in-memory streams. n - 1 temporary files becomes 0. That's a good bargain!

To be fair, in today's workplace, I completely understand this type of idea being ignored. The program already works the way it was designed! The business cycle demands that we push a feature or product and then move on. That's okay. No one should be ashamed of having a working product. You did your job. Mission accomplished! But if we're paying attention, systems often have emergent properties waiting to be reaped.

Doug McIlroy first advocated for the idea of pipes in 1964. It wasn't until 1972, 6 years later, that pipes were actually implemented. In hindsight, it's obvious that pipes are a good idea but arriving at their implementation took quite an effort. It speaks to the complexity of systems. It takes some time just staring at the problem before a practical solution materializes.

htmx §

One of the early ideas about how the internet should work was REST ⁴. REST, which stands for Representational State Transfer, was described by Roy Fielding in his doctoral dissertation ⁵. REST is an architectural style for enabling efficient data transfer in a distributed hypermedia system. This description of hypermedia from Wikipedia ⁶, distils how hypermedia is used in the context of a REST architecture.

Hypermedia is used as a medium and constraint in certain application programming interfaces. HATEOAS, Hypermedia as the Engine of Application State, is a constraint of the REST application architecture where a client interacts with the server entirely through hypermedia provided dynamically by application servers. This means that in theory no API documentation is needed, because the client needs no prior knowledge about how to interact with any particular application or server beyond a generic understanding of hypermedia.

Each request in a REST architecture represents a branch in the client's state. Each response that a client receives from the server includes the next available requests the client can make. This mechanism encourages a virtuous cycle of interaction for end users. All the client has to do is render the hypermedia and provide the underlying mechanism for enabling hypermedia controls.

Why does this matter? Well, imagine if for every website that existed the web browser (the client) had to implement each websites' particular method of interaction. Or if you had to download a different browser for every website you wanted to visit! Rather inconvenient.

Carson Gross took the ideas of REST combined with transclusion to implement his library htmx ⁷. htmx generalizes hypermedia controls to all HTML elements. htmx puts all its chips on HATEOAS. However, there is nothing novel about htmx. It's the culmination of ideas that makes it powerful. Instead of transferring data using an intermediate representation like JSON or XML, the data and available user actions are embedded as hypermedia, which the browser renders natively.

Like REST, htmx uses hypermedia as the driving force for interaction. Each request returns an HTML fragment (or full page) that can be swapped in place, or added to the browser's DOM. This isn't any different than what other Javascript frameworks already enable. htmx isn't breaking ground in the end users' overall experience. Instead, it's a demonstration of HTML as a fully realized hypermedia and web browsers capable of transclusion without the need for Javascript. It's about re-enthroning HTML as a first-class citizen of the web for which it was designed.

In a way, htmx goes back to the basics of what makes the web work so well: delivering hypermedia, HTML, to a hypermedia client, the browser. It's a rediscovery of the original architecture of the web. htmx just leverages what is already there. This discovery as it were, wasn't immediately obvious to Gross. In an interview ⁸ with The Changelog podcast he describes his experience:

Well, I think one thing that I’ve realized is I built Intercooler, and then htmx is—I didn't appreciate hypermedia really when I built the things. There's this idea that like existence precedes essence. That's a philosophical idea where something has to exist, and then you can look at that and say "Okay, what's the essence of this thing?" Like, strip away all the particularities of this thing. Like, what's the essence of it? And that was very much my experience with htmx, where I had to build it and then understand "Okay, wait a minute, this is really generalizing hypermedia controls." When I created Intercooler and htmx, I didn't think "Oh, I know what I'm doing. I'm generalizing hypermedia controls."

But when I looked at what I had done and I went back and read some of the historical writings on hypermedia, then I kind of developed that idea, like "Okay, this is why this is working as well as it is." Because it always felt like I wasn't writing that much code, and I was getting a lot of bang for my buck out of it. And I think the reason for that is that it was taking this sort of like higher level idea of hypermedia and then generalizing it, is the way that I would say it.

So I just think this is one of those things that you have to do it before you can really talk about it, and so I did it with IntercoolerJS and then htmx, and that put me in a position to write these more sort of like deep essays on hypermedia. And the thing that I came to understand about hypermedia as I got into it is it's one of a few different ways to build distributed systems.

If your listeners are familiar with Roy Fielding, he wrote a really famous dissertation sort of early on for the web...And it’s pretty academic and hard to read, so I don’t—there’s a reasonable summary of it in the hypermedia systems book that you can read if you want...But what he talks about in there and what I've come to appreciate about hypermedia is it's just one of these sort of fundamental ways you can build a distributed system. And it was a really effective way, as evidenced by the fact that the World Wide Web sort of took over everything, in the late '90s and early 2000s. So developing an appreciation for what's unique about it, what are its strengths, what are its weaknesses...That's just something I've been able to do over the, last, say, 5 or 10 years, as I've been working on on these things.

Carson Gross's experience building htmx underscores that an idea isn't born fully-formed. It's a loose amalgamation. It's only through persistence and failure that an idea can be turned into a concrete implementation. Systems are inherently complex and so it takes time and a deep understanding of it before you can effectively intervene. Even then, finding leverage in a software system is often counterintuitive.

Before the release of htmx, there have been many successful frontend libraries that completely ignored REST. While htmx works with the grain of REST and the intended architecture of the web, it goes against the grain of what's popular. This is mostly due to the fact that most software is not written for the sake of writing software. We stop at the first or second local maximum that gets the job done. Any changes or improvements after that seem ancillary to the ultimate purpose. For most things that seems ok. And maybe that's why changing software systems is so hard, because you have to descend from your current mental model of how things work before you can ascend past it.

Litestream §

The final example of systems thinking is the project Litestream ⁹. Litestream is a tool for continuously replicating changes to a SQLite ¹⁰ database. There are other ways to backup SQLite databases, but Litestream's replication strategy is what we are interested in.

SQLite has two journaling modes ¹¹: write-ahead Logging (WAL) and rollback journal. WAL mode works by writing changes to a separate WAL file. Once a WAL file reaches a certain size, SQLite will automatically checkpoint the file and commit it to the database.

Well, the creator of Litestream, Ben Johnson discovered that you could control SQLite's checkpointing system by opening a long running read transaction. As long as the transaction is open the WAL file cannot be checkpointed by SQLite or other processes. Litestream can then safely replicate the WAL file to persistent storage. Here is the actual Go source code for this process ¹².

func (db *DB) execCheckpoint(mode string) (err error) {
	// Ignore if there is no underlying database.
	if db.db == nil {
		return nil
	}

	...

	// Ensure the read lock has been removed before issuing a checkpoint.
	// We defer the re-acquire to ensure it occurs even on an early return.
	if err := db.releaseReadLock(); err != nil {
		return fmt.Errorf("release read lock: %w", err)
	}
	defer func() { _ = db.acquireReadLock() }()

	// A non-forced checkpoint is issued as "PASSIVE". This will only checkpoint
	// if there are not pending transactions. A forced checkpoint ("RESTART")
	// will wait for pending transactions to end & block new transactions before
	// forcing the checkpoint and restarting the WAL.
	//
	// See: https://www.sqlite.org/pragma.html#pragma_wal_checkpoint
	rawsql := `PRAGMA wal_checkpoint(` + mode + `);`

	var row [3]int
	if err := db.db.QueryRow(rawsql).Scan(&row[0], &row[1], &row[2]); err != nil {
		return err
	}

	...

	// Reacquire the read lock immediately after the checkpoint.
	if err := db.acquireReadLock(); err != nil {
		return fmt.Errorf("release read lock: %w", err)
	}

	return nil
}

Litestream uses a SQL transaction as a coarse-grained mutex. As a side-effect, the uncommitted transaction, prevents all processes from checkpointing the WAL. When the transaction is rolled back, Litestream immediately issues a manual checkpoint, and creates a new read transaction. In this brief window, there is a risk that another process could issue a checkpoint. Litestream works around this by generating another snapshot if it detects that it missed a WAL page ¹³.

SQLite documents the checkpointing ¹⁴ behavior of the WAL and how applications can control it. Litestream leverages this process so that it gains sole (or nearly total) ownership of checkpointing the WAL. What's cool here is that Ben Johnson, the creator of Litestream, discovered this one day as he was just playing around with the database. Here is how he described his thinking in an interview ¹⁵ with The Changelog:

When I thought about what was the thing that was keeping me from running SQLite in production, replication and disaster recovery was really the main thing. I actually spent a long time trying to figure this problem out. The code itself isn’t even huge. You can open up the code, it’s not gonna blow your hair back, or anything; it’s not that fancy. But trying to figure out how to actually make it happen was a long journey...I originally actually ported SQLite to Go. This is a kind of a thing I do, where like I don’t understand code until I really work with it and move it around...And the idea wasn’t necessarily to release that code, but just really try to understand what was going on underneath.

In the process of trying to gain an understanding of what makes SQLite work, Ben Johnson discovered a way to intervene in the system without having to rewrite it. Litestream works with the grain of how SQLite behaves. In fact, as far as SQLite is concerned it is just operating the way it was designed. Litestream is an example of seamless intervention in a system. The system continues to work as it was designed, yet new properties are created without disturbing the underlying feedback loops.

Conclusion §

As software engineers, we are surrounded by systems that we must work around or with to get our jobs done. Changing software systems often introduces unintended complexity. Everyone must draw the line at how much complexity they are willing to accept. Systems thinking is a way around that. From our examples above, we can draw the following conclusions.

First, finding leverage in a software system takes time. In all of the examples above, it took a long time, sometimes years, before an effective solution could be found. You must have a holistic understanding of how the system interacts with itself and its environment. It takes a curious mind and the determination to see your solution through to completion.

Second, points of intervention may not always be obvious or are counterintuitive. Because systems tend to be self-perpetuating, our perception of how a system works will continually be reinforced; unless we can find a strong counterexample of how we think it should work. Unix developers weren't aware of the inefficiency of creating temporary files until they actually tried using pipes. htmx is unusual in its architecture as compared to the likes of React or Vue. Questioning conventional wisdom may not make us popular but it may reveal new properties of a system.

And finally, solutions may require that we reframe the context in which a system operates. As an analogy, imagine how a lion behaves in a zoo versus on the safari. It's still a large, carnivorous feline, but its environment influences its behavior. The underlying functions of shell commands are the same but when connected via pipes they have emergent properties because they are working in a different context. Litestream changes the context in which SQLite operates even though SQLite is functioning as designed. The system continues to work, but a new context reveals a different view of the system's behavior.

Systems thinking as applied to software engineering opens doors for new ways of finding leverage to solve problems. Dramatic shifts like these are possible as we strengthen our understanding of the domain and validate our ideas through experimentation. In each example, the people responsible for making these changes were deeply curious. They sharpened their hypotheses by just trying things out. We can take inspiration from their work to find leverage in our own. By maximizing leverage in software systems we open new doors of possibility.

Footnotes §