The Code

Percentiles (50th, 99th, 99.9th, 99.99th) are more accurate and informative than averages in the vast majority of distributed systems. Using a mean assumes that the metric under evaluation follows a bell curve but, in practice, this describes very few metrics an engineer cares about. “Average latency” is a commonly reported metric, but I’ve never o... See more

Backpressure is the signaling of failure from a serving system to the requesting system and how the requesting system handles those failures to prevent overloading itself and the serving system. Designing for backpressure means bounding resource use during times of overload and times of system failure. This is one of the basic building blocks of cr... See more

For the last several years, work-sample were our most important hiring decision factor.

We relied on them almost completely, and in doing so, we multiplied the size of our team and retained every single person we hired

duplication is far cheaper than the wrong abstraction

All non-trivial abstractions, to some degree, are leaky.

Another observation I’ve made is that starting with a performant core can ultimately drastically simplify the architecture of a software project, relative to a given level of functionality.

If you find yourself in this situation, resist being driven by sunk costs. When dealing with the wrong abstraction, the fastest way forward is back . Do the following:

1. Re-introduce duplication by inlining the abstracted code back into every caller.


2. Within each caller, use the parameters being passed to determine the subset of the inlined code that th

... See more

What sets distributed systems engineering apart is the probability of failure and, worse, the probability of partial failure. If a well-formed mutex unlock fails with an error, we can assume the process is unstable and crash it. But the failure of a distributed mutex’s unlock must be built into the lock protocol.

It is all love with me and this observation and I make it as a card-carrying member of the tribe to which it is directed, but here goes: there may be no cohort of professionals less qualified to assess barely-tangible socio-psychological attributes like “passion” and “confidence” than the modern software nerd.