Hiring and firing a DBA

2026/09/07

Hiring our DBA

For the past sixty days, Buttondown 1I take liberties here and throughout the essay: this was all Matias' doing. He also wrote a great piece on our migration to Planetscale, and also cites Planetscale's recommendation for this experiment. ran a scheduled LLM invocation once a day. Every morning it would look at the previous few days of database load in PlanetScale, and — if, and only if, it was sufficiently confident — it would open a pull request containing exactly one performance improvement to the seam between our app and our database.

I think of this as a longitudinal study, and a fairly cheap one: LLMs are extremely well suited to performance optimizations, precisely because such optimizations can be evaluated and then trivially discarded. 2We used GPT 5.3 Codex High for this, but I suspect our outcomes would have been broadly similar with whatever model we reached for.

Two months down, and here's where we're at:

1. We decreased our overall database reads by 25% for no cost or effort beyond that of reviewing the PRs.
2. We are stopping the experiment.

Let me frame the rest of the post with the most important takeaway for a prospective reader: this was an undoubtedly positive-value experiment, and I recommend everyone try it if they're able. Even setting the epistemology aside, the performance benefits we banked were unambiguously worth the small amount of time I'd spend reviewing PRs every morning. The only reason I'm not dwelling on the upside is that I think the flaws are more instructive than the wins.

So, with that out of the way —

Firing our DBA

25% improvement! That's great! Why stop the party?

The single biggest reason is that the LLM ran out of easy targets. In the first thirty days, I approved 75% of incoming PRs. In the latter thirty, that number dropped to 30%.

The theoretical underpinning of the whole enterprise is that there's a sweet spot: the risk of a given PR has to round down to zero, while the value of the performance benefit has to stay high enough to outweigh the attention and energy required to review it. 3This will be covered in a post to come, but a note I'd have here for the "get rid of code review" maximalists is that at three of the PRs I rejected would have caused an incident if merged. Without looking too hard at the Insights dashboard, it's clear the LLM ran out of obvious wins and it started getting desperate. Most of the PRs it submitted toward the end were bad code that made choices I didn't like — adding branching logic to defer work under some very esoteric circumstance, say, which would have a real positive performance impact and an equally real, and larger, negative technical-debt one.

The second reason is that the LLM consistently overclaimed — both the scale of the improvement it was making and its confidence in being able to make any change whatsoever.

I led with the exciting headline: it cut 25% of our reads. Which is great! But that impressive feat is itself roughly 25% of what it claimed it was going to do. Yes, the math there is not a typo. Largely, I think, this is because we did not — for obvious reasons — give it access to the actual production dataset, only the metadata about it. So it was calibrating a lot of its estimates against synthetic datasets it would conjure up itself, and synthetic data is a flattering mirror.

The combination of the two meant this experiment had crossed from positive value into negative-value territory.

What about upskilling?

I'm entirely amenable to the counterargument that I should treat this as a harness-engineering problem instead: give the model better instructions, or tooling to approximate the prod dataset, or coaching on which sacrifices to refuse. And those responses are correct, in a certain sense: they probably would have resulted in a better signal-to-noise ratio. But they still amount to me spending unjustifiable time on this experiment. It was valuable to me because it was free real estate, so to speak — not because I have a deep and systemic database problem. (Once again: thank you to PlanetScale for making that no longer the case.)

Zoom out further and you arrive at the worry that haunts everything shaped like this: there really is only so much low-hanging fruit. Every time a frontier lab ships a new model your proverbial ladder gets a little taller, but at some point you have to move on from your current copse of trees. Which is not to say ladders aren't useful, only that they're not jetpacks.