The Benchmark Where My Own Migration Lost

Our admin search ran an unanchored regular expression against a collection of roughly 41,000 records. Every keystroke, a full scan. Obvious thing to replace, and replacing it was right.

What I didn't expect was the benchmark coming back mixed, or that the mixed result would be the most useful thing I produced.

Regex fails in a specific shape

The problem with unanchored regex matching isn't that it's slow. It's that it's slow unpredictably.

Most queries were fine. A few hundred milliseconds, no complaints. The failure lived in the tail. On one measured query shape, regex gave a p50 of 674 milliseconds and a p95 of 138 seconds. Half the time it felt instant. Occasionally it took over two minutes, and on a single-threaded runtime that doesn't only punish whoever typed it.

An average hides this completely. Any dashboard reporting mean response time would have shown a mildly slow search and no emergency.

The replacement

A proper search index with normalized, prefix-anchored tokens, plus a backfill to populate the token field across existing records. The backfill moved 40,844 records in about three seconds, which was the one part of the project that behaved exactly as expected.

Then I benchmarked both paths side by side on the same data, checking latency and that both returned the same rows.

The results were mixed

On the query shape above, the index gave 4,932ms at p50 and 6,054ms at p95, against regex's 674ms and 138,578ms. Seven times slower at the median, twenty-three times faster at the tail. That's the trade I wanted: give up typical-case speed to kill the two-minute worst case.

On a selective job query the index won everywhere. 6ms p50 against 10ms, 28ms max against 318ms. Easy.

Then on a broad query returning about twenty thousand rows, regex won outright. 56ms at p50 and 69ms at p95, against the index's 1,132ms and 129 seconds. The new system had its own catastrophic tail on that shape, worse than the thing I was replacing.

I could have skipped that third benchmark. Or run it, seen the result, and framed it as an outlier pending tuning. Both are common. Both are a kind of lying.

What the losses bought

Publishing all three rows changed the decision from "migrate the search" to "migrate each read path to whichever implementation wins on that path." Better decision, and not one I'd have reached on my own optimism.

Everything went out behind a feature flag, per read path, so each could be flipped independently and flipped back. Paths where the index clearly won went first. The path where regex won stayed on regex until the shape of that query changed.

There's a practical argument for honest benchmarking that has nothing to do with virtue. The losses tell you where the boundary is. A report containing only wins tells you your change is good. A report containing losses tells you which conditions it's good under, and that's what you need six months later when someone asks whether to use it somewhere new.

The bugs were about meaning

Two of the more interesting problems had nothing to do with speed.

Full email addresses stopped matching. Tokenized search splits text into pieces, which is what you want for names and wrong for identifiers. Searching a complete email address should return the one person who owns it, not everyone sharing a token with it. Emails and similar identifiers had to be indexed verbatim, with equality semantics, alongside the tokenized fields.

Multi-word queries returned everything. Autocomplete-style matching treats terms independently by default, so hendri test matched documents containing either term rather than the phrase. Users typing two words expect results to narrow, not explode. Restoring substring-like semantics for multi-word input was a correctness fix no latency graph would ever surface.

Replacing a search implementation isn't a performance project. It's a semantics project with a performance benefit, and the semantics are what users notice.

Two operational lessons

Treat index definitions as infrastructure. Search indexes drift between environments, and when a query hits a generation that doesn't match what the code expects you get a hard failure on a page that used to work. The endpoint should degrade rather than error, and provisioning should run through the same reviewed automated path as any other deployment step, not whatever someone ran by hand once.

And never let request handling create infrastructure. I found a path that could trigger index creation from inside an API request, which takes around 24 seconds. A user-facing request should never be able to do that. Harmless every day until it happens under load, at which point it's the whole incident.

The instinct after weeks on a migration is to present it as a win, because it mostly is one and that's what the audience wants. The row where my replacement lost by a factor of twenty is the reason anyone should trust the other rows, and the reason we shipped the right thing on each path instead of the same thing everywhere.