Nobody Needs a Real-Time Dashboard

Analytics is the feature everyone asks for and nobody budgets for. It gets scoped as "just show the numbers", then quietly becomes the slowest and most complained-about part of the product.

Ours got to where the seven slowest analytics endpoints averaged somewhere between five and ninety seconds.

The first visitor pays for everyone else

The complaint that started it was specific. The overview panel on one page is unusable. I pulled the numbers: that endpoint took about 56 seconds cold and a quarter of a second warm.

Look at that gap and the instinct is "good, the cache works." It doesn't. It means the cache works for the second person.

Think about who the first person is. Analytics gets opened at the start of the day by whoever is preparing for a meeting. Traffic goes quiet overnight, cache entries expire, so the first request of the morning is guaranteed to be the cold one. It comes from the person with the least patience and the most influence over whether your product looks good. Everyone after them has a lovely time.

Caching without thinking about that optimizes for the visitor who matters least. That reframing changed what I built more than any query tuning.

Three ways the cache made it worse

Everyone recomputes at once. When an expensive entry expires, every request arriving in the gap misses, and they all start computing the same thing. The database takes ten copies of a query that was already too slow as one. Single-flight fixes it: first request computes, everyone else waits on that result. We moved roughly seventy analytics reads onto a shared lock-and-cache helper so it stopped being something each endpoint got right or wrong alone.

The lock expired mid-compute. Single-flight only works if the lock outlives the work. Ours didn't. A query ran longer than its own lock, the lock released, the waiting requests concluded nobody was computing, and started computing. The stampede came back in disguise. Locks around slow work need a TTL derived from how slow the work actually is, plus room for the day it's slower.

And I over-corrected on TTLs. Migrating everything onto one caching path, I widened lifetimes fairly uniformly. Great for cumulative totals. Wrong for panels people check to confirm something they just did. A user submits, checks, sees the old number, and stops trusting the dashboard, which is worse than the slow version was. I walked sixteen panels back to ten minutes. Freshness isn't a global setting, it's a per-number product decision.

Stop computing on read

All of that is damage control around a bad premise: that the way to answer "how many in the last thirty days" is to count them when someone asks.

Rows arrive continuously. Nobody edits last Tuesday. So compute each day once, store the per-day totals, and answer by summing a handful of small rows. A collection with millions of records collapses to a few thousand rollup rows, one per day per metric per scope.

On a seeded dataset at five million rows, a thirty-day total came back in about 124 milliseconds, because it summed thirty rows instead of five million. Fifty concurrent reads finished inside a second. The read stopped needing a cache to be survivable. Caching became an optimization instead of life support.

The work moves to a scheduled job, which has its own problems worth naming. Several replicas all boot and all try to schedule the same job, so the schedule needs deduplicating. And recomputation is heavy enough that it shouldn't share a process with requests users are waiting on, so a dedicated worker owns it.

A fast wrong number is worse than a slow right one

This is the part I'd argue hardest for.

The moment you precompute you have two sources of truth: the rollup, and reality. If they drift you've built something worse than the slow version, because the slow version was correct. Nobody says "the dashboard is wrong" once and forgets it. They stop using it, they check numbers by hand, and the feature you spent a quarter on becomes something people work around.

So before trusting it I ran the rollup against raw counts on the busiest day I could find, 260,278 records across four metrics, and checked every metric-and-scope combination came out identical. Fifty-five of fifty-five matched. Building the full history took about a minute and produced roughly eighteen thousand rows.

That check cost a fraction of what the rollup cost to build, and it's the only reason I was willing to put it in front of anyone.

The bit I didn't expect

Somewhere in the middle I realized we were solving a slightly wrong problem.

The stated requirement was that the numbers be current. But watch how people use an analytics page and almost nobody needs a number accurate to this second. They need to know how old it is, so they don't quote something stale in a meeting. And they need to be able to force a fresh one when it matters, and see that it worked.

Neither requires real-time anything. Both are cheap. We put a last-updated label and a refresh control on every statistics panel, all hundred and nine of them, so there was no panel where the answer to "is this current?" was a shrug.

That killed most of the demand for real-time freshness, because the anxiety was never about latency. It was about not knowing whether to trust the number. A visible timestamp answers that for free. A refresh button hands control to the person who cares, at the moment they care, instead of making the system pay for freshness continuously on everyone's behalf.

Which is the thing I'd tell anyone starting this work. "Make it real-time" sounds like a technical requirement. Usually it's a person telling you they don't know whether to believe the screen.