[{"data":1,"prerenderedAt":132},["ShallowReactive",2],{"journal-\u002Fjournal\u002Fwhen-every-endpoint-is-slow":3},{"id":4,"title":5,"body":6,"date":117,"description":118,"draft":119,"extension":120,"meta":121,"navigation":122,"path":123,"seo":124,"stem":125,"tags":126,"__hash__":131},"journal\u002Fjournal\u002Fwhen-every-endpoint-is-slow.md","When Every Endpoint Is Slow, None of Them Is the Problem",{"type":7,"value":8,"toc":108},"minimark",[9,13,16,21,24,27,30,38,41,45,48,51,54,57,61,64,67,70,73,77,80,83,86,89,92,95,98,102,105],[10,11,12],"p",{},"The report was useless in a specific way: the platform is slow. Not one screen. Everything. Unrelated endpoints taking ten to ninety seconds, for an hour, then recovering on their own.",[10,14,15],{},"The instinct is to open the slowest endpoint and start optimizing. That instinct is wrong here, and knowing why is most of the job.",[17,18,20],"h2",{"id":19},"unrelated-things-failing-together","Unrelated things failing together",[10,22,23],{},"One slow endpoint? Suspect the endpoint. Its query, its indexes, its payload.",[10,25,26],{},"Endpoints that have nothing to do with each other, all slow at once? They're not the cause. They're witnesses. Something underneath them is saturated: the connection pool, CPU, the cache, or the event loop.",[10,28,29],{},"People forget the last one. Your JavaScript runs on a single thread. One request doing heavy synchronous work doesn't only make itself slow, it makes everything else wait, including requests that would have finished in five milliseconds. Which is exactly what got reported. Everything degrades together, everything recovers together, and no single culprit shows up in the logs.",[10,31,32,33,37],{},"So the first move wasn't to look at an endpoint. It was to measure the shared resource. Node exposes loop timing through ",[34,35,36],"code",{},"perf_hooks",", so I added a monitor that samples the delay continuously and logs mean, p99 and max with a warning threshold.",[10,39,40],{},"The invisible thing now has a number.",[17,42,44],{"id":43},"loki-turns-it-felt-slow-into-a-quantity","Loki turns \"it felt slow\" into a quantity",[10,46,47],{},"Logs from every pod land in Loki, which lets you ask questions across the fleet and filter by tenant. That's the difference between \"a user says it's slow\" and knowing how often, for whom, how badly.",[10,49,50],{},"Forty-eight hours of one tenant's traffic gave me the shape: 7.63% of thirty-second windows contained a loop block longer than fifteen seconds. Worst single block, 143 seconds. For over two minutes that process served nothing at all.",[10,52,53],{},"Which reframes everything. Nobody needs to explain why an unrelated endpoint took 90 seconds. It was queued behind a blocked thread. The question becomes what's blocking, and how often.",[10,55,56],{},"The same approach sized the individual offenders. One recommendation endpoint had 48% of its requests over five seconds, p95 of 51.7 seconds, worst case past seven minutes. A paginated users endpoint sat at a median of 55 seconds. Numbers like that end arguments about whether something is worth fixing.",[17,58,60],{"id":59},"the-boring-causes","The boring causes",[10,62,63],{},"With targets identified, the rest is explain plans and unglamorous fixes.",[10,65,66],{},"Deep pagination. On an unfiltered list of about 41,500 rows, page one came back in 0.6 seconds and page one thousand took 20. Same query, same index. The cost is the database walking and throwing away everything it skipped. Fine at page 3. Ruinous at page 1000.",[10,68,69],{},"A sort the index couldn't serve, forcing an in-memory sort that spilled to disk. Ninety-seven seconds on its own. Fixed with a compound index whose key order matched the filter-then-sort shape of the query. Sorting is where \"works fine locally\" goes to die, because locally the collection fits in memory.",[10,71,72],{},"And a payload nobody looked at. One list endpoint returned documents with large embedded arrays. A single row measured 2,378,413 bytes. Roughly 2.3MB, per row, on a paginated list. The query was fine. The index was fine. We were shipping megabytes nobody rendered. Projecting the heavy fields out was one line, and no amount of query tuning would have found it, because the query was never the problem.",[17,74,76],{"id":75},"where-the-ai-pair-helped","Where the AI pair helped",[10,78,79],{},"I pair with Claude on work like this. I want to be precise about where it helped rather than vague and impressive.",[10,81,82],{},"It helped with the mechanical middle. I knew I wanted to count thirty-second windows containing a loop block above a threshold, grouped by tenant, over 48 hours. Turning that sentence into correct query syntax on the second try instead of the fifth kept me in the investigation instead of in documentation. Same for reading verbose explain output against the index definitions.",[10,84,85],{},"It helped with the tedious prerequisite. Reproducing a scale problem needs data at scale, and writing seed scripts for hundreds of thousands of related records is dull work sitting between you and the experiment. Handing that off meant I ran experiments I'd otherwise have talked myself out of.",[10,87,88],{},"It helped me write the explanation for people who don't read explain plans.",[10,90,91],{},"It did not decide what to investigate. Nothing in the tooling was going to tell me that unrelated endpoints degrading together means look at a shared resource. That came from having seen the pattern, and it's the step that determined whether any of this went anywhere.",[10,93,94],{},"It also didn't tell me when my own measurements were lying. My load-testing tool reported a p95 from five samples. A 95th percentile over five runs is the maximum of five runs wearing a better name. It looked completely fine on the dashboard. I caught it because I know what a percentile requires, and changed it to nearest-rank over a hundred samples.",[10,96,97],{},"That's the boundary. Huge leverage on the mechanical, syntactic and tedious parts. The two ends, picking the question and deciding whether to believe the answer, didn't move.",[17,99,101],{"id":100},"the-habit","The habit",[10,103,104],{},"Measure the shared resource before optimizing anything individual. If the platform is slow everywhere, the endpoint you're staring at is probably a victim, and tuning it buys a small win that convinces you you're making progress while the real cause keeps happening.",[10,106,107],{},"And attach numbers early. \"It feels slow\" invites opinions. \"7.63% of thirty-second windows had a block over fifteen seconds, worst case 143\" ends the discussion and starts the work.",{"title":109,"searchDepth":110,"depth":110,"links":111},"",2,[112,113,114,115,116],{"id":19,"depth":110,"text":20},{"id":43,"depth":110,"text":44},{"id":59,"depth":110,"text":60},{"id":75,"depth":110,"text":76},{"id":100,"depth":110,"text":101},"2026-07-10","Investigating a platform-wide slowdown with Grafana and Loki, what the event loop revealed, and an honest account of where an AI pair actually earned its keep.",false,"md",{},true,"\u002Fjournal\u002Fwhen-every-endpoint-is-slow",{"title":5,"description":118},"journal\u002Fwhen-every-endpoint-is-slow",[127,128,129,130],"engineering","performance","debugging","ai","nw96xMWonibao1PmE8paMWZexosW8xd5Vnf27DiTrVo",1787915616681]