The user's note was one line: I feel we can be much faster. That feeling is usually right and almost never about the thing you would guess, so this pass began by refusing to guess. Every hot path got a benchmark that prints its cost before a single line changed, and the viewer got a way to say, from inside a headless browser, which phase of its frame the time actually went to. What follows is what the numbers said, in the order they said it.
The service: a million samples per request
A sampling report is the answer to "what was running in this selection". The service stored every sample in a flat vector and, for each request, walked all of it, testing each sample against the window. That is fine once. But the viewer asked for a report and a call tree on every frame of a selection drag, and each request walked a million samples to find the thousand inside a one-millisecond window.
The fix is the obvious one and the interesting part is why it was not there: samples from several perf rings interleave within a pass, so the vector is not naturally sorted, and a sort on every push would have been worse than the walk. Now the store marks itself dirty when a push lands out of order and sorts lazily at the first request after — the stable sort finds the sorted prefix and merges the short tail in near-linear time — and a window becomes two binary searches.
Three more things fell out once the walk was gone. The trees were built as a serde_json::Value per node before being printed, and a bottom-up tree over a thousand samples is tens of thousands of nodes; they are written straight into a string now. The maps were keyed by frame id through SipHash, which guards against attacker-chosen keys that frame ids are not; a multiplicative hasher replaced it. And a million samples of a real program are a few thousand distinct stacks seen over and over, so the selection is folded into (thread, stack) → count first and each stack walked once.
| 1M samples, 200 call paths | before | after | |
|---|---|---|---|
| report, 1 ms window | 4.61 ms | 1.55 ms | what a drag frame asked for |
| tree, bottom-up, 1 ms window | 102.9 ms | 5.39 ms | 19× |
| report, whole capture | 174.0 ms | 111.5 ms | |
| tree, top-down, whole capture | 2670 ms | 359 ms | the stall when a capture stops |
Then the request itself. The viewer now holds report requests to one per 200 ms while the mouse button is down and sends the final selection the moment it is released. The service did item one per frame before; the viewer parsed the JSON per frame. Neither needed to.
Resolving an address it had already resolved
In the capture loop, every program counter of every sample was symbolized: two linear scans over the module list, a binary search for the symbol, two string allocations, then a hash of the name to find the id the viewer draws. Sampled addresses repeat — a hot loop is the same few thousand addresses forever — and every one of those repeats paid the full price to arrive at an id it already had. A per-capture map from address to id makes a repeat one lookup. The symbolizer is fixed for the life of a capture, so the map is too.
| 1.2M frames, 3,000 distinct addresses | before | after | |
|---|---|---|---|
| per frame | 146 ns | 3 ns | 49× |
| broadcast, 10,000-event batch | 226 µs | 44 µs | one copy instead of three; none without a viewer |
For honesty: a live capture on this machine cannot sample (no CAP_PERFMON), so the live pass I could run — the API test app, scope rings and thread states only — was already dominated by its own 5 ms sleep, with the work at a few tens of microseconds. The service's wins are on the paths the benchmarks exercise. The benchmarks are checked in as ignored tests and run with one command.
The viewer: threads times lanes, three times a frame
The viewer lays out its track rail every frame. On a large capture that was where the frame went, and the shape of the cost was the classic one: for each thread, scan every lane of the index to find that thread's own. It did this in the skeleton, again when assigning Y positions, and again per block height. With a process filter set — the normal live-capture case — the discovery loop ran an any() over all lanes for each lane: quadratic.
A sampled capture of one busy process is 100 threads with a 40-deep flame graph each, so 4,200 lanes. The benchmark for that shape said 10.3 ms per frame for the layout alone, before anything was drawn. The index now carries a lane generation, bumped only when a lane appears or disappears — not on every event — and the strip keeps a catalogue keyed on it: per-thread leaves in draw order, the pids that own lanes, the scheduler cores. Each former scan is a lookup. Then the row maps got the same multiplicative hasher as the service, and the row sort stopped looking both sides up in a map inside its comparator.
| 100 threads, 4,200 lanes | before | after | |
|---|---|---|---|
| layout per frame | 10.27 ms | 0.40 ms | 26× |
| layout per frame, with a process filter | quadratic | 0.09 ms | |
event_count() | 7.2 µs | 0.0 µs | a counter, not a sum over lanes |
Asking the browser where the time went
Native benchmarks say what a function costs. They do not say whether the function is the frame. For that the viewer's self-profile pane, added earlier in this series, now publishes its per-phase totals to the page as window.__orbit_self, so a harness driving headless Chrome can read the breakdown that is otherwise only painted on a canvas. This is the tool that found the two costs below, neither of which was on the list.
The first was in the WebSocket drain. After every batch of events the viewer refreshed its content bounds by walking every event of every lane — a cost proportional to how much had already arrived, paid per batch. And the frame decoder drained its leftover buffer after each decoded frame, which is a memmove of everything still queued: a burst of a thousand small batches in one chunk was a thousand copies of the chunk. Running bounds on the index and decoding by offset took the drain on the demo flood from 5.1 ms to 0.14 ms per frame.
The second was older than the port. On a perfectly still view of a 320-thread trace, the timeline re-listed every visible primitive every frame. The dirty key that should have skipped it watches a layout generation, and the generation advanced by one every frame. The reason took a diagnostic field in the readout to see: sync() seeded the row map from the skeleton, which lists only the rail lanes, so each frame it evicted the packed flame-graph lanes that apply_layout put straight back. The map never compared equal to itself. That seeding was for a lerp animation the rows no longer have; it had been marking every frame dirty for as long as the code existed.
| headless Chrome, viewer CPU frame | before | after | |
|---|---|---|---|
| live demo, whole frame | 6.44 ms | 1.62 ms | 4× |
| of which WebSocket ingest | 5.10 ms | 0.14 ms | 36× |
| 320-thread trace, still, whole frame | 3.0 ms* | 0.94 ms | *with the every-frame rebuild; layout already fixed |
| primitive rebuilds in 120 frames | 120 | 2 | |
| layout generation over 30 frames | +30 | +0 |
One number I am deliberately not selling: the headless frame interval on that trace went from 128 ms to 100 ms. The CPU frame is under a millisecond of that hundred; the rest is SwiftShader rendering the timeline in software, which is the environment and not the viewer. On a real GPU the CPU frame is the budget that matters, and it went from tens of milliseconds to under one. The run that closes this is on the user's machine with the pane open.
What is left
During a live capture the timeline still re-lists everything visible each frame, because data is arriving and it must; only the lanes that changed need it, and that is an incremental listing nobody has written yet. The whole-capture top-down tree is still a 359 ms stall when a large capture stops; the cost that remains is hashing a million stacks, and folding them at ingest would remove it. Both are named in the metrics file, next to the numbers that say how much they are worth.