A Ring Per Core

A profiler's manual instrumentation is the feature people use most and trust least. Every ORBIT_SCOPE in a hot loop is a bet that the measurement costs less than the thing being measured, and the moment that bet stops paying, the macros get commented out and the profiler stops being used. So the write path is the whole design. Everything else — interning, nesting, and any ordering beyond a single thread's — is the consumer's problem, and the consumer is allowed to be slow.

The title is where this started, and it is left standing because the reasoning that led there is worth keeping. A ring per core was the natural first answer. It turned out to be the wrong one, for a reason that only shows up in userspace, and the design that replaced it — a ring per thread — is both simpler to defend and portable off Linux, which the producer has to be.

The shape of it

An instrumented process writes events into its own shared-memory segment. The service maps that segment, drains it, interns the names, and hands the events to the timeline. The two sides never talk, never synchronise, and never block each other.

The first instinct was one ring per core. A ring per thread is genuinely single-producer and needs no atomic at all, but it scales with something you do not control: a process with four thousand threads gets four thousand rings. Cores are a number you know at startup and can cap. That instinct was half right — capping is exactly what you want — and half wrong, because a per-core ring is not safe to write with a plain store, for a reason that only appears in userspace. The next section is the whole of that reason. It is why the rings ended up following threads after all, just not one apiece.

Why per-core does not hold

The appeal of a per-core ring is that it looks single-producer for free: a thread reads its core id, and since only one thread runs on a core at a time, only one thread is ever writing to that core's ring. No atomic needed — a plain store into the next slot.

That reasoning is exactly right in the kernel and exactly wrong in userspace, and the difference is preemption.

What rseq is, and why it stopped being the answer

Restartable sequences are a Linux mechanism, merged in 4.18, for exactly this problem: letting userspace run a short critical section that is guaranteed to have executed entirely on one CPU, without a lock and without a syscall.

You register a small structure with the kernel through the rseq syscall — glibc 2.35 and later does this for every thread automatically and exposes it as __rseq_abi. Inside it, the kernel keeps a field that always holds the id of the CPU the thread is currently on, updated on every migration. That alone is useful: reading your CPU id becomes a thread-local load instead of a sched_getcpu() call.

The clever part is the second half. You also register a critical section descriptor: a start address, an end address, and an abort address. While the thread's instruction pointer is inside that range, the kernel promises that if the thread is preempted or migrated, it will not simply resume where it left off — it will jump to the abort handler instead. So you write the sequence as "read the CPU id, compute the slot, store", and if anything interrupts you anywhere in the middle, you land in the abort path and start over. What you get is an atomic-free, lock-free critical section with true per-CPU exclusivity, which is precisely the guarantee the per-core design assumed it already had.

It is what makes genuinely per-CPU userspace data structures possible, and it is why allocators like tcmalloc use it for their per-CPU caches. It needs the critical section written so it can be restarted from the top with no side effects, and on most architectures wants a hand-written assembly sequence with the abort handler placed by the linker behind a specific signature.

So rseq would make a per-core ring sound. Two things keep it from being the answer here. The first is that it is Linux-only and wants hand-written assembly, and this is the one part of Orbit that does not run in Orbit: the producer is compiled into the profiled application, which may well be a game shipping on Windows, so a portable path has to exist regardless. The second is the one I missed at first. rseq buys per-CPU exclusivity — and the design does not want per-CPU anything. The whole hazard was born the instant a ring was chosen by the core a thread happened to be on. Choose it by the thread instead, and migration is a non-event: there is no window to close, and nothing to close it with.

So: a ring per thread

Drop the core id entirely. Each thread hashes its own id to a ring and writes only there, for its whole life. However often the scheduler bounces it between cores, it never changes rings, so the preemption window simply does not exist — and the producer reads no CPU id, registers no rseq, and depends on nothing Linux-specific. That portability is the point, not a bonus: this code compiles into somebody else's application, wherever it ships.

The map from thread to ring is one rule for every thread, with no fast path and no fallback. It is a hash rather than tid % ring_count because thread ids are not arbitrary numbers: they are handed out in runs, and Windows hands them out in multiples of four, which a plain modulo over sixteen rings would fold onto four of them and leave twelve idle. Multiplying by the 64-bit golden-ratio constant and taking the high bits pushes that low-bit structure up and out. And the hash is stateless, which matters more than the arithmetic: a claimed ring would have to be released when a thread exits, and a thread-local destructor in someone else's application may run late or never. An earlier version handed out exclusive rings from a pool and sent the overflow to a shared one — two policies wearing one name, since with bounded rings and unbounded threads the sharing is guaranteed by pigeonhole anyway. Exclusivity was never a policy, only a lucky case, and it leaked rings the moment it lost.

Which is the catch, and the reason the atomic never went away. Because the rings are capped and the threads are not, several threads share a ring, so every ring is multi-producer whether you like it or not. The slot is claimed with a relaxed fetch_add — still lock-free, still no CAS loop, about twenty cycles uncontended. rseq would not have removed it: rseq gives per-CPU exclusivity, and the contention here is two threads on one ring, which it does not touch. The plain-increment path the per-core design wanted is simply unreachable once rings are shared. The sharding still earns its keep — as a contention argument now rather than a correctness one — because sixteen rings keep collisions rare.

How many rings is a knee, not a preference. More of them means less contention when several threads claim at once; fewer means each gets a larger share of the memory budget, which decides how long one busy thread runs before it laps its ring. At thirty-two threads claiming flat out, a whole scope costs 22.9 ns on one ring, 14.0 on eight, 12.9 on sixteen and 12.5 on 128 — most of the benefit gone by eight, all of it by sixteen. Against that, at an 8 MiB budget 128 rings leave each with 1024 slots, so a single hot thread laps after a thousand events, versus 8192 slots at sixteen. Sixteen buys ninety-five per cent of the contention win and eight times the headroom against the likelier failure, which is one thread doing most of the work while the rest sit idle. A hundred and twenty-eight is the cap the format allows a reader to trust, not the number a process opens.

There is a test for the sharing, and it is the kind worth writing: eight threads pushing to one ring, then a check that every producer's events survived with a slot of their own. Replace the fetch_add with a plain increment and it fails immediately.

Fifty-six bytes

The record is fixed width, which is what makes a claim a single index bump rather than a variable-length reservation. It is fifty-six bytes: twenty-four of fixed fields and thirty-two of name, and the name is written in full, inline — no interned id, no hash, no registration protocol on the write path.

The record is a fixed-width write either way, so those bytes are stored whether they carry a name or not; what a name adds on the write path is a length and a copy into the local record, which measures within the noise of doing nothing. What it buys is a segment that describes itself: no protocol for registering a name before an event can use it, no race between the two across threads, no table to ship to the service, and nothing left dangling when a process dies mid-run. Static and dynamic names take one path. Interning happens on the consumer, which is allowed to be slow. A name too long for one record spills into continuation records keyed by the same (tid, scope_id), and the reader rebuilds it.

The thirty-two inline bytes are simply what is left of a cache line after the fixed fields, and they got there in two steps. When the merge went (below), the slot's second atomic went with it and its eight bytes became text; when the producer-side name id went, its four bytes followed. The record carries more of the name now precisely because it carries less machinery.

The write path

One atomic add, the payload in plain stores, one release store to publish. No lock, no compare-and-swap loop, no allocation, no syscall, no branch on a queue being full.

pub fn push(&self, ring: usize, event: ScopeEvent) {
    let claim = self.ring_header(ring).write_cursor.fetch_add(1, Relaxed);
    let slot = self.slot(ring, claim);                            // claim % slots_per_ring
    unsafe { std::ptr::write_volatile(slot.event.get(), event) }; // payload
    slot.seq.store(claim + 1, Release);                           // publish
}

The cursor only ever increases; it never wraps back. A slot index is claim % slots_per_ring, and the claim number doubles as the sequence stamp, so a consumer can always tell a slot's current occupant from the one that occupied it a lap ago.

Where the time actually goes

An earlier version of this post said the ring cost about ten nanoseconds an event and that "a good half of that is the clock read". The second half of that sentence was wrong, and wrong in the way benchmarks usually are: the benchmark pushed a pre-built event with a constant timestamp, so it never read the clock at all. Here is the same write path taken apart properly, on the shared path that every scope actually takes.

Two things fall out of that table, and both matter more than the single number I first reported.

The clock read is 9.3 ns, more than half. Optimising anything else while leaving that alone is the wrong order of work. clock_gettime through the vDSO already reads the timestamp counter internally; what costs the nine nanoseconds is the vDSO's own generality. Reading rdtscp directly and converting with the same calibration the kernel already publishes is the larger prize — with the caveat, below, that it has to keep producing CLOCK_MONOTONIC values or the timeline comes apart.

The atomic claim is 4.3 ns, a quarter of a scope, against a plain increment of 0.2. That gap is exactly what rseq's per-CPU exclusivity would have bought a per-core ring — and it is unreachable here, because the rings are shared and the claim has to be atomic whatever CPU the thread is on. The number is a good reason to respect the atomic. It is not a reason to reach for rseq.

The one fence

Past the one atomic claim, the write path has exactly one fence, the commit — no CAS loop, nothing else. And the fence needs a qualification that costs nothing on the machine most people are reading this on, and one instruction elsewhere.

The payload stores can be plain. The store that publishes the slot cannot: without release ordering, the compiler or the processor may make the sequence stamp visible before the bytes it vouches for, and a consumer reads a slot the producer has not finished writing. On x86-64 the hardware does not reorder stores with other stores, so a release store compiles to an ordinary mov and the claim is literally true. On aarch64 it is one stlr. This codebase targets both, so the honest version of the claim is: no fences where the hardware gives them to you for free, and one instruction where it does not.

Drain, no merge

The consumer copies each ring's committed prefix out in one pass, and that is the end of it. There is no merge. The split between copying and consuming is still the reason a slow consumer never stalls a producer — producers keep writing while the consumer works on a snapshot — but the second half of the old pipeline, a k-way merge into one globally timestamp-ordered stream, is gone.

It is gone because nothing wanted what it produced, and it cost a great deal. Because a thread always writes the same ring, its events sit at increasing claim numbers in that ring, so reading a ring in claim order yields every thread's events in the order that thread wrote them. That is the only order anything downstream needs: a scope's start precedes its stop, a name's head precedes its continuations, and the viewer's lanes are keyed by (pid, tid, kind, depth), so a lane holds one thread's events and they arrive already sorted. What the merge added on top was ordering between threads, which no consumer ever asked for.

What it cost was a horizon: a single global gate, held at the minimum frontier across every ring, so that nothing could be emitted until every ring had proven it could produce nothing older. That gate turned one stuck producer into everyone's problem. A thread stopped at a breakpoint, or starved of its CPU quota, held down not just its own events but every other thread's, because the horizon is a minimum and any one ring can pin it. Deleting the merge turns that global stall into a local one: a claim nobody has published blocks its own ring and nothing else.

A stalled ring waits on liveness, not the clock

A local stall is still a stall, and it has to end. When the drain reaches a claim that has been handed out but not yet committed, it cannot read past it — claims are read in order — so it stops on that ring and waits. The question is how long, and the wrong answer is a timeout.

The first version used one: 250 ms, on the theory that no live producer could take longer to finish a write. That was simply untrue, and every counter-example is more likely while profiling, not less — a thread stopped at a debugger breakpoint, a cgroup that has spent its CPU quota, a major page fault under memory pressure, a guest descheduled by its hypervisor. Any of them outlasts 250 ms easily, and throwing away the events of a process that is alive and about to publish them is the profiler corrupting its own measurement.

So the consumer decides by liveness, not by the clock. If the producing process is still there, the claim may yet be published however long it has waited, and waiting is the correct thing to do. If the process is gone, nothing it claimed can ever be published, so the claim is written off at once, counted as dropped, and stepped over. That is a question orbit-service can answer — it already reads /proc for everything else — and it is why only the producer has to run everywhere, not the consumer.

A thirty-second backstop remains, but not as a timeout. It exists only to bound the case this design has not thought of — a recycled pid answering the liveness question wrongly, say. Half a minute is far too long to be a timeout and exactly right for a last resort: it will never fire in normal operation, and it keeps the worst case finite. When it does fire it drops one event and says so, which is a far better failure than a timeline that silently stops.

Migration is a non-question

Everything above invites the question that broke the earlier design, so it is worth answering plainly: if a thread migrates between entering a scope and leaving it, is the record still valid? Yes, and for the least clever reason available. Starts and stops are matched by (tid, scope_id) — never by ring, never by adjacency, never by cross-core timestamp order — and a name reassembles by the same key. Nesting depth is worked out by the reader from the order of a thread's own starts and stops, so a scope that is never stopped costs nothing but itself instead of skewing everything after it. None of it depends on which core anything ran on.

Under a per-core ring this took real care, because a start and its stop could land in different rings and the consumer had to knit them back together across the merge. Under a per-thread ring the question barely arises: a start and its stop go to the same ring however many times the scheduler moved the thread, because the ring is chosen by the thread. Migration was a hazard the per-core design had to survive. The per-thread design does not have it.

One thing does follow from sharing, and it is worth stating rather than rounding off. Within a single ring, claims from different threads are not in timestamp order: two producers can read the clock in one order and claim slots in the other. That would matter if anything needed a ring's events sorted by time, but nothing does — each thread's events are read in its own write order and handed to its own lane, and the interleaving of different threads inside one ring is invisible once they are split apart by tid. The old merge, and the second slot atomic that fed it a lower bound, existed to paper over exactly this seam. Nobody was looking at the seam.

Why not the TSC

A timestamp counter read is faster than clock_gettime, and at more than nine nanoseconds an event the clock is over half the cost. Treating it as a clock of its own is still the wrong choice, and not for the usual reasons about invariant TSC or cross-socket skew.

Manual scopes have to appear on the same timeline as scheduling slices, thread states and sampled callstacks, all of which arrive from perf stamped with CLOCK_MONOTONIC. Two clocks means two timelines that cannot be interleaved, and a scope that cannot be lined up against the context switch that interrupted it has lost most of its value. The vDSO makes clock_gettime a function call rather than a syscall, which is what makes it affordable at all. This is why the rdtscp idea above is a conversion, not a substitution: read the counter directly, but turn it into CLOCK_MONOTONIC with the calibration the kernel already publishes, so the timeline stays one timeline and only the vDSO's generality is shed.

This is also the shape of a bug found earlier in this port, in a different module: a timestamp taken from a clock that counted from its own first call rather than absolute CLOCK_MONOTONIC. Everything downstream looked several decades out. Clocks are worth being pedantic about.

The reader assumes the writer is hostile

The segment is written by another process that may be buggy, mid-crash, or simply a different build. So the reader maps its rings read-only — a consumer must not be able to corrupt the process it is observing, and only the one control page that carries the capture flag is mapped writable — and validates every field of the header against the mapped length before reading a single event: magic, version, record size, ring count in range, slot count a power of two, and the described layout actually fitting in the file.

The magic is stored last, with release ordering, so a reader that opens the segment while the writer is still initialising it sees "not initialised" and retries rather than mapping garbage at plausible-looking dimensions. Overflow is handled the same way: if a producer laps the consumer, the drain detects it from the sequence stamps, skips to the oldest slot still resident, and reports how many events were lost. Losing events under sustained overload is correct behaviour for a design whose first rule is that producers never wait. Lying about it would not be.

What this feeds

When the earlier version of this post was written none of this was reachable from an instrumented program. Most of the list it ended on is now built, and built on top of the write path rather than tangled into it.

• The instrumentation API. ORBIT_SCOPE and its siblings are real, in orbit-api behind include/orbit.h: a handful of functions exported with a C ABI for C, C++ and Python, and offered natively for Rust. Each caches the thread id once — gettid is a 60 ns syscall that would otherwise dwarf the scope it stamps — and does nothing else the ring does not.
• Interning. Names arrive inline and are interned on the consumer, exactly where the write path pushed the work. Nothing on the hot path registers anything.
• Nesting. Depth is not stamped at all; the reader reconstructs it from the order of a thread's starts and stops, so an unmatched start skews nothing after it.
• Service integration. The events reach orbit-service, which interns the names and turns them into scopes on the timeline.
• Still open: cross-process throughput. The crate's own tests exercise the shared-memory path with a writer and a reader in one process. A second process mapping the same segment under sustained load is the measurement not yet taken.

The numbers in this post are in docs/blog/metrics/scope-rings.txt; the forty-nine tests are in rust/crates/orbit-scope-ring, and the cost figures come from the two benchmarks in its tests/cost.rs.