The Hook That Would Not Arm

A profiler that only samples tells you where time tends to go. Dynamic instrumentation tells you what actually happened: this function, this call, these nanoseconds. Orbit has always had both, and the Rust port had neither half of the second one wired to the UI. Wiring it took an afternoon. Proving it took longer — the syscall that arms a probe kept refusing, and the first version of this post blamed the wrong capability for it. This is the corrected account: what the refusal actually was, why the obvious fix does not help, and the one that does.

The half that already existed

The pleasant surprise was that the viewer already spoke the entire protocol. Its capture strip has a hook picker: it asks the service to load symbols, polls until they are ready, searches function names as you type, and puts the ids you tick into the capture request alongside a dynamic_instrumentation_method. All of it was written when the viewer was talking to the C++ service.

What the Rust service answered was three stubs. load_symbols returned "not implemented in the Rust service yet"; the status endpoint reported unavailable; the search returned an empty list. So the picker rendered, greyed out, forever waiting for symbols that were never going to arrive. The work was entirely on the service side, which is a good position to be in: the contract was already fixed, and there was nothing to design.

The one part that is not symbolization

Indexing functions is nearly the same job the sampler already does — read /proc/<pid>/maps, load each mapped ELF's symbol table — except keyed by name instead of by address. One thing is genuinely different, and it is the thing worth being careful about.

A uprobe is not placed at a virtual address. It is placed at a byte offset into a file: the kernel finds the inode, and every process that maps that file gets the breakpoint. So a function's ELF symbol address has to be walked back through whichever PT_LOAD segment contains it. For the test target's executable segment that arithmetic is a no-op, because the linker put p_offset and p_vaddr at the same value:

LOAD 0x001000 0x0000000000001000 ... 0x00025d 0x00025d R E
LOAD 0x002da0 0x0000000000003da0 ... 0x000270 0x000288 RW

The two interesting segments of the test binary. The first is the easy case. The second is why the conversion exists.

The second segment shows both reasons not to assume. Its file offset and its virtual address differ by 0x1000. And its size in memory (0x288) exceeds its size in the file (0x270) — that tail is .bss, which exists when the program runs but has no bytes in the file at all. An address landing there has no file offset, and asking for one has to return nothing rather than a plausible-looking number.

The reason to spend a tested function on twelve lines of arithmetic is the failure mode. Get this wrong and nothing complains: the kernel accepts the offset, writes a breakpoint into the middle of some unrelated instruction, and the target either produces nonsense or crashes far away from the mistake. There is no error to read. So the conversion is its own function with its own tests, including one that pins the .bss tail to None.

Per task, not per CPU

Orbit's C++ opens uprobes on every CPU in the cpuset. That has a known consequence, documented in this port a day earlier while chasing a different question: when a thread migrates mid-probe, one logical hit can surface in two per-CPU ring buffers, and UprobesUnwindingVisitor carries a (sp, ip, cpu) filter to throw the second one away. The underlying mechanism appears to be the execute-out-of-line preemption window, though that part remains a strong inference rather than a confirmed kernel bug.

The Rust collector opens probes per task instead, with inherit set so threads created later are covered too. A thread has exactly one buffer wherever it runs, so the classic per-CPU double — one hit surfacing in two ring buffers — cannot be constructed here. The stack-pointer filter is ported anyway, off by default and switchable on, because a later capture showed migration still perturbs the stream in a subtler way; untangling that is a follow-up post's story, not this one's. The cost of the per-task layout is file descriptors — two per hook per thread — which is why the number of hooks is capped at sixteen rather than left open.

Trading one problem for another is the honest description, though, because per-task probes create a different one. Entry and exit now live on separate rings, and there is no ordering between two rings drained microseconds apart. A naive loop will hand you a function's exit before its entry and then compute a duration by subtracting a timestamp that has not arrived yet. So hits are buffered and paired in timestamp order behind a 100 ms horizon — the same delayed-ordering idea the collector's main loop uses, in miniature. Two of the unit tests exist purely to pin that: one feeds the exit first, one checks that nothing is paired before the window has passed.

The wall

Then it did not work.

orbit-service: indexed 2303 functions across 3 modules for pid 588922
orbit-service: armed 0 of 2 hooks (0 probes)
orbit-service: hook not armed -- orbit_inner_work: Permission denied (os error 13)
orbit-service: hook not armed -- orbit_outer_work: Permission denied (os error 13)

Everything up to the syscall is right. The syscall says no.

The reflex here is to reach for perf_event_paranoid, which is the knob that governs almost everything else in this port. It was already at -1, the most permissive setting there is, and it made no difference — because for this PMU it is never consulted. In the kernel, perf_uprobe_event_init (kernel/events/core.c) calls capable(CAP_SYS_ADMIN) and returns -EACCES before perf_event_paranoid enters the picture at all. The uprobe PMU is gated on that one capability, full stop.

The first version of this post drew the wrong conclusion from that dead end. It reasoned by analogy with sampling and scheduling, which the kernel opens for CAP_PERFMON, and declared that uprobes wanted CAP_PERFMON too — and left the hook un-armed on a box that could not grant it. That was wrong, and measuring it on this kernel (7.0) settled the matter: a process holding CAP_PERFMON and nothing else gets the same EACCES from the minimal uprobe attr. CAP_PERFMON, which every other perf path in this service honours, does not open a uprobe. The check the kernel actually makes is for CAP_SYS_ADMIN, and that is the whole of it.

Grant that, and the wall is gone. Run the service under sudo, or give the binary the capability directly with sudo setcap cap_sys_admin,cap_perfmon,cap_dac_read_search+ep <orbit-service> — cap_sys_admin for the uprobes, cap_perfmon for sampling and the scheduler tracepoints, cap_dac_read_search for reading target binaries the invoking user cannot. The same two hooks that came back 0 of 2 now arm 2 of 2, the probes fire, and the calls come back as spans. The status line stops apologising and reads instrumenting 2 of 2 functions.

What the refusal narrowed it to

The refusal was worth reading precisely, because it localised the problem before the capability confirmed it. The uprobe PMU has no fixed type number — the kernel assigns it at boot and publishes it in sysfs, here as 9. A wrong type number comes back ENODEV. A malformed perf_event_attr comes back EINVAL. What came back was EACCES, the answer from inside the uprobe PMU's own initializer: the attr routed correctly and was turned away on privilege, not on shape.

That rejected two whole classes of mistake and pointed the remaining finger squarely at a capability — which is why, once CAP_SYS_ADMIN arrived, nothing else in the arming path had to change. A single test pins both outcomes: it arms a probe on this process's own binary, and if the run is privileged it asserts the probes armed and a return actually came back, otherwise it asserts the refusal was EACCES rather than the ENODEV/EINVAL that would mean a bad attr. A companion test goes the whole way when privileged — a_uprobe_fires_on_a_function_of_this_process hooks a real function in the test binary, calls it in a loop, and checks the paired calls come back with sane durations; unprivileged it prints UPROBE TEST SKIPPED: needs CAP_SYS_ADMIN and passes.

Making the failure visible

The worse outcome than a hook that does not arm is a hook that does not arm quietly. From the UI, an unarmed probe and a function that simply never ran look identical: an empty track. Someone would reasonably conclude the profiler was broken, or that their function was dead code.

So the arming outcome now travels back to the browser. The service carries a one-line instrumentation status, /api/status reports it, and the viewer prints it under the hook picker — muted when functions were armed, amber when they were not:

no hooks armed: uprobes need CAP_SYS_ADMIN. Run the service with sudo,
or: sudo setcap cap_sys_admin,cap_perfmon,cap_dac_read_search+ep <orbit-service>

The refusal the browser now shows. It names CAP_SYS_ADMIN, the capability that arms the probe.

Getting this line right mattered as much as getting the arming right, because a wrong hint costs a reader an afternoon. Two omissions are deliberate. It does not mention perf_event_paranoid, even though every other permissions message in this service does — for every other feature the paranoia knob is the answer, and here it is not consulted at all. And it no longer claims CAP_PERFMON is enough, which the message itself said for a while and which sent this post's first draft down the dead end above. A message that points at the capability the kernel actually checks is the difference between the feature reading as broken and reading as one command away.

What is still missing

User-space trampolines. The viewer offers two methods, and Orbit's default is the trampoline-based one, which is far cheaper than a uprobe because it does not trap into the kernel per call. The relocation and placement machinery is ported; installing it into a live process is not. A request for that method is honoured through uprobes and says so in the log rather than silently substituting.
Arguments and return values. The pairing carries slots for both, because the ported FunctionCallManager has them. Nothing fills them yet.
The rate the rings can take. Probes fire now; how many per second before the kernel starts dropping hits is a separate question, chased in a later post with a stress test built for it. The short answer is that the per-CPU rings overflow above roughly a million and a half scopes a second, and the pairing reports what is lost rather than hiding it.

A note on getting it wrong first

The tempting move when a syscall refuses is to write down the first plausible reason and move on. This post did exactly that once: it reasoned from the wrong analogy, named CAP_PERFMON, and left the hook un-armed. What made that cheap to correct was the discipline the port already had — the numbers on record. docs/blog/metrics/phase-8-dynamic-instrumentation.txt has the sysfs values, the JSON the endpoints returned, the program headers of the target so the offset arithmetic is checkable by hand, and the exact EACCES the arming syscall gave. None of that changed when the diagnosis did; only the conclusion drawn from it did.

Because the failing syscall was written down with its errno rather than summarised as "needs a capability", the fix was a one-line message change and a privileged re-measure, not an excavation. The account that is precise enough to be proven wrong is the one you can actually correct. The hook arms.