Why Orbit’s kernel-uprobe stop was slow

Self-contained report: the kernel lock, why fewer process-wide fds are not the production fix, the shared-tracefs change, and the 10/20/50 attach/detach numbers from this run.

1. The issue

Orbit’s historical kernel-uprobe path opens one uprobe and one uretprobe PMU fd per CPU per function (pid=-1, cpu=0..N-1):

fds = 2 × F × NCPU

Each of those fds is its own create_local_trace_uprobe — not another consumer of one probe. Closing a capture therefore destroys F × NCPU × 2 independent probes.

perf_uprobe_destroy holds the global event_mutex around the entire close:

  1. uprobe_apply(false)
  2. uprobe_unregister_nosync — VMA walk under dup_mmap_sem + register_rwsem
  3. uprobe_unregister_sync() — RCU tasks-trace and uretprobes SRCU
  4. tracepoint_synchronize_unregister

Parallel close() cannot overlap those grace periods; they queue on event_mutex. Wall time stays

O(F × NCPU × 2 × (VMA walk + 2–3 RCU GPs))

Stop time growing with core count is expected from the per-CPU fd model. ParallelFor does not help: it issues more closes at once, but they still serialize on the same mutex.

Old path — every sample fd is a local probe

Two functions × three CPUs shown. Each chip is a PMU fd. Every close takes event_mutex.

Historical per-CPU uprobe PMU fds all serialize on event_mutex For each function and CPU, an uprobe fd and a uretprobe fd each perform create_local_trace_uprobe. All closes go through one global event_mutex. F functions × N CPUs × 2 PMU fds → event_mutex function 0 function 1 … function F-1 u cpu0 ur cpu0 u cpu1 ur cpu1 u cpu2 ur cpu2 u cpu0 ur cpu0 u cpu1 ur cpu1 u cpu2 ur cpu2 … … … … … … global event_mutex one close at a time — ParallelFor still queues here Each fd = create_local_trace_uprobe. Close holds the mutex for a VMA walk + 2–3 RCU grace periods.

2. Why pid=target, cpu=-1 is not the production fix

Opening one uprobe + one uretprobe for the process (2×F fds) was measured. It is not how production samples:

Production keeps pid=-1, cpu=N so every thread on every core is sampled. The bench still times the process-wide experiment; it is a side table, not the fix.

3. The solution

Register each probe once via tracefs (uprobe_events, group orbit) and open the per-CPU fds as TRACEPOINT consumers of that named probe.

New path — many sample fds share two named probes per function

TRACEPOINT fds still deliver per-CPU samples. Only the named probes pay the unregister cost.

Shared tracefs registration with per-CPU TRACEPOINT consumers Each function has one named uprobe and one named uretprobe. Many per-CPU TRACEPOINT fds share those two registrations. Expensive unregister is 2 times F. 2 named probes per function; per-CPU fds are TRACEPOINT consumers function 0 function 1 … function F-1 orbit:u_0 orbit:ur_0 orbit:u_1 orbit:ur_1 orbit:u_… orbit:ur_… TRACEPOINT fds (cpu0..N-1) c0 c1 c2 … c0 c1 c2 … expensive unregister = 2 × F named probes — not 2 × F × NCPU Shutdown still closes every sample fd, then UndefineTracefsUprobe for each name Sample layout (PERF_SAMPLE_REGS_USER / STACK_USER) matches the old PMU helpers. Coverage stays pid=-1,cpu=N. The shared registration is what makes stop cheap.

close() under event_mutex (historical local PMU)

This is why core count multiplies stop time. Shared-tracefs pays this sequence once per named probe, not once per sample fd.

perf_uprobe_destroy holds event_mutex for the whole close close of a local PMU fd locks event_mutex, then uprobe_apply false, uprobe_unregister_nosync with a VMA walk, uprobe_unregister_sync with RCU grace periods, and tracepoint_synchronize_unregister, then unlocks. close(fd) → lock event_mutex → four steps → unlock close(fd) event_mutex 1. apply(false) disable the probe 2. unregister_nosync VMA walk 3. unregister_sync RCU + SRCU GPs 4. sync unreg tracepoint A second close() waits on event_mutex until this whole sequence finishes. Grace periods do not overlap. Historical cost: repeat once per PMU fd (2×F×NCPU). Production cost: once per named probe (2×F). That is why stop time was proportional to cores, and why ParallelFor cannot erase it.

4. Measured attach/detach (10 / 20 / 50)

Times are this run only. Missing columns are n/a — numbers are never invented. Speedup = percpu_stop / shared_stop. percpu = historical local PMU; shared = production tracefs path.

kernel: not measured on this static copy
ncpus: n/a
pid model: production samples use pid=-1,cpu=N (every thread, every core). pid=target,cpu=-1 is a fewer-fd experiment only (single tid + inherit).
uprobe PMU: n/a
tracefs: n/a

nfunctions ncpus percpu fds named probes percpu start percpu stop shared start shared stop speedup
10 n/a 2×10×NCPU 20 n/a n/a n/a n/a n/a
20 n/a 2×20×NCPU 40 n/a n/a n/a n/a n/a
50 n/a 2×50×NCPU 100 n/a n/a n/a n/a n/a

percpu_fds = 2×F×NCPU; named_probes = 2×F.

pid=target,cpu=-1 (experiment, 2×F fds; not production)

nfunctionsstartstop
10n/an/a
20n/an/a
50n/an/a

n/a Bench not run on this static copy. Open the HTML written next to cwd after a real run to see timings.

command: not run

legend: percpu = historical local PMU (event_mutex × fds); shared = production (one tracefs unregister per function). Verdict uses the existing policy: FAIL only if shared start/stop is more than 250 ms worse than percpu PMU.

5. How to re-run

Off by default, even as root. The 10/20/50 bench is not part of a normal LinuxTracingIntegrationTests invocation.

./src/LinuxTracingIntegrationTests/run_uprobe_attach_detach_bench.sh
# or: sudo ORBIT_UPROBE_BENCH=1 ./bin/LinuxTracingIntegrationTests --gtest_filter='*UprobeAttachDetachBench*'

The script finds the binary (or takes a path), exports ORBIT_UPROBE_BENCH=1 (true/yes also work), re-execs under sudo, and runs only this test — timings plus the tracer e2e (start/stop/start + function-call check). It writes uprobe_attach_detach_bench.txt and uprobe_attach_detach_bench.html in the current directory.