One Timeline, Many Machines

The request in docs/TODO.md was for one viewer that shows many machines, with clocks lined up well enough that a scope on one box can be read against a scope on another. Post 22 worked out how, and ended on five decisions. This post takes the first of them, pack the machine into the top byte of the pid, as decided by the plan's own recommendation, and builds the core: the link, the identity, the clocks, the fan-out, and the rail. The transports the plan added around that core, ssh with install-on-demand, a token on the API, the Machines panel, stay in the plan; the last section says exactly which parts are not here.

The shape: a hub and its remotes

Nothing is configured in advance and there is no distributed build. Any orbit-service is a hub the moment another machine is linked to it, and a remote is an ordinary service doing ordinary work: it captures what it always captured and hands the frames it would give a browser to the hub instead. The hub is the service whose URL you open. Browsers never talk to a remote.

The plan chose the hub over the browser for four reasons, and the decisive one still holds: aligning two hosts needs round-trip timing at microsecond resolution, and a browser coarsens performance.now() on purpose. Service to service, both ends read the capture clock, CLOCK_MONOTONIC on Unix and QueryPerformanceCounter on Windows, the clocks the events are stamped with. The hub exposes that clock to the server crate through one hook, LiveService::set_clock, and the same hook now stamps POST /api/events, which on Windows had been reading a different clock than the capture.

One link, two roles

A link is one WebSocket. The hub side runs hub_role, the remote side remote_role, and neither knows which WebSocket crate carries the bytes or who opened the socket: each socket is pumped into a pair of channels, a Link, by a twenty-line adapter per crate (axum for an accepted connection, tokio-tungstenite for a dialed one, and a pair of in-process channels for the tests). The roles are plain async loops over those channels. That is what lets a remote dial in and still be a remote: the hub accepts the socket on /api/join and runs the hub role over it, exactly as it does over a socket it dialed to a remote's /api/peer.

Pings are their own tiny messages, never appended to a batch, because a probe queued behind a 64 KB batch measures the batch. The remote answers from inside its select loop; if that loop is busy forwarding a burst the answer is late, the round trip is long, and the filter in the next section throws the sample away. A dropped link is redialed every two seconds by whichever side dialed, and the remote resubscribes to its own broadcast before saying hello so nothing between the two is missed.

Identity: a byte and a word

The viewer keys everything on a 32-bit pid, and two machines will each have a pid 1234. The plan's recommendation was to pack the machine into the top byte at hub ingest, and that is what is built: the hub's own ids keep a zero top byte and read exactly as before; a remote's pid and tid become machine << 24 | id. Linux's pid_max ceiling is 2²², so 24 bits hold any real id on any of the three platforms.

Three more collisions are handled the same way, at ingest, per link. Interned string ids are a per-service sequence, so the hub interns each remote string into its own table and keeps a map from the remote's id to its own; a name it has never seen falls back to ? rather than to a wrong label. A remote's process name gets @host appended, which is both the label on its process row and, as the next-but-one section shows, how the viewer learns the machine's name without a new frame type. And scheduler slices carry a core id in a byte: a remote's cores are shifted past the hub's and past every earlier link's, so machine 1's core 3 draws as lane 15 on a 12-core hub. The Scheduler track is still one track; 255 lanes is its ceiling and a slice past it is dropped and counted.

Clocks: trust the fast round trips

A probe is the NTP exchange: t1 as the hub sends, t2 and t3 as the remote receives and answers, t4 as the hub receives. Offset and round trip fall out of the four, and the error of any single sample is at most half its round trip, under the one assumption that both legs took the same time. The method is about making that assumption true: keep a window, find the fastest round trip in it, and keep only the samples within 1.5× of it plus 30 µs of scheduler slack. Those are the probes that met no queue on either leg.

The fit does what the plan said and no more. It is applied once, at ingest; events already in the ring are not moved when a later probe refines it. The raw probes are not yet written into the bundle, so a saved capture cannot be re-fitted offline; that is one of the deferred items. What the unit test proves is the estimator: a remote 12.345 ms ahead and 30 ppm fast, probed through jitter that queues one leg in four for up to 2 ms, comes back with the rate within 3 ppm and a mid-window conversion within 60 µs.

What the e2e measured is the other half, the one with a real socket in it. Two services on one Windows box, debug build, Windows loopback: the true offset is zero, so every microsecond the fit reports is error.

Every shift is inside the half-round-trip band the fit itself promises, and every one is under 90 µs. It is also plainly a debug build on a loopback that is slow by Unix standards; the plan expected 20–60 µs round trips on one box and a wired LAN at 100–300 µs, and those measurements are still to be taken with a release build and two machines. The viewer does not yet draw the band; GET /api/peers reports it, per link, as the best round trip and the probe count, and the manual says to read it before trusting a cross-machine delta.

The data path at ingest

Record fans out

The hub's process list is its own list plus every link's, each remote entry under its packed pid and named name @host, with a machine field for anything that wants to group. Pick one and press Record: the start handler reads the top byte, sends that machine start with the real pid, sends every other link start with pid 0, and starts its own capture with no target. Pick a local process and every link gets pid 0. Stop goes to every link. A capture with no target is what it has always been, the scheduler, the service, and every process on that machine that instruments itself, so a remote that was not chosen still contributes its manual scopes and its core lanes.

The hub remembers the packed pid as the capture's target, which is what makes the viewer pin that remote process's track the way it pins a local target. Hook lists are not sent across: a remote target captures with the default settings, and /api/symbols/load or a function search on a packed pid fails with a plain error, because the symbols live on the machine that owns the process and the proxying the plan describes is not built.

In the viewer

The rail already drew a machine tree per MachineId for the demo's fake remote. The enum gained Peer(u8), from_pid reads the top byte, and the header that read MACHINE REMOTE now looks for a process of that machine whose name ends in @host and prints the host; the hub's own rail prints the service's machine name from /api/status. That is the whole viewer change, and the wasm pack was rebuilt with it. There is no Machines panel yet: the links are made from the command line or POST /api/peers/connect, and their state is read from GET /api/peers.

How to use it

orbit-service --join <hub-host>:44766

orbit-service --connect render-01,build-mac:44766
curl -X POST http://127.0.0.1:44766/api/peers/connect -d '{"target":"ci-runner"}'

What is not there

Against post 22, these are the parts that stay in the plan, so a reader of that post knows what to expect. No ssh transport and no install-on-demand: a remote needs a running service and a reachable port in one direction. No token on the API: linking a machine is as open as the API was, which the manual says in so many words. No Machines panel in the viewer, no alignment band drawn on the rail, no amber lag glyph. Reports, trees, function search, disassembly and hooks on a remote process are not proxied; the timeline, the names and the manual scopes are what cross. The bundle gets the packed pids and the @host names but no machines entry and no clocks table, so a saved capture cannot be re-fitted. One Scheduler track with the remotes' cores appended, not a Scheduler per machine. And the numbers above are one box, debug build, loopback: the LAN and WAN rows of the plan's precision table are still expectations.

How it is checked

Nine unit tests in peers.rs: the pid packing and its guards, the probe arithmetic, the fit through queued probes, the offset-only fit, target parsing, the frame filter, the machine table's indices and core lanes, the ingest of a remote's frames onto the hub's clock, and two services linked by in-process channels, hello to fit to start to stop. One e2e, multi_machine_smoke.py, with two real services on two ports: the link comes up with a fit, twenty spans posted to the remote's /api/events land on the hub under machine 1 with their stamps moved by less than the band, OrbitTestRust started beside the remote appears in the hub's list as OrbitTestRust.exe @WINBOX, a Record on the hub for that pid starts the remote and streams 10,374 of its scopes back under packed thread ids, and Stop stops both. On Windows the remote and its target get a scope directory of their own, so the hub can only see the target through the link. The smoke runs on the Windows, macOS and Linux workflows.