DRAFT — not yet reviewed for publish. Verify the loopback claim, the test name, and every repo reference against the current source before deploying.

A fail-closed datapath for decentralized training

caletta labs · built · cross-machine exchange measured · node-loss demo pending

Decentralized training only matters if the pieces that move between machines cannot be quietly corrupted. A weight fragment that arrives wrong, or from the wrong peer, or out of round, should be rejected, not folded into the model. We built the datapath fail-closed: the default on any integrity doubt is to stop, not to proceed.

What moves, and how it is checked

The datapath moves content-addressed weight fragments over a live peer-to-peer link built on the published mlx-go-iroh transport, itself a layer over go-iroh. Fragments are content-addressed and verified against their published digest; a signed envelope binds each contribution to its sender. Tampering, replay, and a mismatched digest are all rejected at the correct edge with a stable diagnostic.

What is proven today

The transfer and integrity path is proven on loopback: a test moves an encoded fragment between two real endpoints on one machine and asserts the receiver accepts only what verifies. That is a real result about the protocol and its failure behavior.

A first cross-machine result now sits alongside it. Two Apple Silicon machines trained, exchanged, and robust-merged real LoRA adapters over the same transport family, and the merged files were byte-identical on both hosts in three consecutive rounds. The run is small and its transport limits are documented rather than hidden; the numbers and the failure ladder are in Two Macs, one adapter. Cross-machine exchange of real weights is measured. What loopback proved about integrity still holds at that boundary.

What a field demo still has to show

The honest gap is resilience under loss. Moving fragments correctly, on loopback and now between two machines, proves the transport and the integrity gate. It does not prove that the remaining peers keep making progress when a node disappears mid-round, recover from peer-held state, and finish without a privileged coordinator. That node-loss continuity is the work ahead, and it is the difference between compute that coordinates and compute that cannot be switched off.

The transport layer is public and cloneable now. The training datapath that rides on it is built and in release preparation; this note describes it rather than linking a repository a reader cannot yet open.

← groundwork · travis@tmc.dev