A fail-closed datapath for decentralized training

caletta labs · built · cross-machine exchange measured · node-loss demo pending

Decentralized training only matters if the pieces that move between machines cannot be quietly corrupted. A weight fragment that arrives wrong, or from the wrong peer, or out of round, should be rejected, not folded into the model. We built the datapath fail-closed: the default on any integrity doubt is to stop, not to proceed.

What moves, and how it is checked

The datapath moves content-addressed weight fragments over a live peer-to-peer link built on the published mlx-go-iroh transport, itself a layer over go-iroh. Fragments are content-addressed and verified against their published digest; a signed envelope binds each contribution to its sender. Tampering, replay, and a mismatched digest are all rejected at the correct edge with a stable diagnostic.

What is proven today

A loopback test moves an encoded fragment between two endpoints on one machine and checks that the receiver accepts only a valid fragment. This establishes the protocol's fragment gate and its tested failure behavior on loopback.

A separate run used two Apple Silicon machines. Each fine-tuned a LoRA adapter; the machines exchanged and merged those adapters, then wrote byte-identical files in three consecutive rounds. Two Macs, one adapter records the numbers and transport failures. The run proves cross-machine adapter exchange and deterministic merge. It does not extend the loopback fragment-gate test into a cross-machine proof of every integrity failure.

What a field demo still has to show

Coordinated group training and resilience under loss remain unproven. The loopback test covers the fragment gate; the two-machine run covers adapter exchange and merge. Neither shows that a group trains as one system, continues after a node disappears mid-round, or recovers peer-held state without a privileged coordinator. Each needs its own acceptance test.

The transport layer is public and cloneable now. The training datapath that rides on it is built and in release preparation; this note describes it rather than linking a repository a reader cannot yet open.

← groundwork · travis@tmc.dev