DRAFT — not yet reviewed for publish. Vision content: verify every measured number against the run records, and every maturity label against current source, before deploying.

Democratized distributed training

Open-weights models, trained and improved by cohorts of hardware people already own. Where the work on this site is going, stated plainly.

caletta labs · vision · substrate measured, endgame ahead

Here is the endgame we are building toward: anyone with capable hardware can join a cohort, contribute verified work to an open-weights model, and hold auditable credit for the contribution. What open source did for code, applied to training. Not a metaphor for it, the actual mechanics: admission, verification, settlement, and a model whose improvement belongs to the people who trained it. This is the training arm of an owned stack, a mesh that learns from you, not about you. It is deliberately not a standalone compute network; that is a race others already lead, and it is not the point. The point is that the loop you own, route, distill, train, gets its training rung from hardware that answers to no one but its owners.

Islands, not nodes

The unit of membership is not a machine, it is an island: a single Mac; a cluster of Macs linked by Thunderbolt; a CUDA box or a rack of them. Inside an island, bandwidth is cheap and trust is settled, so work runs at full speed with data and pipeline parallelism. Between islands, bandwidth is scarce and trust is zero, so what crosses the boundary is rare, compressed, and verified. The wire contracts between islands are canonical byte records, neutral to architecture by construction, so an all-CUDA island and an all-Mac island can serve the same cohort without either pretending to be the other.

The intra-island substrate is further along than a vision note usually gets to claim. A pure-Go collective layer drives RDMA directly over Apple Thunderbolt with no C++ dependency, verified by linkage inspection, and in a two-host benchmark it matched the C++ latency floor, edging ahead at the tail, with zero heap allocations per collective. That result is proven at two machines and 16 KiB payloads on Apple hardware only, and it is not yet wired into the mesh layer above it. Both halves of that sentence matter: the floor is real, and so is the scope.

Trust is what makes it open

Permissionless participation is a security problem before it is a systems problem. A stranger's cycles are worthless if their gradients can quietly poison the model, and a stranger's honesty is unknowable if checking it costs more than the work. The trust plane runs the whole width of a contribution: purpose-scoped credentials admit a peer; sortition selects who verifies whom; verification is sampled recomputation, not zero-knowledge machinery; verifier observations are evidence and never silently become credit; settlement is a separate, multi-party decision; and the credit that results is auditable after the fact. Robust merging is equal-peer by proof, so influence cannot be bought with inflated contribution weight.

The honest frontier: verifying that a single inference ran correctly on untrusted hardware has published answers in the field. Verifying a training contribution, work that spans many steps and touches optimizer state, does not. That open leg is the one we are building for, and the first bounded proof is deliberately modest: a defined invalid contribution is rejected while the ungated path accepts the same input.

The wire is the constraint

Measured on our own fleet, consumer Wi-Fi sustains roughly 20 megabits per second, an order of magnitude below what published datacenter-grade decentralized training assumes. A full weight sync that takes a fifth of a second on loopback takes 81 seconds across the room. So the design treats inter-island bytes as the scarce resource: full-weight sync is the rare event, everything else that crosses the wire is compressed with declared error bounds, and every byte is metered on both ends of the link. The GPU is not the bottleneck of democratized training. The wire is, and engineering for the wire people actually have is the difference between a demo and a substrate.

Teaching the mesh to spend its bytes

The open research bet rides on top of that constraint: if some compact carrier of a model's internal state, a residual sketch, sparse features, a gradient sketch, or plain top-k logits, preserves causal fidelity at matched bytes, then islands could sync compact state often and full weights rarely, and the scarce inter-island byte bill drops. This is active research with a pre-registered design: candidate carriers compete at matched byte budgets, and the outcomes where plain logits win or no carrier clears the bar are recorded as valid results, not failures. No acceleration claim exists today. The studies exist to earn one honestly or to refute it cleanly, and either verdict makes the mesh smarter about what its bytes buy.

Where it stands

The substrate under this vision is the measured work already on this site: two machines training one adapter to byte-identical merges, a fail-closed datapath, and a claim ledger that says what is measured, what is analyzed, and what is not yet run. The trust plane is landed in code. The compression codecs carry declared bounds. The distributed paths are loopback-proven where they are labeled loopback-proven, and the field demo is still ahead. What makes the ambition credible is not that the gap is small. It is that the gap is enumerated: every step between here and the endgame is named work with a named test, and the ledger will show each one landing or failing in public.

← groundwork · travis@tmc.dev