Democratized distributed training

Open-weights models, trained and improved by cohorts of hardware people already own. Where the work on this site is going, stated plainly.

caletta labs · vision · substrate measured, endgame ahead

The intended system lets an owner of capable hardware join a cohort, contribute verified work to an open-weights model, and receive auditable credit. Admission controls who enters a run. Verification checks sampled work. Settlement decides which contributions count. The resulting model remains open to the people who trained it. This is the distributed-training part of the route, distill, train loop, not a general market for renting computers.

Islands, not nodes

A cohort admits islands, not individual devices. An island may be one Mac, several Macs linked by Thunderbolt, one CUDA host, or a rack. Within that boundary, the operator controls trust and bandwidth; data and pipeline parallelism can use the local links. Traffic between islands is scarce, compressed, and verified. Canonical byte records keep those exchanges independent of architecture, allowing Mac and CUDA islands to join the same cohort.

The intra-island substrate has one bounded measurement. A pure-Go collective layer drives RDMA over Apple Thunderbolt without a C++ dependency, as checked by linkage inspection. In a two-host benchmark at 16 KiB, it matched the C++ median latency, measured a lower tail latency, and allocated no Go heap objects per collective. The result covers two Apple machines at that payload size. It is not yet wired into the mesh layer.

Trust is what makes it open

Open participation makes every contribution untrusted. Purpose-scoped credentials admit a peer; sortition selects verifiers; sampled recomputation checks part of the work; and a separate multi-party settlement decides credit. Verifier observations remain evidence rather than turning automatically into credit. Equal-peer merging also prevents a participant from buying influence by claiming a larger contribution weight.

Published systems address verification of a single inference on untrusted hardware. A training contribution spans many steps and touches optimizer state, so it poses a different problem. The first bounded test here is smaller: reject one defined invalid contribution while an ungated path accepts the same input.

The wire is the constraint

Measured on our own fleet, consumer Wi-Fi sustains roughly 20 megabits per second, an order of magnitude below what published datacenter-grade decentralized training assumes. A full weight sync that takes a fifth of a second on loopback takes 81 seconds across the room. So the design treats inter-island bytes as the scarce resource: full-weight sync is the rare event, everything else that crosses the wire is compressed with declared error bounds, and every byte is metered on both ends of the link. The GPU is not the bottleneck of democratized training. The wire is, and engineering for the wire people actually have is the difference between a demo and a substrate.

Teaching the mesh to spend its bytes

A pre-registered study will compare residual sketches, sparse features, gradient sketches, and top-k logits at matched byte budgets. If one carrier preserves causal fidelity, islands could exchange compact state often and full weights rarely. Plain logits may win, or no candidate may clear the bar; both are valid outcomes. No acceleration claim exists today.

Where it stands

Three records support this direction: two machines exchanging and merging one adapter, a fragment gate tested on loopback, and the claim ledger. Trust-plane code and bounded compression codecs exist. Some distributed paths are tested only on loopback, and the field demonstration remains ahead. opentinker will become the participant-facing surface only after its cohort and adversarial-verification gates pass.

← groundwork · travis@tmc.dev