Democratized distributed training

Open-weights models, trained and improved by cohorts of hardware people already own. Where the work on this site is going, stated plainly.

The intended system lets an owner of capable hardware join a cohort, contribute verified work to an open-weights model, and receive auditable credit. Admission controls who enters a run. Verification checks sampled work. Settlement decides which contributions count. The resulting model remains open to the people who trained it. This is the distributed-training part of the route, distill, train loop, not a general market for renting computers.

Islands, not nodes

A cohort admits islands, not individual devices. An island may be one Mac, several Macs linked by Thunderbolt, one CUDA host, or a rack. Within that boundary, the operator controls trust and bandwidth; data and pipeline parallelism can use the local links. Traffic between islands is scarce, compressed, and verified. Canonical byte records keep those exchanges independent of architecture, allowing Mac and CUDA islands to join the same cohort.

The intra-island substrate has one bounded measurement. A pure-Go collective layer drives RDMA over Apple Thunderbolt without a C++ dependency, as checked by linkage inspection. In a two-host benchmark at 16 KiB, it matched the C++ median latency, measured a lower tail latency, and allocated no Go heap objects per collective. The result covers two Apple machines at that payload size. It is not yet wired into the mesh layer.

Trust is what makes it open

Open participation makes every contribution untrusted. Purpose-scoped credentials admit a peer; sortition selects verifiers; sampled recomputation checks part of the work; and a separate multi-party settlement decides credit. Verifier observations remain evidence rather than turning automatically into credit. Equal-peer merging also prevents a participant from buying influence by claiming a larger contribution weight.

Published systems address verification of a single inference on untrusted hardware. A training contribution spans many steps and touches optimizer state, so it poses a different problem. The first bounded test here is smaller: reject one defined invalid contribution while an ungated path accepts the same input.

The wire is the constraint

Preliminary observations on our fleet put consumer Wi-Fi near 20 megabits per second. A full weight sync appeared to take a fifth of a second on loopback and 81 seconds across the room. The raw record did not survive, so these figures await remeasurement, as recorded in the claim ledger. So the design treats inter-island bytes as the scarce resource: full-weight sync is the rare event, everything else that crosses the wire is compressed with declared error bounds, and every byte is metered on both ends of the link. The next training-floor test will check whether home-network bandwidth constrains the run before memory or compute.

Teaching the mesh to spend its bytes

A pre-registered study will compare residual sketches, sparse features, gradient sketches, and top-k logits at matched byte budgets. If one carrier preserves causal fidelity, islands could exchange compact state often and full weights rarely. Plain logits may win, or no candidate may clear the bar; both are valid outcomes. No acceleration claim exists today.

Where it stands

Three records support this direction: two machines exchanging and merging one adapter, a fragment gate tested on loopback, and the claim ledger. Trust-plane code and bounded compression codecs exist. Some distributed paths are tested only on loopback, and the field demonstration remains ahead. opentinker will become the participant-facing surface only after its cohort and adversarial-verification gates pass.

← groundwork · travis@tmc.dev