Make every model call earn its keep.
llm-router scores each request on cost and quality, then sends it to a frontier API or a model on hardware you control. It also captures consented traffic as training and evaluation data you keep. One self-hosted binary; no hosted control plane.
Built and tested; publication-pending. Routing and the capture path run today; the distillation classifier and fine-tune training on top are still in build. Source private today, available for review on request.
What it does- Learned per-prompt routing. Each call is scored on cost and quality and sent to a frontier API or a local model, chosen per request rather than by a static rule.
- No hosted control plane. A single Go binary against a single SQLite file. No Postgres, no Redis, no call-home, no network at decision time. The router sits on your side of the line, and it stays there.
- Self-hostable gateway and embeddable library. A drop-in, OpenAI-compatible endpoint you run on your own hardware, or the same routing as a small Go package with nothing to provision.
- Capture off the request path. Routing decisions are traced and a cheaper model shadows the request out of band, writing training datasets from your own traffic without adding latency to the live call.
- Reproducible evaluation. An offline eval harness, so a cost-quality claim is a measured number you can rerun, not an assertion.
Keep frontier calls where their measured quality justifies the price. Move the rest to hardware inside the operator's custody domain. The offline evaluator measures the cost-quality trade; the capture path preserves the consented examples, outcomes, and evaluations needed to train a model later.
Routing and capture run today. Local inference uses mlx-go; distributed-training experiments combine mlx-go-ccl, mlx-go-iroh, and mlx-mesh. The distillation classifier and end-to-end fine-tune loop remain in build. The first owed proof is one scorable task whose unplug result can be rerun with the published evaluation harness.
source private repo, available for review on request. travis@tmc.dev
docs in progress
contact travis@tmc.dev