Guarded Training and Evaluation on Rented GPUs
Training on rented GPUs, with failures kept in view.
TuneHarness grew from repeated training, provisioning, and model-evaluation experiments on rented GPUs. Its operating tools check hardware, reconcile instances, track costs, and gate releases on task-specific evaluations. Completed training work uses single-node multi-GPU execution; distributed fleet training and supervision by trained models remain design goals. Saved-output comparisons, native checkpoint probes, and newer runtime checks have separate evidence and limitations, rather than establishing a universal reliability or cost advantage.
Interactive CLI Walkthrough
A stdlib-only policy layer with explicit verification checks
Explore real terminal sessions recorded from actual marketplace runs. Click each workflow tab to inspect how the harness provisions, preflights, judges, and heals training jobs.
Defensive Engineering
Four invariants that turn strangers' rigs into reliable compute
Cloud orchestrators assume friendly datacenter hosts. Marketplace hosts are consumer rigs run by strangers. tuneharness assumes every host is lying until verified by renter-side code, while closed-loop automation turns every incident into code.
Post-Boot CUDA Preflight
An instance that passes nvidia-smi can still fail on real compute due to driver/toolkit mismatches. The harness runs an immediate torch.cuda.init() and bf16 matmul. Bad hosts are terminated and replaced in ten seconds, preventing the 105-minute silent idle burns typical of naive rentals.
Provider-Neutral Seam & Distributed Scaling
Exactly one module talks to vendor CLIs (vast.ai and RunPod). The policy layer (guardrails, quarantine, reputation scoring, cost ledger, watchdog, and deploy gates) never mentions the provider. Marketplace adapters separate provider access from policy. Multi-node placement and distributed fault tolerance require their own execution evidence.
Money as a First-Class Failure Mode
The append-only ledger records create, stop, and destroy events with timestamps, rates, and reasons. Automated watchdogs sample GPU utilization on a cron and stop idle instances. Stop and destroy are asymmetric: automation may stop, but only the operator destroys.
Deploy Gate with Teeth
No model reaches production without beating its prompted generalist baseline on held-out data. Evaluations are judged on the worst of two seeds, tested on 10 exact failure canaries, and evaluated on format-disjoint out-of-distribution slices. Six candidate models have been refused to date.
Applied model-operations research
TuneHarness continues experimental work originally conducted under the VastOps name: operational specialists, voice turn detection, document field extraction, tool-calling method comparisons, prompt-schema ablations, noisy ASR benchmarks, and small-transformer training. Single-node multi-GPU DDP runs are documented; they do not establish multi-node or frontier-model training. Read the experiment history and its limits.
Specialist-model experiments and rental records support research into evaluation gates, compute reliability, and operating cost. The ledger is a dated operator sample, not a marketplace failure probability. Distributed training is a research extension requiring separate measured runs. Read Cisco Caceres's research agenda.
Illustrative Scenario Model
Interactive GPU Marketplace Risk & Cost Calculator
Explore a cost scenario using editable workload assumptions and fixed example hourly rates. The outputs are calculated estimates, not measured savings, current price quotes, or a forecast. The 62% default is the observed share of failed instances in one dated, 121-instance operator ledger; it is not a marketplace-wide failure rate. The modeled fast-fail comparison uses assumed failure-handling times, so it does not establish savings caused by TuneHarness.
Deterministic Deploy Gate
Where regressions die as candidates
A deploy gate that only ever passes models is theater. This gate earns its keep by saying no. Inspect real candidate promotion trials from our production history.
Empirical Grounding
The models, measured on held-out tasks
All evaluations are held-out and judged on the worst of two seeds. Baseline comparison is qwen3.5:35b-a3b (35B MoE generalist) running on our 24GB serving box.
| Model | Task | Size | Baseline Reference | Fine-Tune Result | Honest Slice / Caveat | Train Cost |
|---|---|---|---|---|---|---|
ops-doctor v4 | GPU-fleet action policy | 8B | Incumbent v2: 96.7% | 99.0% action acc, 1.2% missed | Same-simulator eval | ~$0.13 |
log-doctor v1 | Raw train.log triage (8 states) | 4B | Prompted 35B: 93% acc | 100% acc, 0% missed, 0.56s | Read 100% as eval exhausted | ~$0.35 |
ambie-eou v2 | Voice turn detection | 0.6B | Prompted 35B: 85% @ 524ms | 98.3% acc, 0.0% early-commit, 140ms | External OOD: 90.0% acc | ~$0.30 |
sign-fielder v3 | E-sign field placement | 4B | Prompted 35B: F1 0.421 | F1 0.922, sig recall 99.2% | Advisory OOD structural: 76.1% | ~$3.60 |
tool-caller sft | Schema-constrained tool calling | 1.7B | Prompted 4B: val 0.854 | val 0.874, valood 0.849 | 1.0000 valid with zero grammar tax | ~$0.26 |
Continuous Hardening
The Four Operational Feedback Loops
The defining property is the self-learning, self-hardening closed loop: observed operational failures inform code changes, while the models produced by the platform actively supervise and monitor subsequent training runs. The system gets harder to break every time it breaks.
Incidents to Code
Observed hardware faults, stale drivers, frozen container pulls, and billing incidents inform automated controls and preflight assertions. Includes WAN flap grace windows, SSH proxy fallbacks, and autonomous OOM healing.
Protocolized Research
Training days start with automated knowledge checks for new architectures, toolchain updates, and live market rates. Research triggers action only if a candidate plausibly beats the incumbent by 2 points or halves cost.
Self-Learning Operational Supervision
The models produced by the pipeline actively supervise future training runs. The monitoring ladder runs deterministic detection first, small specialist triage second (ops-doctor and log-doctor), and frontier LLM escalation only for ambiguous residue with capped spend budgets.
Tests Police the Tests
The harness maintains 100% branch coverage with mutation testing over money-critical paths. When a mutation gate broke silently, the fix made unverified execution a fatal exit, killing ten surviving mutants in cost accounting.