Guarded Training and Evaluation on Rented GPUs

Training on rented GPUs, with failures kept in view.

TuneHarness grew from repeated training, provisioning, and model-evaluation experiments on rented GPUs. Its operating tools check hardware, reconcile instances, track costs, and gate releases on task-specific evaluations. Completed training work uses single-node multi-GPU execution; distributed fleet training and supervision by trained models remain design goals. Saved-output comparisons, native checkpoint probes, and newer runtime checks have separate evidence and limitations, rather than establishing a universal reliability or cost advantage.

Total Compute Spend
$41.62
Across 14 active training days with every boot failure and dead host included.
Failures in this sample
62.0%
75 of 121 distinct instance creations in the dated ledger sample did not deliver a verified GPU.
Spend on Dead Hosts
$4.96
Recorded spend on those 75 failed instances; this sample does not estimate marketplace-wide failure rates or counterfactual savings.
Gate Refusals
6 of 16
Refusals are the point: candidate regressions die in staging before ever touching live serving.

Interactive CLI Walkthrough

A stdlib-only policy layer with explicit verification checks

Explore real terminal sessions recorded from actual marketplace runs. Click each workflow tab to inspect how the harness provisions, preflights, judges, and heals training jobs.

tuneharness cli session
LIVE REPLAY

Defensive Engineering

Four invariants that turn strangers' rigs into reliable compute

Cloud orchestrators assume friendly datacenter hosts. Marketplace hosts are consumer rigs run by strangers. tuneharness assumes every host is lying until verified by renter-side code, while closed-loop automation turns every incident into code.

INVARIANT 01

Post-Boot CUDA Preflight

An instance that passes nvidia-smi can still fail on real compute due to driver/toolkit mismatches. The harness runs an immediate torch.cuda.init() and bf16 matmul. Bad hosts are terminated and replaced in ten seconds, preventing the 105-minute silent idle burns typical of naive rentals.

Time to detection: 9.4s median
INVARIANT 02

Provider-Neutral Seam & Distributed Scaling

Exactly one module talks to vendor CLIs (vast.ai and RunPod). The policy layer (guardrails, quarantine, reputation scoring, cost ledger, watchdog, and deploy gates) never mentions the provider. Marketplace adapters separate provider access from policy. Multi-node placement and distributed fault tolerance require their own execution evidence.

Conforms across vast.ai & RunPod
INVARIANT 03

Money as a First-Class Failure Mode

The append-only ledger records create, stop, and destroy events with timestamps, rates, and reasons. Automated watchdogs sample GPU utilization on a cron and stop idle instances. Stop and destroy are asymmetric: automation may stop, but only the operator destroys.

Total bad-host loss: $4.96 across 75 failures
INVARIANT 04

Deploy Gate with Teeth

No model reaches production without beating its prompted generalist baseline on held-out data. Evaluations are judged on the worst of two seeds, tested on 10 exact failure canaries, and evaluated on format-disjoint out-of-distribution slices. Six candidate models have been refused to date.

Exit code contract: missing metrics = failure

Applied model-operations research

TuneHarness continues experimental work originally conducted under the VastOps name: operational specialists, voice turn detection, document field extraction, tool-calling method comparisons, prompt-schema ablations, noisy ASR benchmarks, and small-transformer training. Single-node multi-GPU DDP runs are documented; they do not establish multi-node or frontier-model training. Read the experiment history and its limits.

Specialist-model experiments and rental records support research into evaluation gates, compute reliability, and operating cost. The ledger is a dated operator sample, not a marketplace failure probability. Distributed training is a research extension requiring separate measured runs. Read Cisco Caceres's research agenda.

Illustrative Scenario Model

Interactive GPU Marketplace Risk & Cost Calculator

Explore a cost scenario using editable workload assumptions and fixed example hourly rates. The outputs are calculated estimates, not measured savings, current price quotes, or a forecast. The 62% default is the observed share of failed instances in one dated, 121-instance operator ledger; it is not a marketplace-wide failure rate. The modeled fast-fail comparison uses assumed failure-handling times, so it does not establish savings caused by TuneHarness.

Training Runs Per Month12 runs
Training Hours Per Run3.0 hrs
Assumed host failure share62%
62.0% was observed in one 121-instance ledger sample; adjust this scenario input as needed.
Cloud scenario cost:$115.20
Marketplace scenario with assumed delays:$52.93
Guarded marketplace scenario:$29.39
Modeled failure-cost difference:$23.54 lower (96%)
Modeled total-cost difference vs. cloud:$85.81 lower (74%)

Deterministic Deploy Gate

Where regressions die as candidates

A deploy gate that only ever passes models is theater. This gate earns its keep by saying no. Inspect real candidate promotion trials from our production history.

Active Candidate:
Final Gate Verdict

Empirical Grounding

The models, measured on held-out tasks

All evaluations are held-out and judged on the worst of two seeds. Baseline comparison is qwen3.5:35b-a3b (35B MoE generalist) running on our 24GB serving box.

ModelTaskSizeBaseline ReferenceFine-Tune ResultHonest Slice / CaveatTrain Cost
ops-doctor v4GPU-fleet action policy8BIncumbent v2: 96.7%99.0% action acc, 1.2% missedSame-simulator eval~$0.13
log-doctor v1Raw train.log triage (8 states)4BPrompted 35B: 93% acc100% acc, 0% missed, 0.56sRead 100% as eval exhausted~$0.35
ambie-eou v2Voice turn detection0.6BPrompted 35B: 85% @ 524ms98.3% acc, 0.0% early-commit, 140msExternal OOD: 90.0% acc~$0.30
sign-fielder v3E-sign field placement4BPrompted 35B: F1 0.421F1 0.922, sig recall 99.2%Advisory OOD structural: 76.1%~$3.60
tool-caller sftSchema-constrained tool calling1.7BPrompted 4B: val 0.854val 0.874, valood 0.8491.0000 valid with zero grammar tax~$0.26

Continuous Hardening

The Four Operational Feedback Loops

The defining property is the self-learning, self-hardening closed loop: observed operational failures inform code changes, while the models produced by the platform actively supervise and monitor subsequent training runs. The system gets harder to break every time it breaks.

LOOP 01

Incidents to Code

Observed hardware faults, stale drivers, frozen container pulls, and billing incidents inform automated controls and preflight assertions. Includes WAN flap grace windows, SSH proxy fallbacks, and autonomous OOM healing.

Remedies recorded in an append-only lesson ledger
LOOP 02

Protocolized Research

Training days start with automated knowledge checks for new architectures, toolchain updates, and live market rates. Research triggers action only if a candidate plausibly beats the incumbent by 2 points or halves cost.

LOOP 03

Self-Learning Operational Supervision

The models produced by the pipeline actively supervise future training runs. The monitoring ladder runs deterministic detection first, small specialist triage second (ops-doctor and log-doctor), and frontier LLM escalation only for ambiguous residue with capped spend budgets.

Trained models monitor future training runs
LOOP 04

Tests Police the Tests

The harness maintains 100% branch coverage with mutation testing over money-critical paths. When a mutation gate broke silently, the fix made unverified execution a fatal exit, killing ten surviving mutants in cost accounting.

Mutation-tested cost and lease accounting