Financial Grounding

Rented compute: an operator's cost ledger sample

The recorded lifecycle and spend figures below come from one operator's ledger for 121 distinct instances from 11 July through 15 September 2026. In this sample, 75 instances did not deliver a verified working GPU. This bounded sample does not estimate marketplace-wide failure rates or show what the same rentals would have cost without TuneHarness.

Substrate MeasureObserved ValueAccounting Definition & Grounding
Distinct Instance Creates121Counted by distinct provider instance IDs across all campaigns.
Total Create Events127Includes immediate retries following transient API timeouts.
Never-Delivered Instances75 (62.0%)Instances that froze during pull, failed SSH, or died on bf16 preflight.
Time to Running (Median / p95)2.4 / 8.6 minMeasured only when a host successfully delivered a verified GPU.
Mean Alive per Bad Host~12.3 minShort lifetime enforced by 6-min frozen pull watchdog & 10s preflight.
Spend on Bad Hosts$4.9611.9% of total campaign spend. Absorbed without stalled engineering.
Total Campaign Marketplace Spend$41.62Includes all 121 instances, all 75 failures, and local verification.
Gated Model Versions16Model candidate iterations evaluated through the framework gate.
Deploy Gate Refusals6 (38%)Refused due to statistical shortfall, canary failure, or format clash.

Why Two in Three Rented Hosts Fail to Deliver

Roughly two in three creates died before any training occurred, and on documented platform-wide bad days the failure rate reached three in four. That is not the pipeline failing: that is the consumer GPU marketplace being what it is.

Marketplace providers aggregate rigs located in residential basements, student apartments, and makeshift mining frames. Rigs report modern GPUs while running six-year-old NVIDIA drivers, sit behind throttled 20 Mbps residential uplinks that freeze on a 15GB container pull, or crash the moment PyTorch allocates a tensor.

The ledger records $4.96 in spend across those 75 failed instances. That is an observed cost in this sample; it does not by itself measure how much TuneHarness saved relative to a different failure-handling policy.

Interactive Spend Comparison Calculator

Use the calculator below for illustrative scenarios under fixed example hourly rates and editable workload assumptions. Its outputs are modeled estimates, not current provider quotes, measured savings, or forecasts. The default 62% failure share comes from this operator's 121-instance sample and is not a marketplace-wide rate; failure-handling durations are also assumptions.

Training Runs Per Month12 runs
Training Hours Per Run3.0 hrs
Assumed host failure share62%
62.0% was observed in one 121-instance ledger sample; adjust this scenario input as needed.
Cloud scenario cost:$115.20
Marketplace scenario with assumed delays:$52.93
Guarded marketplace scenario:$29.39
Modeled failure-cost difference:$23.54 lower (96%)
Modeled total-cost difference vs. cloud:$85.81 lower (74%)

The Five Money Traps We Closed With Code

Every defensive guard in tuneharness exists because a real incident cost money:

  1. The 105-Minute Stale-Driver Idle Burn: A host passed nvidia-smi, accepted the container, and sat idle while training failed to initialize due to a driver incompatibility. An operator noticed after 105 minutes. We wrote the post-boot bf16 matmul preflight the same evening; now bad drivers are caught and destroyed within ten seconds.
  2. The 30-Minute Frozen Container Pull: Consumer hosts with saturated uplinks frequently freeze on multi-gigabyte layers. The boot progress parser now enforces a hard six-minute progress deadline.
  3. The Duplicate Instance Leak: When an API call timed out, retrying naively left an unrecorded instance running on the provider. The lease manager now reconciles by deterministic label before issuing any retry.
  4. The Ctrl-C Abandoned Rental: An operator terminating a local CLI with Ctrl-C once left a GPU instance running overnight. Exit traps now intercept all abort signals, execute an automatic STOP command, and preserve the disk volume.
  5. The Silent Filter That Matched Nothing: A search filter query emitted gpu_ram>=24576 (MB) while vast.ai expected GB, causing the search to return zero offers and appear as a dead market. We added live-market assertion tests that verify filters actually match real inventory.