Financial Grounding
Rented compute: an operator's cost ledger sample
The recorded lifecycle and spend figures below come from one operator's ledger for 121 distinct instances from 11 July through 15 September 2026. In this sample, 75 instances did not deliver a verified working GPU. This bounded sample does not estimate marketplace-wide failure rates or show what the same rentals would have cost without TuneHarness.
| Substrate Measure | Observed Value | Accounting Definition & Grounding |
|---|---|---|
| Distinct Instance Creates | 121 | Counted by distinct provider instance IDs across all campaigns. |
| Total Create Events | 127 | Includes immediate retries following transient API timeouts. |
| Never-Delivered Instances | 75 (62.0%) | Instances that froze during pull, failed SSH, or died on bf16 preflight. |
| Time to Running (Median / p95) | 2.4 / 8.6 min | Measured only when a host successfully delivered a verified GPU. |
| Mean Alive per Bad Host | ~12.3 min | Short lifetime enforced by 6-min frozen pull watchdog & 10s preflight. |
| Spend on Bad Hosts | $4.96 | 11.9% of total campaign spend. Absorbed without stalled engineering. |
| Total Campaign Marketplace Spend | $41.62 | Includes all 121 instances, all 75 failures, and local verification. |
| Gated Model Versions | 16 | Model candidate iterations evaluated through the framework gate. |
| Deploy Gate Refusals | 6 (38%) | Refused due to statistical shortfall, canary failure, or format clash. |
Why Two in Three Rented Hosts Fail to Deliver
Roughly two in three creates died before any training occurred, and on documented platform-wide bad days the failure rate reached three in four. That is not the pipeline failing: that is the consumer GPU marketplace being what it is.
Marketplace providers aggregate rigs located in residential basements, student apartments, and makeshift mining frames. Rigs report modern GPUs while running six-year-old NVIDIA drivers, sit behind throttled 20 Mbps residential uplinks that freeze on a 15GB container pull, or crash the moment PyTorch allocates a tensor.
The ledger records $4.96 in spend across those 75 failed instances. That is an observed cost in this sample; it does not by itself measure how much TuneHarness saved relative to a different failure-handling policy.
Interactive Spend Comparison Calculator
Use the calculator below for illustrative scenarios under fixed example hourly rates and editable workload assumptions. Its outputs are modeled estimates, not current provider quotes, measured savings, or forecasts. The default 62% failure share comes from this operator's 121-instance sample and is not a marketplace-wide rate; failure-handling durations are also assumptions.
The Five Money Traps We Closed With Code
Every defensive guard in tuneharness exists because a real incident cost money:
- The 105-Minute Stale-Driver Idle Burn: A host passed
nvidia-smi, accepted the container, and sat idle while training failed to initialize due to a driver incompatibility. An operator noticed after 105 minutes. We wrote the post-boot bf16 matmul preflight the same evening; now bad drivers are caught and destroyed within ten seconds. - The 30-Minute Frozen Container Pull: Consumer hosts with saturated uplinks frequently freeze on multi-gigabyte layers. The boot progress parser now enforces a hard six-minute progress deadline.
- The Duplicate Instance Leak: When an API call timed out, retrying naively left an unrecorded instance running on the provider. The lease manager now reconciles by deterministic label before issuing any retry.
- The Ctrl-C Abandoned Rental: An operator terminating a local CLI with Ctrl-C once left a GPU instance running overnight. Exit traps now intercept all abort signals, execute an automatic STOP command, and preserve the disk volume.
- The Silent Filter That Matched Nothing: A search filter query emitted
gpu_ram>=24576(MB) while vast.ai expected GB, causing the search to return zero offers and appear as a dead market. We added live-market assertion tests that verify filters actually match real inventory.