System Architecture

The Provider Seam, Verification Layer, and Self-Hardening Closed Loop

How tuneharness decouples marketplace dependencies from hard financial guardrails, validates hardware before training begins, scales to distributed workloads, and hardens itself through recursive model supervision.

architectural topology
STDLIB PYTHON
+-------------------------------------------------------------------------+
|                          OPERATOR CONFIGURATION                         |
|  - Max hourly rate ceiling ($0.35/hr)   - Min reliability floor (0.98)  |
|  - Min download bandwidth (400 Mbps)    - Auto-quarantine cooldown      |
+------------------------------------+------------------------------------+
                                     |
                                     v
+------------------------------------+------------------------------------+
|                         TUNEHARNESS POLICY CORE                         |
|  [Ledger] -> Append-only create, stop, destroy lifecycle tracking       |
|  [Leases] -> Label reconciliation (prevents duplicate instance leaks)   |
|  [Reputation] -> Time-decayed Beta posterior scoring per host           |
|  [Watchdog] -> 30-min idle sampler; stops unmonitored compute burn      |
|  [Gate] -> Deterministic deploy evaluation on held-out test splits      |
|  [Self-Healing] -> Closed-loop OOM remedy and model-supervised triage   |
+------------------------------------+------------------------------------+
                                     |
              +----------------------+----------------------+
              | Provider Neutral Seam (Offer, Instance, State)|
              +----------------------+----------------------+
                                     |
                +--------------------+--------------------+
                |                                         |
                v                                         v
     +---------------------+                   +---------------------+
     |   vast.ai Adapter   |                   |   RunPod Adapter    |
     | - CLI JSON wrapper  |                   | - REST v1 Lifecycle |
     | - Auto-retry parser |                   | - REST v2 Catalog   |
     +---------------------+                   +---------------------+

1. The Seam: Where Vendor Stops and Policy Starts

A stdlib-only Python package wraps vendor CLIs rather than raw HTTP APIs, so vendor API drift is the vendor's problem. On top of that sit guardrailed search and provisioning, rsync-based code sync, tmux-managed training sessions, regex-parsed progress reporting, an idle watchdog, and a lifecycle ledger that records every create, stop, and destroy with price and reason.

The load-bearing decision in that stack is where the vendor stops and the policy starts. Exactly one module talks to the marketplace, and it does so through provider-neutral abstractions: offers, instances, prices, and lifecycle states. The guardrails, the ledger, the delivery verification, the watchdog, the self-healing loop, and the deploy gate never mention vast.ai or RunPod.

The practical consequence is that the provider is a dependency while the policy layer is the product. Swapping or adding a marketplace changes one adapter, and every hard-won rule about money, verification, and honesty carries over intact.

2. Trust Nothing: The Post-Boot Verification Pipeline

The core design principle is that marketplace hosts are untrusted until proven otherwise on every single boot:

3. Asymmetric Stop and Destroy

Money is treated as a first-class failure mode. Automation is permitted to stop an instance (saving GPU costs while billing pennies for disk), but only the human operator is permitted to destroy an instance. A mistaken stop costs a few cents; a mistaken destroy loses a training run and its checkpoints.

Duplicate instance leaks are closed structurally: whenever an instance creation times out, the lease manager reconciles by label against the vendor API before any retry, preventing orphan instances from billing forever.

4. The Self-Learning and Self-Hardening Closed Loop

The platform hardens itself with every failure: observed hardware faults, stale drivers, frozen container pulls, filter bugs, and billing incidents inform controls and preflight assertions.

Concurrently, the models produced by the pipeline actively supervise subsequent training runs. The monitoring ladder runs deterministic detection first, small specialist triage second (ops-doctor for GPU-fleet policy and log-doctor for raw log parsing), and frontier LLM escalation only for ambiguous residue with capped spend budgets. This creates a recursive closed loop where the operational framework and the intelligence layer continuously reinforce each other.

5. Research Direction: Distributed Training & Heterogeneous Fault Tolerance

Documented experiments include specialist runs and bounded single-node multi-GPU DDP and QLoRA. Larger distributed training is a research direction, not evidence of arbitrary-scale or frontier-model training. See Cisco Caceres's research agenda.

Proposed work would test the following designs under explicit topology, failure, cost, and recovery measurements:

6. Comparison with SkyPilot and dstack

SkyPilot and dstack are excellent schedulers for multi-cloud placement across friendly, trustworthy providers. tuneharness solves the complementary problem: survival, fraud prevention, and honesty when renting cheap consumer GPUs from strangers.

CapabilitySkyPilotdstacktuneharness
Multi-cloud spot orchestrationStrongStrongNone (by design)
Renter-side compute preflight (bf16 matmul)NoNoYes (10s fail-fast)
Frozen-pull watchdog (6 min kill)Assumes datacenterAssumes datacenterYes
Renter-side quarantine & Beta reputationNoNoYes
Ceiling a project's own config cannot raiseNoNoYes (operator locked)
Append-only cost ledger surviving destroyNoNoYes
Artifact-hash-before-destroy guaranteeNoNoYes
Deterministic deploy gate with canary testsOut of scopeOut of scopeYes