Research reports / Dated research notes
Standalone verifier-selection engineering report (HTML, PDF, Markdown and aggregate results)
Research execution fieldnote | 6 October 2026 UTC
Cisco Caceres. AI assistance supported implementation, literature screening, artifact review and writing.
This developer-run report reproduces six historical tool-calling output sets exactly, evaluates a 120-task local verifier/selection pilot with isolated execution, and measures acoustic robustness on frozen speaker-separated recordings. It retains the first verifier contract's failures and the severe-noise ASR and detector failures. A checked attention/optimization reference runs on two local hosts; Echo's recovery fixture exercises actual CLI processes. The sections below identify the inputs, measurements and limits of each result.
These are engineering and exploratory findings, not outside peer review, a novel-method claim or frontier-model performance. Paid model inference, rented compute and training expenditure for this reported work: $0. Local electricity, amortized hardware and labor were not measured.
This note separates original freezes from subsequent execution and audit. Original protocols remain historical records; follow-up source changes, decoder warnings and artifact recovery are dated additions, not retroactive preregistration. The dated additions below distinguish completed Echo V1/V2 qualifications, the V5.1 native environment diagnostic failure, completed ODSS speech-detection diagnostics, and ASVspoof 5 audio preparation that remains incomplete.
Saved tool-calling artifacts: exact rescore, incomplete model recovery
All six historical prediction files contain 400 unique IDs matching their respective split prefixes. Current evaluation reproduced every stored numerical metric and compact correctness record. This is saved-output reproduction: no new generation or training occurred, and stored latency fields are not new performance measurements.
| Split | Call/no-call items | Prompted macro | Historical SFT macro | Difference |
|---|---|---|---|---|
| Validation | 360/40 | 0.673611 | 0.887500 | +0.213889 |
| Out-of-domain validation | 369/31 | 0.652942 | 0.849987 | +0.197045 |
| No-call canaries | 0/400 | 0.627500 | 0.927500 | +0.300000 |
Macro equally weights call exact-match and correct no-call accuracy when both classes exist. Canaries measure only no-call accuracy. Post-hoc paired item bootstrap and per-class exact McNemar diagnostics are exploratory; prefix sampling, shared tool/template clusters and pretraining exposure limit inference. No production or population superiority is established.
A later read-only Git recovery pass recovered a complete 3,441,185,608-byte safetensors container, SHA256 db9639dd57242d536149ef49e78735c759748eee19868a3b2e256f9dad2b2b59. Ancestry and matching metadata identify abl-sft-names, a Qwen3-1.7B schema-ablation arm (1,200 steps, learning rate 3e-5, seed 4242), not the original evaluated sft-1.7b checkpoint. At recovery, configuration/tokenizer were absent; the original container alone was not a complete runnable checkpoint. No original selected checkpoint was identified in inspected locations; other archives may exist. Corrupt temporary packs were tested on copies; originals were preserved. Substitution or retraining would constitute a new experiment.
A separately labeled prospective reconstruction now loads the recovered ablation locally. All 310 tensor shapes match the pinned public Qwen3-1.7B configuration, including its tied output head. We used that base snapshot’s cached configuration/tokenizer and the recovered generation configuration; the configuration matched its pinned upstream bytes. PyTorch 2.14.0+cpu and Transformers 5.16.1 loaded the weights, restored the tied head and produced finite logits on a two-token synthetic-input forward pass (CPU BF16, 1.361 seconds including loading). The original recovered files remain unchanged. This establishes local execution viability under declared reconstruction assumptions, not tool-calling accuracy, original training provenance or an exact historical generation reproduction. The original selected SFT remains unidentified.
Artifact: examples/tool-caller/results/record/reproduction-20261006.json; methods and guarded commands: examples/tool-caller/REPRODUCTION.md. Compact tracked records support limited text-free recomputation. Raw corpora and predictions remain private. Qwen3-1.7B and ToolACE publish Apache-2.0 metadata; xLAM irrelevance publishes CC-BY-4.0. Preserve exact revisions, attribution and any gated access requirements rather than inferring source-content rights from repository licensing.
Matched reconstructed tool-caller comparison: completed, with errors retained
The class-balanced macro correct fraction was 69.7222% for the reconstructed names-ablation and 34.7222% for publisher Qwen3-1.7B: a descriptive 35.0 percentage-point difference. Both retained all 400 registered records.
| Endpoint | Reconstructed ablation | Publisher |
|---|---|---|
| Call correct | 223/360 (61.94%) | 52/360 (14.44%) |
| No-call correct | 31/40 (77.50%) | 22/40 (55.00%) |
| Strict JSON contract | 400/400 | 321/400 |
| Parser valid | 400/400 | 362/400 |
The reconstructed model still made 146/400 errors: 137/360 call errors and 9/40 no-call false calls. This does not establish production readiness. The protocol registers both macro and pooled descriptive outputs without an exclusive primary/secondary ranking. Pooled correctness is 254/400 (63.5%) versus 74/400 (18.5%), a 45.0-point difference; its class weighting differs from macro.
Both 400-case collections completed with known usage for all records, no recorded input/output truncation and 400 EOS observations each. Input tokens were 54,968 per arm; returned tokens were 18,976 and 18,456. Greedy CPU BF16, names-only schema, 384 new-token cap and 4096 input-token limit matched. Concurrent CPU/cache/storage conditions were uncontrolled: worker wall times are not CPU costs or controlled serving latency.
The names-only prompt omits parameter signatures, descriptions and required-argument schemas. A catalogue-trained model may retain these in its weights; matching prompts does not establish matching prior information. For this original names-only comparison, no full-schema, native tools-slot, reasoning-enabled or optimized publisher control was evaluated. Accuracy uses permissive parsing plus argument/value comparison; strict JSON compliance is a separate endpoint. This does not establish general tool-calling superiority.
This exposed historical prefix retains five exact training-content matches and eight earlier probe exposures; semantic and pretraining overlap remain unknown. Reconstructed metadata is not recovered training provenance. The original selected historical SFT remains unidentified. Historical six-arm saved-output rescoring used a different generation contract and is not interchangeable with these new collections. The comparison protocol was frozen before publisher generation, after ablation collection had begun and partial outputs were exposed; this is not blind preregistration. No population interval, p-value, causal SFT benefit, method novelty, fresh-data or frontier transfer is claimed.
Source report SHA256 8415cad472e44d3bbfdd7358ccb6881a769354c0af36c1fad26575cf7d40c28e; frozen protocol 713aec34daf88a4a9b154bd0ef08ac7260c4dfcdee56cf5ef5459da8b6d1111e. The accompanying aggregate carries both terminal-summary, plan, source-lock, collector, event and ordered-receipt commitments. Public aggregate inspection cannot rerun generation without authorized private source/model/data access. Paid external spend was $0; electricity, hardware and labor are unmeasured. AI assistance and same-owner checks are disclosed, not outside review.
Dependencies: Qwen3-1.7B pinned publisher revision, ToolACE, xLAM irrelevance. Respect exact revisions, attribution and access terms; raw corpora and predictions are not republished.
A subsequent separately frozen Warden CPU diagnostic qualified both complete FunctionGemma files: all 236 BF16 tensors were finite, the tied head was verified, and repeated synthetic forward outputs of shape [1, 3, 262144] were finite and exactly equal. The only missing parameter was the expected tied lm_head alias. Both arms passed under the pinned runtime; supervisor receipt SHA256 31ccbb328fddcfd9f815a34e324c6cacedcf4bd5d9f80333182f40b41060b3e6, worker summary commitment e6da037affa5e802c590fe15c06aab43f6f20aba7e9ce818266025756b0c213e. This later load/finite-forward qualification does not prove task accuracy, token-generation behavior or training lineage. The truncated checkpoint-400 remains unqualified.
Download the aggregate comparison and source commitments. The artifact contains no raw outputs, per-item IDs, private absolute paths or worker-time cost claims.
Completed publisher information-condition control and argument sensitivity
A subsequent separately frozen full-schema publisher collection completed all 400 slots. Full schemas and the truthful system instruction both changed from the names-only condition; this is a descriptive information-condition comparison, not an isolated schema-only intervention.
| Condition | Call correct | No-call correct | Equal-weight macro | Pooled correct | Incorrect |
|---|---|---|---|---|---|
| Recovered names-ablation | 223/360 | 31/40 | 69.7222% | 254/400 (63.5%) | 146 |
| Publisher names-only | 52/360 | 22/40 | 34.7222% | 74/400 (18.5%) | 326 |
| Publisher full-schema + instruction | 235/360 | 27/40 | 66.3889% | 262/400 (65.5%) | 138 |
The same publisher weights changed from 74/400 to 262/400 pooled correct (+47 percentage points); equal-weight class macro changed from 34.7222% to 66.3889% (+31.6667 points). Both registered descriptive summaries are retained. Full-schema publisher still made 138/400 errors (125 call errors and 13 no-call errors). Its 262 pooled correct versus reconstructed names-ablation's 254 does not imply a better overall model: their class outcomes, information conditions and model priors differ.
All 400 full-schema slots completed with known usage, no observed input/output truncation, 346 strict-format and 370 permissive-parser-valid responses. Input tokens were 171,638 versus 54,968 for names-only; returned tokens were 21,377 versus 18,456. These are token accounting, not energy/latency measurements. Full schemas and instruction changed together; reasoning-enabled, optimized and native-tools publisher controls were not run.
The separately frozen, post-collection type-stricter argument comparator reconciled all original labels across the three arms and changed 0/1,200 correctness labels. It distinguishes boolean from numeric values, disallows false-valued argument-container coercion and avoids lossy large-integer normalization. This is argument-correctness sensitivity, not strict JSON-format scoring; the original published scorer and results remain unchanged. Zero changes weaken those specific scoring concerns for this measured cohort, not for all future outputs. Calls were not executed.
Five exact training-content matches and eight prior probe exposures remain in the historical prefix. This does not establish causal training benefit, fresh/general tool-calling superiority, novelty, frontier transfer or production readiness. The separately reported FunctionGemma lossy-native invocation-collapse diagnostic remains unchanged; its cause is unknown from aggregate review. External paid model/compute spend was $0; electricity, hardware/labor and latency economics were not measured. Developer/AI-assisted internal checks are not outside review.
Download the aggregate information-condition and sensitivity results. It contains class/format/usage/paired counts and source commitments, without outputs, per-item IDs or private paths.
FunctionGemma native input fit and incomplete schema inventory
A subsequent tokenizer-only native fit retained all 400 historical inputs within the 4,096-token cap, maximum 1,081, with zero oversize inputs and zero model loads. Input fit establishes neither generation accuracy nor equivalent schema information. A separate native-schema audit examined 1,107 tool representations: declarations parsed on 381 records and 19 remained unsupported. Field-comparison coverage is therefore 381/400 and incomplete. The audit used synthetic user substitution without queries or reference-answer fields. Omission/addition counts are field events that can co-occur, not record error rates. These tokenizer-only stages establish neither an equivalent full-schema native interface nor task accuracy.
Download the aggregate native input-fit and schema inventory. It includes source-result and template commitments without raw queries, predictions, record IDs or private paths. Read the draft methods case study, an editorial publication copy of the committed research draft; it is not outside peer review. The subsequently completed publisher information-condition and argument-sensitivity results are reported above.
FunctionGemma native comparison: completed invocation-collapse diagnostic
A separately frozen same-lossy-native FunctionGemma comparison completed both 400-record collections. The base scored 228/360 call and 0/40 no-call correct; checkpoint-600 scored 0/360 call and 40/40 no-call. The checkpoint showed invocation collapse in this condition, not improved tool calling. Class-balanced correctness rose from 31.6667% to 50% (+18.3333 percentage points), while pooled correctness fell from 228/400 (57%) to 40/400 (10%, −47 points). This illustrates why neither macro nor pooled should replace the two class outcomes.
Paired call cells were 228 base-only and 132 both-wrong; all 40 no-call cases were checkpoint-only.
The base had 347 native-call parses and 53 malformed outputs; checkpoint-600 had 400 natural-text classifications, no native-call parses and eight output truncations. All 400 cases per arm completed with known usage; input tokens were 139,237 each, returned tokens 18,686 versus 21,026. The base stopped on token50 for all400; checkpoint had392 EOS1 stops and8 capped continuations.
No-call success measures ordinary nonempty text without invocation, not semantic refusal quality. Aggregate-only review does not explain the content or cause of the checkpoint's behavior, and no repair or retry was performed.
Both arms used the same lossy declarations; the 19 unsupported schema-audit records stayed in the denominator. Five training-content overlaps and eight prior probe exposures remain; this is not equivalent full-schema information, a directly comparable Qwen endpoint, fresh or causal generalization, frontier transfer, or outside peer review. Paired analysis SHA256 5d690a197d9392c70c6d6cce6d1236008857ea3c30b263ddf824ec0ce4990ff1.
Public methods and aggregate arithmetic checker
A separately frozen full-schema publisher control subsequently completed all 400 records, reported above; its tokenizer-only preflight retained all records with no context overflow. It does not rewrite the original names-only study. Full schemas and the truthful system instruction change the information condition; this remains the exposed historical prefix, not a fresh or causal training evaluation.
Download the public methods appendix and the standalone aggregate arithmetic checker. The checker uses Python's standard library: python3 check-tool-caller-comparison.py tool-caller-names-comparison-20261006.json. It checks public aggregate arithmetic, not generations, correctness labels or private-source reproduction.
What the downloadable aggregates reproduce
The tool-caller JSON supports arithmetic checks of published class counts, paired contingency tables and registered descriptive differences. For each arm, equal-weight macro is (call_correct / 360 + no_call_correct / 40) / 2; pooled correct fraction is (call_correct + no_call_correct) / 400. The macro difference is 35 percentage points and pooled difference 45 percentage points. Both summaries were registered; neither is an exclusive prospective primary endpoint. Paired cells retain all 400 matched slots. Strict-format validity and permissive parser validity are distinct from argument/value correctness and are reported separately.
The downloadable check-tool-caller-comparison.py checks those identities against the public aggregate. Passing this check does not reproduce correctness labels, generations, item selection, exposure detection or parser behavior. SHA256 commitments identify a fixed source; a hash alone supplies neither source access nor permission.
The frozen renderer compares calls in the same order with the same function names. Argument dictionaries are normalized by key order; integer and floating-point values are converted to floats, while list order and string case are preserved. Argument equality is permissive, not type-strict: Python equality can treat boolean True as equal to numeric 1 after normalization (and False as equal to 0). Missing or other false-valued argument containers are also mapped to an empty dictionary by the comparator. Strict JSON shape is measured separately; it is not an additional requirement of the permissive correctness endpoint. Generated function calls are never executed. These are existing frozen scoring semantics, not a revised scorer or a claim of execution success.
Both arms used names-only catalogues, greedy generation, 384 maximum new tokens, 4096 maximum input tokens, bfloat16, four intra-op threads and one inter-op thread, with pinned Torch/Transformers versions recorded in the aggregate. Names-only catalogues omit signatures, descriptions and required-argument schemas. Catalogue-trained weights may retain information omitted from publisher prompts. For the original published names-only comparison, no full-schema/native-tools/reasoning-enabled/optimized publisher control was completed. A separately versioned full-schema control is currently running and has no completed results in this appendix. Concurrent CPU/cache/storage conditions were uncontrolled, so timing cannot establish a speed advantage. External spend is $0; electricity and total economic cost were not measured.
The verifier-selection pilot aggregate describes cached replay, not three independently regenerated model pools. Its original 49-task subset remains fixed; reference qualification and hidden correctness coverage are separate. Aggregate totals cannot reconstruct task-cluster bootstrap intervals or individual policy decisions. Public intervals are conditional on the collected pool and multiple exploratory contrasts are unadjusted. Full permitted task-level decisions and separately supplied grading outcomes would be necessary for exact interval replay; they are excluded from this aggregate-only release.
For genuine reproduction, an approved access bundle must provide the frozen protocol and permitted source versions, model/tokenizer/configuration commitments, data-rights boundaries, exact rendering/scoring/selection contracts, environment, qualified isolated grading runner, failure accounting and authorized decisions/outcomes. Original selected SFT 1.7B remains missing; reconstructed names-ablation is a different artifact. FunctionGemma evidence is finite forward/load qualification, not task accuracy. H1 five-source builds, installed namespace/dependency compatibility and all 24 reference execution remain pending. Internal model/agent assistance is not outside scientific review.
Verifier development: retain contract failures and saturated success
V2 produced fourteen candidate responses that failed the required contract because they used Markdown. These remain failures; they were not repaired or discarded as warmups. V3 changed the collection contract before its own execution. Twelve candidates across six development tasks then passed both public and held-out hidden tests in an actual disposable networkless VM. Neither version is the proposed confirmation pilot.
The safe V3 VM outcome deposit contains 43 cases: seven qualification cases (one expected acceptance), twelve public candidates, twelve hidden candidates, and six canonical reference cases for each grader. All candidate and reference cases pass. Reports record no NIC, no host mounts and no observed root escape. Guest script hashes match installed source; a separate same-owner review matched five installed files to their manifest and passed six guard tests. The summary records the executed host-runner hash separately from a later strengthened version; the executed snapshot is preserved. The launch template hash was not deposited. Inspection of strengthened current source does not establish those exact bytes were used at historical launch; future runs should deposit all launch hashes.
The saturated candidate set cannot estimate specificity against incorrect candidates, demonstrate benefit from a reliability-aware policy, test H1, measure joint-error advantage, establish natural task-family shift or support frontier transfer. Generated code and tests share an interpreter: accidental-exit guards do not establish grading integrity against malicious candidates. Isolation is engineering evidence, not an independent assessor's attestation.
Methods: examples/verifier-reliability/docs/PILOT-PROTOCOL.md, DEVELOPMENT-PROTOCOL-V3-20261006.md and scripts/NATIVE-VM-GRADING.md. The pinned HumanEval source is MIT-licensed and useful for engineering qualification; its 2021 tasks cannot establish fresh-data generalization.
The updated research proposal remains an open hypothesis. Its closest-overlap screen includes ADAP, CAPS, DAJ, GRACE, Inference-Time Pessimism and Hedging/HedgeTune. Some entries remain abstract-only screens. Matched information and cost, full algorithm review and applicable baselines are necessary before novelty or advantage claims. No author implementation was copied merely because a repository exists.
Local verifier pilot: descriptive engineering results
The unfiltered first-candidate comparator passes 109/120 tasks (90.83%). Soft best-of-n reaches 91.39% at the three larger token caps, a +0.56 percentage-point three-seed task-mean difference; its paired descriptive 95% interval is [0, +1.67 percentage points]. The lowest cap yields no gain. Other unfiltered policies match the first-candidate result; all policies on the fixed 49-task public-qualified subset score 47/49 (95.92%). These are exploratory cached-pool results, not confirmatory superiority.
A clearly post-hoc headroom diagnostic finds 437/480 hidden-test candidate passes, with 108 all-pass pools, ten all-fail pools and only two mixed pools. An oracle choosing any passing candidate could reach 110/120 tasks, leaving just one task (0.83 percentage points) of possible improvement over the first candidate. This small headroom limits what the pilot can reveal about selection methods; it is not a test of the proposed H1.
Collection used pinned qwen3-coder:30b and qwen3.5:9b through Warden Ollama 0.35.1. Five owned networkless VM batches reconciled all outcomes. All 9,288 public-only decisions were frozen before hidden-label analysis. A separate same-owner agent recomputed the 40 result groups and 10,000-resample task bootstraps from metadata; that check is disclosed assistance, not outside peer review.
The completed collection covers 120 frozen HumanEval tasks, four generated candidates per task (480 candidates), and 480 paired verifier calls. All 960 requests are accounted for. Hidden grading covers all 480 candidates; public grading covers 196 candidates on the original fixed 49-task subset. Invalid source contracts remain failures without execution.
Actual collection used 276,882 prompt tokens and 100,216 returned tokens. Invalid-judge abstentions: 0. Automatic contract-failure grading slots: 4 (slots count hidden and public tests separately). External API spend was $0; local electricity was not measured.
The prospective reconstructed extractor passed 51/57 public reference controls. The original eligibility subset remains 49 tasks; newly passing references were not added. Exact byte parity with the historical inline extractor is not claimed.
Selection was frozen before hidden-label analysis. The table describes cached replay on the same collected candidate pools, at four conservative input-plus-output token caps and three policy seeds (0, 1, 2). Policy seeds and candidates are not independent benchmark samples. Costs below are counterfactual policy consumption, distinct from actual collection cost.
On the 49-task public-qualified subset, single remains an unfiltered first-candidate comparator: it does not invoke or require a passing public test. The other five policies apply the public test filter. Subset membership and public-test filtering are distinct.
| Task subset | Token cap | Policy | Task-mean success | Paired difference vs single [descriptive 95% CI] | Mean consumed tokens | Cap-exhausted seeds |
|---|---|---|---|---|---|---|
| 120 unfiltered | 4,608 | best_of_n | 90.83% | +0.00% [+0.00%, +0.00%] | 566.7 | 360/360 |
| 120 unfiltered | 4,608 | bounded_best_of_poisson | 90.83% | +0.00% [+0.00%, +0.00%] | 566.7 | 360/360 |
| 120 unfiltered | 4,608 | single | 90.83% | +0.00% [+0.00%, +0.00%] | 395.0 | 0/360 |
| 120 unfiltered | 4,608 | soft_best_of_n | 90.83% | +0.00% [+0.00%, +0.00%] | 566.7 | 360/360 |
| 120 unfiltered | 9,216 | best_of_n | 90.83% | +0.00% [+0.00%, +0.00%] | 3,109.4 | 9/360 |
| 120 unfiltered | 9,216 | bounded_best_of_poisson | 90.83% | +0.00% [+0.00%, +0.00%] | 3,109.4 | 9/360 |
| 120 unfiltered | 9,216 | single | 90.83% | +0.00% [+0.00%, +0.00%] | 395.0 | 0/360 |
| 120 unfiltered | 9,216 | soft_best_of_n | 91.39% | +0.56% [+0.00%, +1.67%] | 3,109.4 | 9/360 |
| 120 unfiltered | 18,432 | best_of_n | 90.83% | +0.00% [+0.00%, +0.00%] | 3,142.5 | 0/360 |
| 120 unfiltered | 18,432 | bounded_best_of_poisson | 90.83% | +0.00% [+0.00%, +0.00%] | 3,142.5 | 0/360 |
| 120 unfiltered | 18,432 | single | 90.83% | +0.00% [+0.00%, +0.00%] | 395.0 | 0/360 |
| 120 unfiltered | 18,432 | soft_best_of_n | 91.39% | +0.56% [+0.00%, +1.67%] | 3,142.5 | 0/360 |
| 120 unfiltered | 36,864 | best_of_n | 90.83% | +0.00% [+0.00%, +0.00%] | 3,142.5 | 0/360 |
| 120 unfiltered | 36,864 | bounded_best_of_poisson | 90.83% | +0.00% [+0.00%, +0.00%] | 3,142.5 | 0/360 |
| 120 unfiltered | 36,864 | single | 90.83% | +0.00% [+0.00%, +0.00%] | 395.0 | 0/360 |
| 120 unfiltered | 36,864 | soft_best_of_n | 91.39% | +0.56% [+0.00%, +1.67%] | 3,142.5 | 0/360 |
| 49 public-qualified | 4,608 | best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 541.9 | 147/147 |
| 49 public-qualified | 4,608 | bounded_best_of_poisson | 95.92% | +0.00% [+0.00%, +0.00%] | 541.9 | 147/147 |
| 49 public-qualified | 4,608 | first_public | 95.92% | +0.00% [+0.00%, +0.00%] | 328.1 | 6/147 |
| 49 public-qualified | 4,608 | single | 95.92% | +0.00% [+0.00%, +0.00%] | 328.1 | 0/147 |
| 49 public-qualified | 4,608 | soft_best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 541.9 | 147/147 |
| 49 public-qualified | 4,608 | uniform_public | 95.92% | +0.00% [+0.00%, +0.00%] | 328.1 | 147/147 |
| 49 public-qualified | 9,216 | best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 9,216 | bounded_best_of_poisson | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 9,216 | first_public | 95.92% | +0.00% [+0.00%, +0.00%] | 372.8 | 0/147 |
| 49 public-qualified | 9,216 | single | 95.92% | +0.00% [+0.00%, +0.00%] | 328.1 | 0/147 |
| 49 public-qualified | 9,216 | soft_best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 9,216 | uniform_public | 95.92% | +0.00% [+0.00%, +0.00%] | 1,304.6 | 0/147 |
| 49 public-qualified | 18,432 | best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 18,432 | bounded_best_of_poisson | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 18,432 | first_public | 95.92% | +0.00% [+0.00%, +0.00%] | 372.8 | 0/147 |
| 49 public-qualified | 18,432 | single | 95.92% | +0.00% [+0.00%, +0.00%] | 328.1 | 0/147 |
| 49 public-qualified | 18,432 | soft_best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 18,432 | uniform_public | 95.92% | +0.00% [+0.00%, +0.00%] | 1,304.6 | 0/147 |
| 49 public-qualified | 36,864 | best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 36,864 | bounded_best_of_poisson | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 36,864 | first_public | 95.92% | +0.00% [+0.00%, +0.00%] | 372.8 | 0/147 |
| 49 public-qualified | 36,864 | single | 95.92% | +0.00% [+0.00%, +0.00%] | 328.1 | 0/147 |
| 49 public-qualified | 36,864 | soft_best_of_n | 95.92% | +0.00% [+0.00%, +0.00%] | 2,540.5 | 0/147 |
| 49 public-qualified | 36,864 | uniform_public | 95.92% | +0.00% [+0.00%, +0.00%] | 1,304.6 | 0/147 |
Task means average three seeds; paired bootstrap intervals resample tasks and retain paired seed means. These are conditional descriptive intervals for these fixed tasks and candidate pools, with multiple exploratory comparisons. They do not establish confirmatory superiority.
HumanEval is a familiar public benchmark released in 2021. Training contamination, task familiarity, model-family dependence, fixed candidate pools, and a small public-qualified subset limit generalization. No H1, joint-verifier learning, family-shift robustness, frontier-scale capability, or novel research contribution is established. This result has not received external independent review.
Execution timing represents outer wrapper wall time (15-second bound), with an inner five-second wall cap and three-second CPU cap. CPU usage is unmeasured. Raw responses, source code, hidden tests, and private machine paths are excluded from this summary.
Download the aggregate pilot results and commitments. Source/response/test payloads are excluded. Methods: examples/verifier-reliability/docs/POSTCOLLECTION-PROTOCOL-V1.md; analysis SHA256 4fcd3b98255aa15d7a6f58a303a159fb143f9a411c4bc4426c4d424bc67df3cb, frozen-decision SHA256 da9a0cc203fbc44e7511a384cf2257f11becc18c06ab72d8735e44bfdf4d4d25.
Acoustic research: frozen public speech, actual failures
Twelve cached LibriSpeech target speakers were separated into development three, calibration three and test six. Six disjoint train-source speakers supplied interference. Thirty-six complete-recording 16 kHz PCM16 WAVs represent clean, 10 dB and 5 dB conditions of twelve targets, not 36 independent speakers. Byte, finite-signal, no-clipping, unique decoded PCM, speaker/source separation and analytic component-SNR checks passed. Source/cache acquisition and earlier exposure lack independent custody; simulated read-English babble is not federal radio, clinical audio or real deployment noise.
OpenSLR LibriSpeech and the pinned dataset card declare CC-BY-4.0. Source and transformation attribution are retained. Licensed inputs do not establish model-pretraining nonexposure. No proprietary AMBIE model was evaluated.
Pinned local baseline: openai/whisper-tiny.en revision 87c7102498dcde7456f24cfd30239ca606ed9063, CPU float32, four threads, greedy beam one, maximum 256 new tokens, explicit pinned English normalizer with no fallback. Data, core weights, decoder, normalizer and execution source were frozen before inference; separate training-source warmup was not counted. Secondary tokenizer-file and full host environment-lock coverage remain incomplete. Model dependencies derive from Whisper and the model publisher, not proprietary architecture work.
| Six test speakers | Word errors / reference words | WER | CER | Human speech flagged synthetic by Noise or Voice |
|---|---|---|---|---|
| Clean | 7/155 | 4.52% | 1.40% | 4/6 |
| 10 dB babble | 25/155 | 16.13% | 9.26% | 4/6 |
| 5 dB babble | 274/155 | 176.77% | 159.39% | 5/6 |
WER and CER can exceed 100% through insertions. The 5 dB test condition contains 233 word insertions and one conservative returned-token-length warning at 256. Terminal EOS identifiers were not deposited; this warning does not prove truncation or why generation stopped. Zero ASR processing errors occurred. Retain insertion-heavy failures rather than clipping rates or silently regenerating them. Six test speakers and 155 reference words support a bounded diagnostic, not a population estimate or critical-field/radio claim.
The frozen Noise or Voice complete-recording DSP at threshold 0.5 processed all 36 human recordings without abstentions or errors. Its false alarms across all twelve targets were 5/12, 9/12 and 11/12; test-only counts appear above. There are no synthetic examples, so synthetic miss rate, EER, binary accuracy, speaker authentication and fraud-detection performance are not estimable. Do not tune the threshold on these test observations and relabel the same set blind.
Commitments: acoustic manifest 8636b20bedd6d44b9d58f322cfecf272a59c8f9c2626f401d00becd522ce28fe; detector source ca6a48f4f408ec1f83e54546c036dfb8f84c47c0ca9336c7e20ca642cd8b2e51; detector score deposit e683068d6ccc134782a29f6dec14be1b8737509d646b00bf8dc91f9373f1876c. Methods and safe counts are under examples/acoustic-evaluation/; waveforms, references and hypotheses are excluded from publication. A later durable-root verifier amendment is an audit convenience, not a change to the original frozen inference protocol.
Mathematical reference and Echo qualification
An original standard-library educational reference implements one causal attention head, cross-entropy, Q/K/V derivatives, SGD/Adam and incremental KV caching using the established equations of Attention Is All You Need and Adam. Nine tests passed locally and on Feral Python 3.9.6. A 24-coordinate central-difference fixture has maximum gradient error 5.824e-12; full/step cache outputs match exactly. These are mechanism checks, not transformer training, novel architecture or proof of personal frontier research expertise. Learned projections, multihead blocks, positional encoding and full training are absent. Methods: examples/model-mechanics/README.md.
Echo's isolated recovery fixture executed nine actual CLI checks and eight selected source-test checks without model inference. It exercises private state persistence, advisory hold/expiry, task ownership and fail-closed recovery; scripted boundaries and logical local hosts do not represent arbitrary autonomous agents or separate physical hosts. Broker checks use owned loopback resources, not the user's broker. This qualification does not compare coding success or demonstrate coordination benefit. The separately qualified V1 original synthetic coding diagnostic is complete: eight tasks, two seeds and three conditions produced 48 retained attempts, with zero accepted pairs (single, plain two-worker and Echo each 0/16). All 138 model requests have known usage: 126,116 prompt tokens and 32,975 returned tokens (159,091 total). External paid inference cost was $0; hardware, electricity and labor were not measured. The frozen analysis binds source/model/VM identities and grades joint acceptance separately from individual features. The negative result exposes tool-contract failures, including missing write payloads; it establishes no coordination benefit, superiority or equivalence. A separately versioned V2 API/schema revision passed its own exact-source scripted qualification and completed another 48 local-model attempts on the same exposed fixtures. Single-context acceptance was 8/16; ordinary two-worker messaging and Echo each accepted 0/16. All 385 requests, 409,184 prompt tokens, 23,387 returned tokens and 13 merge conflicts are retained; usage was complete for every attempt. The matched Echo-versus-plain all-failure contrast establishes neither equivalence nor benefit. Changed prompts and validation prevent a fresh-holdout or isolated V1-to-V2 causal claim. Aggregate diagnostics and commitments. V1 failures remain unchanged and are not repaired, replaced or pooled with V2. Methods: Echo docs/research/recovery-fixture-and-coding-protocol.md.
Contribution, access and next evidence
The contribution here is artifact recovery and strict re-scoring, frozen bounded acoustic diagnostics, transparent failure reporting, local collection/isolation infrastructure, checked educational mathematics and coordination qualification. Existing models, datasets, libraries and classic algorithms are credited separately from original integration and audit work. AI-generated assistance is not an external reviewer or independent replication.
Repository/source access is currently private or gated. Local commands and deposited hashes describe how authorized holders can verify artifacts; they do not imply that readers already have a publicly runnable source release. Publicly shareable counts and commitments exclude raw audio, references, hypotheses and generated code. An outside reviewer needs permitted access, exact released revisions and their own execution deposit.
Next verifier evidence needs candidate pools with sufficient incorrect/correct diversity, applicable matched baselines and new task families; the completed cached-pool pilot does not answer the joint-verifier hypothesis. Acoustic preparation has acquired checksum-verified official ASVspoof 5 metadata and selected 192 records before scoring: 24 development and 24 evaluation speakers, balanced human/synthetic classes, disjoint speaker IDs and attack labels, and matched uncoded channels. No audio was acquired at the metadata/header-pilot stage. A subsequent separately frozen full-archive stream is underway and has retained selected recordings after their containing archives verify; the full 192-record acquisition and detector scoring remain incomplete. The database paper maps the selected development labels A09/A10 to ToucanTTS variants and evaluation labels A17/A18 to ZMM-TTS and its Malafide derivative; these are not four independent generators, and both base families use HiFi-GAN vocoding. Source-utterance linkage and spoof-content rights remain unresolved. A bounded official-API TAR-header pilot read 500 valid headers (256,000 bytes) in 261.542 seconds, matching none of the 192 selected recordings. It acquired only a partial index, no audio, and left the selection unchanged; range headers do not verify whole-archive integrity. The initial Echo V1 diagnostic remains zero of 48; V2 completed with single 8/16 and both two-worker conditions 0/16, without evidence of coordination benefit. Original checkpoint recovery remains distinct from an ablation reimplementation. Dated execution addenda must report failures, budget accounting and limitations before any stronger research claims.
ODSS two-class diagnostic: false flags and misses retained
Later on 6 October 2026 UTC, we completed a distinct bounded two-class diagnostic using ODSS, rather than relabelling the earlier all-human study. We froze 64 paired English recordings: 32 development records from four speakers (16 human/16 FastPitch-HiFiGAN), and 32 test records from four disjoint speakers (16 human/16 VITS). Source-utterance IDs are separated across splits. Synthetic/human counterparts share an utterance stem. Synthesis-training exposure is unknown; generator-directory separation does not prove independent vocoders. No exact source or prepared PCM duplicates were found.
The exact previously frozen JavaScript DSP detector ran locally at its unchanged 0.5 threshold on complete mono 16 kHz PCM16 recordings. All 64 produced determinate scores, with zero processing errors or abstentions. Development: 11/16 human false flags and 5/16 synthetic misses, balanced correct fraction 50%. Test: 10/16 human false flags (62.5%) and 4/16 synthetic misses (25%), balanced correct fraction 56.25%. Test class-correct fractions were 6/16 human and 12/16 synthetic. Failures remain in the report; no threshold was calibrated from these results.
A prespecified paired-speaker bootstrap (seed 20261006, 10,000 resamples) gives a descriptive test 95% percentile interval 40.625–78.125% for balanced correct fraction. Four test clusters make this interval fragile. This is an exploratory developer-run diagnostic, not independent detector validation, a superiority/equivalence test, calibrated confidence, authentication, streaming, radio or general-population evidence.
We verified 256 exact HTTP 206 ranges and 64 member receipts against the frozen selection; each compressed payload passed bounded deflate/CRC checks. Whole ZIP publisher MD5 was not independently verified because we did not download the whole 2.4 GB archive. A first acquisition attempt rejected ZIP data-descriptor mode before payload; a prospectively frozen V2 validated signed descriptors before fetching payloads. During earlier metadata inspection, an oversized LICENSE window overlapped a following WAV header and potentially a small compressed-payload prefix; no complete audio was decoded or retained from that window. The over-read remains a protocol deviation, with original response hashes unavailable. ASVspoof 5's original 192-record metadata selection was not altered and has no detector result.
ODSS's publisher and archive license declare both human and synthetic audio CC-BY-SA 4.0. The selected Hi-Fi TTS source is CC-BY 4.0. Attribution: Artem Yaroshchuk, Christoforos Papastergiopoulos, Luca Cuccovillo and collaborators, ODSS/WIFS 2023, DOI 10.1109/WIFS58808.2023.10374863; Hi-Fi TTS by Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg and Yang Zhang, 2021. Audio, per-record inputs and source-code access remain private; the safe aggregate provides count/hash evidence, not a public full-source reproduction claim. AI assistance contributed to acquisition, protocol and analysis tooling. No paid external inference or compute was used; electricity was unmeasured.
The frozen score deposit SHA256 is 77f6f655c7208b6d17bfc5cb38834d29385260bbd018e987a2cc0c3b95af44e6; metadata-only analysis SHA256 is 114dafb3c974fde1d02ef71c5ca20a564e06fc93adbe9e884234b98ab9595a88. A second agent replayed the exact pinned analysis and matched all aggregates and bootstrap intervals; this is an internal audit within the same workflow, not external review.
Download the aggregate-only ODSS diagnostic (JSON). Publication metadata records its SHA256; no audio, per-record scores or private source code are included.
Native V5.1 environment diagnostic: dependency failure retained
A separately frozen native run attempted the full pinned environment on an owned Ubuntu 22.04 guest with Python 3.10.12. The guest identity matched the frozen run metadata, and all 168 expected package versions matched exactly. This is a failed qualification, not a successful environment build.
Eleven of twelve ABI/import smoke checks passed. Numba 0.55.0 failed with ModuleNotFoundError for pkg_resources; the pinned setuptools version was 84.0.0. The sanitized diagnostic retained 1,617 stderr bytes without prefix truncation. The supervisor classified the enclosing full-environment attempt as FAILED_PARTIAL_FULL_ENVIRONMENT at full168-install-fixed-smoke with an outer ValueError; the nested ABI diagnostic identifies the specific missing module. The final reference report was absent, so all 24 registered reference tasks and 9 controls were retained as NOT_RUN_PREREQUISITE_FAILURE.
No model/task execution or H1 evidence resulted, and no dependency pin was changed. The owned guest and temporary owned files were removed and the domain UUID was independently confirmed absent. Receipt commitment: plan SHA256 5b4a36e11762f0c50f73707dfedf6eb2145a4e0732749676b87698d042cf07a5. The next gate is to resolve the pinned dependency incompatibility, rerun exact environment qualification, and pass every registered reference/control before dispatching any research task. This diagnostic is not evidence for or against the verifier hypothesis.
Native V7.2 prerequisite qualification: full environment passed
A prospectively frozen compatibility amendment changed one runtime dependency, setuptools 84.0.0 to 81.0.0, while preserving the other 167 artifact entries, all 71 original direct pins, five source builds and the registered checks. The bootstrap/build setuptools pin remained unchanged. The V5.1 failure above remains part of the record.
The completed owned, networkless Ubuntu guest matched all 168 expected package versions. All five source builds and twelve fixed ABI/import smoke checks completed successfully, including Numba. All 24 reference solutions passed active, nonempty suites without skips, errors or failures. The nine controls matched their expected outcomes: two positive checks and seven negative or incomplete-execution checks. All 33 registered slots were retained; reference subprocess cleanup and removal of the owned guest and temporary files were reported successful. No model calls occurred. Actual receipt SHA256: 88329e56a168273403569e1e898e4bae3e7bee8a0d28bbf71923f3391dc83039.
This qualifies the registered full-environment prerequisite, not the original H1 hypothesis or general runtime compatibility. The separate outer ownership qualification flag remained false. A second agent reconciled source commitments, package identities, smoke diagnostics, references, controls and runtime inventory within this same workflow; this is an internal audit, not outside reproduction.
The 45,730-entry runtime inventory covered 2,646,746,508 regular-file bytes. At that stage only the inventory was exported and the executable environment was removed during guest cleanup. The subsequent V5 custody milestone is reported below. These checks produced no verifier-study or frontier-model result. Paid external inference and rented-compute spend remained $0.
Download the completed native comparison aggregate and commitments.
Subsequent runtime byte custody: V5
A separately frozen V5 run repeated the registered full-environment qualification: all 168 package identities, five source builds, twelve ABI checks, 24 references and nine controls. After exact owned guest shutdown, the host retained 2,646,208,612 regular-file bytes and the full 45,730-entry runtime manifest in private artifacts. Payload, manifest and qualification-report SHA256 commitments are recorded in the linked aggregate. The owned guest and temporary source disks were removed; this establishes byte custody, not a qualified restored runtime.
The earlier V7.2 inventory-only stage remains above. The new runtime is not claimed byte-identical to that earlier removed environment. Scoped native interpreter, loader and compiler identity, safe exact-prefix restoration and repeated 168-version/12-ABI/namespace/24-reference/nine-control qualification remain required before candidate use. Existing native package identity/map preservation is distinct from complete resolved ownership: 1,850 metadata dispositions remain unresolved. The generic outer false flag is hardcoded, not a newly failed acceptance test. No candidate evaluation, H1 finding, portability or frontier-model result is established. Artifacts remain private; external paid inference/rental spend stayed $0 and electricity, hardware and labor were unmeasured.
Download runtime-custody aggregate and commitments.
Post-hoc FunctionGemma retained-output endpoint audit: 6 October 2026
A separate audit replayed all 800 retained outputs against the frozen native parser before evaluating two explicitly named alternative JSON endpoints. All original native labels reproduced: base 228/360 call and 0/40 no-call; checkpoint-600: 0/360 call and 40/40 no-call. Those original tables and interpretations of native invocation remain unchanged.
Strict JSON on the exact stored output, stripping surrounding whitespace only, yielded zero valid parses and zero correct cases in both arms. The second endpoint removes exactly one final literal <eos> only when the recorded stop token is 1 and the stored output ends in that literal. It does not remove interior tokens, duplicate terminators, <end_of_turn>, fences or commentary. On this separately labeled post-hoc endpoint, checkpoint-600 produced 250/360 valid call-case parses and 39/40 valid no-call parses, with 159/360 calls and 36/40 no-calls correct. Pooled correctness was 195/400 (48.75%); class-balanced correctness was 67.0833%. The base remained 0/400 correct on both JSON endpoints. Typed and legacy argument comparisons agreed on every audited label; all eight checkpoint call truncations remained incorrect in the 360-case denominator. Failures, unrun cases and stop-representation mismatches were zero.
| Retained-output endpoint | Base call / no-call correct | Checkpoint-600 call / no-call correct | Checkpoint pooled | Checkpoint class macro |
|---|---|---|---|---|
| Original native parser, unchanged | 228/360 · 0/40 | 0/360 · 40/40 | 40/400 (10%) | 50% |
| Exact stored strict JSON | 0/360 · 0/40 | 0/360 · 0/40 | 0/400 (0%) | 0% |
| One terminal-EOS removal, when stop ID 1 | 0/360 · 0/40 | 159/360 · 36/40 | 195/400 (48.75%) | 67.0833% |
This audit qualifies a blanket interpretation of native invocation collapse: the checkpoint expressed some correct function-call content under a different post-hoc JSON endpoint. It does not revise the native endpoint, show causal model improvement, establish full-schema equivalence or diagnose why the mismatch occurred. Typed equality distinguishes booleans from numbers; finite integers and floats compare numerically, with the original permissive comparator reported separately. Empty-list no-call correctness is not semantic refusal quality. Historical exposure, lossy schema and checkpoint-provenance limits remain. No new generation, retries or paid inference were used; same-owner/AI-assisted checks are not external peer review. A separate 32-fixture diagnostic is not included in these 800 results.
Aggregate endpoint audit and source commitments. The source-bound audit first reproduced all 800 native outcomes. A separate internal metadata reconciliation verified source commitments, 1604 ordered stage events and rate arithmetic; it did not reparse raw predictions or datasets. Neither step is external peer review. Public counts/hash commitments exclude raw outputs, references, item IDs and private paths; they do not enable public item-level correctness rescoring.
Authored structural tool-use diagnostic: matched 160-case follow-up (6 October 2026)
After the historical findings, I authored 160 deliberately simple structural fixtures: 80 single calls, 40 parallel calls and 40 no-call cases. Both the reconstructed names-ablation and publisher Qwen3-1.7B received full argument schemas and the same JSON-output instruction. This changes the information condition and task source; it does not replace any historical comparison, native FunctionGemma table or endpoint audit.
| Retained result | Reconstructed names-ablation | Publisher Qwen3-1.7B |
|---|---|---|
| Single-call correctness | 80/80 | 64/80 |
| Parallel-call correctness | 40/40 | 28/40 |
| No-call correctness | 40/40 | 40/40 |
| Original permissively parsed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Typed parsed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Strict JSON contract AND typed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Call/no-call class macro | 100% | 88.3333% |
| Three-family macro | 100% | 83.3333% |
| Strict output-contract validity | 160/160 | 137/160 |
| Completed / known usage | 160/160 | 160/160 |
| Input / returned tokens | 59,192 / 5,369 | 59,192 / 5,525 |
All 320 requests completed; neither arm had input or output truncation. Paired correctness retained 132 both-correct cases, 28 reconstructed-only cases, zero publisher-only and zero both-incorrect cases. The original permissively parsed endpoint, a typed parsed endpoint and the separate strict JSON-contract conjunct all retained the same correctness counts. Strict output-contract validity alone is a separate measure: publisher has five contract-valid but incorrect cases. These are descriptive counts on this finite fixture set, not population estimates. Zero external API spend does not mean compute has no cost.
The original selected SFT checkpoint is still unidentified. The reconstructed names-ablation is a distinct recovered artifact with unresolved training lineage. Fixtures were authored after earlier findings; unknown pretraining exposure, catalogue priors, toy structure and author selection prevent causal training-benefit, broad generalization or frontier-benchmark claims. This is not an independently blind study or outside peer review. Metadata/source/receipt arithmetic and opaque-output hashes were replayed. A separate retained-output reclassification then reconciled all 320 original labels, with zero disagreements and no generation calls or tool execution. Typed equality and strict JSON-contract-conjoined correctness agree on these fixtures; this does not prove the endpoints agree generally. EOS and token counts remain checked against recorded worker metadata, not independently tokenized. No-call correctness here concerns the fixture contract, not a general semantic refusal evaluation.
Aggregate counts and source commitments. Aggregates cannot let outsiders rescore individual predictions; permitted data, predictions, model lineage and environment access remain separate reproducibility requirements.
ASVspoof 5 acquisition: archive integrity before scoring
At 22:42:10 UTC on 6 October 2026, the ongoing source-bound stream had verified six of thirteen complete publisher archives (45,422,274,560 bytes) and committed 133 of the unchanged 192 selected recordings. A separate internal audit rehashed all 133 committed recordings (18,490,964 bytes) against their member receipts and linked those receipts to the completed containing-archive hashes. The seventh archive was still streaming; neither acquisition nor supervisor had a terminal receipt. This is a dated progress observation, not full acquisition qualification or a detector result.
The complete registered acquisition must finish and pass its custody checks before scoring. Speaker separation does not resolve unknown source-utterance linkage, generator-training exposure or shared-vocoder dependence. Publisher database terms are documented; separate synthetic-content reuse scope remains unresolved. This private research acquisition makes no waveform redistribution or unrestricted commercial-use claim. The earlier metadata/header-only stage remains a historical record.
The internal progress audit commitment is SHA256 068dfe7fc34e15d0c55d79f5555e9f9a17beac59a73f22629aec7c6cf791f240. No audio was decoded or scored by that audit, no raw recordings or identifiers are published, and paid inference/rented-compute spend remains $0.
Subsequent scoped runtime restoration replay · 6 October 2026 Pacific
A fresh owned, networkless VM restored the retained environment. Root reconciliation checked eight registered native files against their frozen bytes, all 168 package versions, 12 ABI checks, 24 references, nine registered controls and the unchanged 45,730-entry tree containing 2,646,208,612 regular-file bytes. The disposable VM was removed. Five historical source-build proofs were reused; their build backends were not reexecuted.
These scoped checks supersede the earlier pending restoration/ABI/reference descriptions for this replay. They do not establish exhaustive ELF dependency closure, model/tokenizer/template identity, collector admission, outside reproduction or the joint-verifier hypothesis. No model calls were made; external paid spend remained $0, and local electricity, hardware and labor were unmeasured.
Download the scoped restoration aggregate.
ASVspoof 5 two-class diagnostic completed · 6 October 2026 Pacific
The original 192-record selection is now fully acquired and scored. This supersedes the earlier dated acquisition-progress status above; it does not change the earlier ODSS results. All thirteen full publisher archives streamed through the pinned collector (104,817,612,800 network bytes). Root reconciliation independently hashed all 192 selected members, verified the pending/verified/committed journal and confirmed agreement with the earlier 96 retained member hashes. Publisher archive MD5 assurance is bound to the pinned streamer; discarded complete archives were not independently reconstructed.
All 192 records passed decoder qualification and a second PCM-commitment replay. The prospectively selected condition retains native sample rate and frame count, selects original channel zero into mono float32 WAV, and applies no resampling or normalization. All 192 transformations passed sample-level replay against the original first-channel float32 conversion. The historical JavaScript DSP snapshot ca6a48f4f408 ran at its unchanged threshold 0.50. Its score deposit was committed before the late-label join; all records produced determinate results, with no processing errors or abstentions.
| Partition | Registered | Human false flags | Synthetic misses | Correct / registered |
|---|---|---|---|---|
| Development | 96 (48 human, 48 synthetic) | 33/48 | 7/48 | 56/96 (58.33%) |
| Evaluation | 96 (48 human, 48 synthetic) | 22/48 | 8/48 | 66/96 (68.75%) |
| Combined descriptive total | 192 (96 human, 96 synthetic) | 55/96 | 15/96 | 122/192 (63.54%) |
The evaluation human false-flag fraction is 45.83%, while the synthetic miss fraction is 16.67%. These substantial errors limit practical usefulness. No threshold was tuned on these outcomes, and no confirmatory superiority test or population confidence interval is claimed. A separate model-assisted internal audit replayed the aggregate and the ordinal → WAV → original FLAC → frozen-label chain. This is workflow verification, not external human peer review or independent blinded custody.
This is a developer-selected, speaker-disjoint internal diagnostic covering two base TTS families, a related adversarial derivative and a shared vocoder. Unknown source-utterance/channel linkage and synthesis-training exposure limit interpretation. It does not establish four-family generalization, calibrated probabilities, speaker authentication, streaming or radio performance, or validation of the current deployed detector. Publisher database terms are documented; separate synthetic-content reuse scope remains unresolved. Audio, source mappings and per-record scores remain private and are not redistributed. External inference/rental spending was $0; local electricity was unmeasured.
Source: ASVspoof 5 publisher deposit. Download the aggregate-only report. The frozen score deposit SHA256 is dda43bb0d6e680f358d9e7748130dc426a3739f0e916188f263cdc343616686f.
Download the canonical note (Markdown) · Source checksum and publication metadata