# Public methods appendix: 6 October 2026

## What the downloadable aggregates reproduce

The tool-caller JSON supports arithmetic checks of published class counts, paired contingency tables and registered descriptive differences. For each arm, equal-weight macro is `(call_correct / 360 + no_call_correct / 40) / 2`; pooled correct fraction is `(call_correct + no_call_correct) / 400`. The macro difference is 35 percentage points and pooled difference 45 percentage points. Both summaries were registered; neither is an exclusive prospective primary endpoint. Paired cells retain all 400 matched slots. Strict-format validity and permissive parser validity are distinct from argument/value correctness and are reported separately.

The downloadable `check-tool-caller-comparison.py` checks those identities against the public aggregate. Passing this check does not reproduce correctness labels, generations, item selection, exposure detection or parser behavior. SHA256 commitments identify a fixed source; a hash alone supplies neither source access nor permission.

The frozen renderer compares calls in the same order with the same function names. Argument dictionaries are normalized by key order; integer and floating-point values are converted to floats, while list order and string case are preserved. Argument equality is permissive, not type-strict: Python equality can treat boolean `True` as equal to numeric `1` after normalization (and `False` as equal to `0`). Missing or other false-valued argument containers are also mapped to an empty dictionary by the comparator. Strict JSON shape is measured separately; it is not an additional requirement of the permissive correctness endpoint. Generated function calls are never executed. These are existing frozen scoring semantics, not a revised scorer or a claim of execution success.


Both arms used names-only catalogues, greedy generation, 384 maximum new tokens, 4096 maximum input tokens, bfloat16, four intra-op threads and one inter-op thread, with pinned Torch/Transformers versions recorded in the aggregate. Names-only catalogues omit signatures, descriptions and required-argument schemas. Catalogue-trained weights may retain information omitted from publisher prompts. For the original published names-only comparison, no full-schema/native-tools/reasoning-enabled/optimized publisher control was completed. A separately versioned full-schema + truthful-instruction control subsequently completed all 400 slots; its methods/results addendum follows. Concurrent CPU/cache/storage conditions were uncontrolled, so timing cannot establish a speed advantage. External spend is $0; electricity and total economic cost were not measured.

The verifier-selection pilot aggregate describes cached replay, not three independently regenerated model pools. Its original 49-task subset remains fixed; reference qualification and hidden correctness coverage are separate. Aggregate totals cannot reconstruct task-cluster bootstrap intervals or individual policy decisions. Public intervals are conditional on the collected pool and multiple exploratory contrasts are unadjusted. Full permitted task-level decisions and separately supplied grading outcomes would be necessary for exact interval replay; they are excluded from this aggregate-only release.

For genuine reproduction, an approved access bundle must provide the frozen protocol and permitted source versions, model/tokenizer/configuration commitments, data-rights boundaries, exact rendering/scoring/selection contracts, environment, qualified isolated grading runner, failure accounting and authorized decisions/outcomes. Original selected SFT 1.7B remains missing; reconstructed names-ablation is a different artifact. Earlier FunctionGemma finite-forward/load qualification did not establish task accuracy; its subsequently completed lossy-native comparison is reported separately with invocation collapse and unknown cause. H1 five-source builds, installed namespace/dependency compatibility and all 24 reference execution remain pending. Internal model/agent assistance is not outside scientific review.

## Completed follow-up, original scoring preserved

A subsequent separately frozen full-schema publisher collection completed all 400 slots. Full schemas and the truthful system instruction both changed from the names-only condition; this is a descriptive information-condition comparison, not an isolated schema-only intervention.

| Condition | Call correct | No-call correct | Equal-weight macro | Pooled correct | Incorrect |
| --- | ---: | ---: | ---: | ---: | ---: |
| Recovered names-ablation | 223/360 | 31/40 | 69.7222% | 254/400 (63.5%) | 146 |
| Publisher names-only | 52/360 | 22/40 | 34.7222% | 74/400 (18.5%) | 326 |
| Publisher full-schema + instruction | 235/360 | 27/40 | 66.3889% | 262/400 (65.5%) | 138 |

The same publisher weights changed from 74/400 to 262/400 pooled correct (+47 percentage points); equal-weight class macro changed from 34.7222% to 66.3889% (+31.6667 points). Both registered descriptive summaries are retained. Full-schema publisher still made 138/400 errors (125 call errors and 13 no-call errors). Its 262 pooled correct versus reconstructed names-ablation's 254 does not imply a better overall model: their class outcomes, information conditions and model priors differ.

All 400 full-schema slots completed with known usage, no observed input/output truncation, 346 strict-format and 370 permissive-parser-valid responses. Input tokens were 171,638 versus 54,968 for names-only; returned tokens were 21,377 versus 18,456. These are token accounting, not energy/latency measurements. Full schemas and instruction changed together; reasoning-enabled, optimized and native-tools publisher controls were not run.

The separately frozen, post-collection type-stricter argument comparator reconciled all original labels across the three arms and changed **0/1,200 correctness labels**. It distinguishes boolean from numeric values, disallows false-valued argument-container coercion and avoids lossy large-integer normalization. This is argument-correctness sensitivity, not strict JSON-format scoring; the original published scorer and results remain unchanged. Zero changes weaken those specific scoring concerns for this measured cohort, not for all future outputs. Calls were not executed.

Five exact training-content matches and eight prior probe exposures remain in the historical prefix. This does not establish causal training benefit, fresh/general tool-calling superiority, novelty, frontier transfer or production readiness. The separately reported FunctionGemma lossy-native invocation-collapse diagnostic remains unchanged; its cause is unknown from aggregate review. External paid model/compute spend was $0; electricity, hardware/labor and latency economics were not measured. Developer/AI-assisted internal checks are not outside review.

[Download the aggregate information-condition and sensitivity results](https://tuneharness.com/research/2026-10-06/tool-caller-information-sensitivity-20261006.json). It contains class/format/usage/paired counts and source commitments, without outputs, per-item IDs or private paths.

## Post-hoc FunctionGemma retained-output endpoint audit: 6 October 2026

A separate audit replayed all 800 retained outputs against the frozen native parser before evaluating two explicitly named alternative JSON endpoints. All original native labels reproduced: base 228/360 call and 0/40 no-call; checkpoint-600: 0/360 call and 40/40 no-call. Those original tables and interpretations of native invocation remain unchanged.

Strict JSON on the exact stored output, stripping surrounding whitespace only, yielded zero valid parses and zero correct cases in both arms. The second endpoint removes exactly one final literal `<eos>` only when the recorded stop token is 1 and the stored output ends in that literal. It does not remove interior tokens, duplicate terminators, `<end_of_turn>`, fences or commentary. On this separately labeled post-hoc endpoint, checkpoint-600 produced 250/360 valid call-case parses and 39/40 valid no-call parses, with 159/360 calls and 36/40 no-calls correct. Pooled correctness was 195/400 (48.75%); class-balanced correctness was 67.0833%. The base remained 0/400 correct on both JSON endpoints. Typed and legacy argument comparisons agreed on every audited label; all eight checkpoint call truncations remained incorrect in the 360-case denominator. Failures, unrun cases and stop-representation mismatches were zero.

| Retained-output endpoint | Base call / no-call correct | Checkpoint-600 call / no-call correct | Checkpoint pooled | Checkpoint class macro |
| --- | --- | --- | --- | --- |
| Original native parser, unchanged | 228/360 · 0/40 | 0/360 · 40/40 | 40/400 (10%) | 50% |
| Exact stored strict JSON | 0/360 · 0/40 | 0/360 · 0/40 | 0/400 (0%) | 0% |
| One terminal-EOS removal, when stop ID 1 | 0/360 · 0/40 | 159/360 · 36/40 | 195/400 (48.75%) | 67.0833% |

This audit qualifies a blanket interpretation of native invocation collapse: the checkpoint expressed some correct function-call content under a different post-hoc JSON endpoint. It does not revise the native endpoint, show causal model improvement, establish full-schema equivalence or diagnose why the mismatch occurred. Typed equality distinguishes booleans from numbers; finite integers and floats compare numerically, with the original permissive comparator reported separately. Empty-list no-call correctness is not semantic refusal quality. Historical exposure, lossy schema and checkpoint-provenance limits remain. No new generation, retries or paid inference were used; same-owner/AI-assisted checks are not external peer review. A separate 32-fixture diagnostic is not included in these 800 results.

[Aggregate endpoint audit and source commitments](https://tuneharness.com/research/2026-10-06/functiongemma-retained-output-endpoint-audit-20261006.json). The source-bound audit first reproduced all 800 native outcomes. A separate internal metadata reconciliation verified source commitments, 1604 ordered stage events and rate arithmetic; it did not reparse raw predictions or datasets. Neither step is external peer review. Public counts/hash commitments exclude raw outputs, references, item IDs and private paths; they do not enable public item-level correctness rescoring.


## Authored structural tool-use diagnostic: matched 160-case follow-up (6 October 2026)

After the historical findings, I authored 160 deliberately simple structural fixtures: 80 single calls, 40 parallel calls and 40 no-call cases. Both the reconstructed names-ablation and publisher Qwen3-1.7B received full argument schemas and the same JSON-output instruction. This changes the information condition and task source; it does not replace any historical comparison, native FunctionGemma table or endpoint audit.

| Retained result | Reconstructed names-ablation | Publisher Qwen3-1.7B |
| --- | ---: | ---: |
| Single-call correctness | 80/80 | 64/80 |
| Parallel-call correctness | 40/40 | 28/40 |
| No-call correctness | 40/40 | 40/40 |
| Original permissively parsed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Typed parsed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Strict JSON contract AND typed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Call/no-call class macro | 100% | 88.3333% |
| Three-family macro | 100% | 83.3333% |
| Strict output-contract validity | 160/160 | 137/160 |
| Completed / known usage | 160/160 | 160/160 |
| Input / returned tokens | 59,192 / 5,369 | 59,192 / 5,525 |

All 320 requests completed; neither arm had input or output truncation. Paired correctness retained 132 both-correct cases, 28 reconstructed-only cases, zero publisher-only and zero both-incorrect cases. The original permissively parsed endpoint, a typed parsed endpoint and the separate strict JSON-contract conjunct all retained the same correctness counts. Strict output-contract validity alone is a separate measure: publisher has five contract-valid but incorrect cases. These are descriptive counts on this finite fixture set, not population estimates. Zero external API spend does not mean compute has no cost.

The original selected SFT checkpoint is still unidentified. The reconstructed names-ablation is a distinct recovered artifact with unresolved training lineage. Fixtures were authored after earlier findings; unknown pretraining exposure, catalogue priors, toy structure and author selection prevent causal training-benefit, broad generalization or frontier-benchmark claims. This is not an independently blind study or outside peer review. Metadata/source/receipt arithmetic and opaque-output hashes were replayed. A separate retained-output reclassification then reconciled all 320 original labels, with zero disagreements and no generation calls or tool execution. Typed equality and strict JSON-contract-conjoined correctness agree on these fixtures; this does not prove the endpoints agree generally. EOS and token counts remain checked against recorded worker metadata, not independently tokenized. No-call correctness here concerns the fixture contract, not a general semantic refusal evaluation.

[Aggregate counts and source commitments](https://tuneharness.com/research/2026-10-06/authored-tooluse-matched160-20261006.json). Aggregates cannot let outsiders rescore individual predictions; permitted data, predictions, model lineage and environment access remain separate reproducibility requirements.
