# Schema information and scoring sensitivity in recovered tool-calling checkpoints

*A methods case study; full-schema information-condition and argument-sensitivity results are completed; fresh structural outcomes remain pending. Native schema inventory is measured.*

**Cisco Cáceres** · 6 October 2026 · Draft for methods review

## Abstract

We report a developer-run comparison between a recovered names-ablation model and publisher Qwen3-1.7B post-trained weights under a reconstructed names-only tool-calling interface. All 400 registered slots per arm are retained: 360 require calls and 40 require no call. The recovered arm achieves 61.9444% call correctness and 77.5% no-call correctness; the publisher arm achieves 14.4444% and 55%. Equal-weight class macro scores are 69.7222% and 34.7222%, while pooled correct fractions are 63.5% and 18.5%. Both summaries were registered descriptive outputs. The recovered arm makes 146 errors. These results are specific to an exposed historical prefix, permissive scoring and an interface that omits parameter signatures, descriptions and required-argument schemas. Catalogue-trained weights may retain information unavailable in publisher prompts. The experiment therefore does not identify a causal training benefit or establish general tool-calling superiority. We present the measurements, scoring caveats, reconstruction lineage and boundaries of aggregate reproducibility. A subsequent full-schema information-condition control and argument-sensitivity analysis are completed below; fresh structural outcomes remain unfinished. Separately, an actual tokenizer-only native-schema inventory measures representational differences on 400 historical records; it does not establish native-interface equivalence or task accuracy.

## 1. Research question and contribution

The question is how tool-call measurements depend on the information supplied to a model and the contract used to score its response. A model comparison requires more than equal prompt strings: training history can supply information absent from those strings, and permissive parsing can accept outputs that violate a strict response contract.

This study contributes an engineering methods case study: explicit reconstruction provenance, retained failure accounting, paired aggregate measurements and disclosure of information and scoring asymmetries. It introduces neither a model architecture nor a new scoring algorithm. The established models, software and statistical summaries are upstream contributions. No novelty, causal effect, population-level superiority, frontier transfer or production-readiness result is claimed.

## 2. Model lineage and evaluation scope

The experiment belongs to the project now called TuneHarness, under its earlier internal name. A recovered names-ablation artifact is distinct from the originally selected SFT 1.7B checkpoint, which remains unidentified. Recovery of an ablation is not recovery of that original selection.

The publisher comparator uses Qwen3-1.7B post-trained weights, not Qwen3-1.7B-Base. Its metadata and generation contract were reconstructed for this evaluation. It is consequently a publisher-weight comparator under reconstructed packaging, rather than an untouched native deployment. Exact source and execution commitments are published with the aggregate [1].

The evaluated cohort is an exposed historical first-400 prefix: 360 call cases and 40 no-call cases. Five exact training-content matches and eight prior probe exposures remain included. They are not removed to improve the result. The comparison protocol was frozen before publisher generation, but after the ablation run began and partial outputs were exposed. It is not blind preregistration. This cohort does not support a held-out generalization claim.

## 3. Information and generation conditions

Both arms receive a names-only catalogue. It omits tool signatures, descriptions and required-argument schemas. Catalogue-trained weights may retain precisely the information absent from the publisher prompt. Identical prompt text therefore does not establish equal prior information.

Both arms use greedy generation, bfloat16 CPU execution, four intra-op threads, one inter-op thread, a 4,096-token input cap and a 384-token output cap. The pinned environment reports Torch 2.14.0+cpu and Transformers 5.16.1. Each registered arm retains 400 slots, including failures; retries are zero. There is no observed input or output truncation in the completed aggregate.

No full-schema, native tools-slot, reasoning-enabled or optimized publisher control was completed as part of the original published names-only comparison. A separately versioned full-schema control subsequently completed; its results are reported in the addendum below. Concurrent CPU, cache and storage conditions were uncontrolled; execution durations cannot establish a speed advantage. External expenditure is $0, while electricity, hardware and labor costs are unmeasured.

## 4. Scoring contract

Published correctness uses a permissive parser and argument/value comparison. Strict response-format validity, parser validity and correctness are distinct outcomes. Generated tool calls are never executed.

The published methods appendix describes ordered call and name matching, dictionary-key normalization and argument/value comparison [2]. The original comparator accepts some Python equality semantics: integer and floating-point values can compare equal, booleans can compare equal to numbers, and false-valued argument containers can be coerced by the scoring path. These behaviors can affect correctness labels. They are part of the historical contract, not proof of semantic equivalence or valid executable tool arguments.

The original scorer and labels remain unchanged. A separately frozen post-collection argument-sensitivity analysis subsequently completed, reported below; the original labels remain unchanged. Public counts do not reveal which cases depend on these edge cases.

For class correct counts `c_call` and `c_no_call`, the two registered descriptive outputs are:

- Equal-weight macro: `(c_call / 360 + c_no_call / 40) / 2`.
- Pooled correct fraction: `(c_call + c_no_call) / 400`.

Neither is an exclusive prospectively ranked primary endpoint. Paired contingency counts compare corresponding retained slots; no p-values or population confidence intervals are reported.

## 5. Completed results

| Measure | Recovered names-ablation | Publisher comparator |
| --- | ---: | ---: |
| Registered and completed slots | 400 | 400 |
| Call correct / total | 223 / 360 | 52 / 360 |
| Call correctness | 61.9444% | 14.4444% |
| No-call correct / total | 31 / 40 | 22 / 40 |
| No-call correctness | 77.5% | 55% |
| Equal-weight class macro | 69.7222% | 34.7222% |
| Pooled correct / total | 254 / 400 | 74 / 400 |
| Pooled correct fraction | 63.5% | 18.5% |
| Strict-contract valid | 400 / 400 | 321 / 400 |
| Permissive-parser valid | 400 / 400 | 362 / 400 |
| Known input tokens | 54,968 | 54,968 |
| Known returned tokens | 18,976 | 18,456 |

The descriptive macro difference is 35 percentage points; the pooled difference is 45 percentage points. The recovered arm's **146 errors** are its own incorrect 400-slot outcomes, not all errors across the study or a measure of general reliability. The publisher arm has 326 incorrect outcomes under the same historical scoring contract. No-call correctness complements the false-positive call rate on that no-call class; it is not an overall false-positive rate across all 360 call/40 no-call cases.

| Paired outcome across all 400 slots | Count |
| --- | ---: |
| Both correct | 71 |
| Recovered only correct | 183 |
| Publisher only correct | 3 |
| Neither correct | 143 |

These cells sum to 400 and reproduce the pooled margins 254 and 74. They describe the retained sample; they do not remove exposure, reconstruction or schema-information confounding. All 800 arm slots have known usage and completed status in the published aggregate [1].

## 6. Hypotheses and possible falsification

The completed comparison motivates hypotheses; it does not establish them.

**Information hypothesis:** Providing missing normalized schemas may change publisher correctness. The subsequently completed full-schema control evaluates that changed information condition on the same exposed 400. Little or no improvement would weaken the claim that prompt schema omission explains the observed gap; improvement would not establish a causal adaptation benefit, because catalogue knowledge and reconstruction remain different.

**Scoring hypothesis:** Some apparent successes may depend on permissive parsing or equality/coercion behavior. The separately frozen sensitivity analysis first reconciled original labels, then retained every slot while evaluating declared alternative contracts. The observed zero changes across 1,200 correctness labels weaken these specific argument-scoring concerns for the measured cohort; they do not establish equivalence on future outputs.

**Structural hypothesis:** A newly authored fixture can separately test directly specified single-call, ordered parallel-call and no-call structure. The frozen 160-case fixture contains 80 single-call, 40 parallel-call and 40 no-call cases. Tokenizer fit alone establishes neither correctness nor generalization. Generation and outcome analysis remain pending. These hypotheses arise after the historical findings; the fixture does not constitute independent confirmatory evidence for a hypothesis selected after those findings. Failure on clearly specified cases would falsify a strong structural-reliability claim for that condition; success would remain limited to the authored diagnostic.

## 7. Measured native-schema inventory and tokenizer fit

FunctionGemma finite-forward/load and synthetic native-interface qualification are separate from task accuracy. An actual tokenizer/template-only audit retained all 400 historical records (360 call / 40 no-call), auditing 1,107 tool representations. Native declarations parsed on 381 records; 19 records were unsupported. Field-comparison coverage is therefore 381 records, and the field inventory is incomplete. No queries or reference-answer fields were used or emitted: synthetic user text substituted for task content. No model was loaded.

| Audited representation event | Count |
| --- | ---: |
| Nonempty default-field omission | 1,017 |
| Empty default-field omission | 344 |
| Unidentified top-level tool-field omission | 121 |
| Format-field omission | 4 |
| Property-name omission | 14 |
| Minimum-field omission | 9 |
| Maximum-field omission | 8 |
| Pattern-field omission | 4 |
| Empty properties-field omission | 1 |
| Empty required-field omission | 1 |
| Added description-field event | 3 |
| Unsupported declaration-parse records | 19 |

These are aggregate field/representation events, not mutually exclusive record counts, population frequencies or independent observations. Parsed-subset omissions must not be extrapolated to the 19 unsupported records. The unidentified tool-field event is not evidence of any particular field's content. Zero omissions for another field would not prove equivalent full-schema information.

A distinct actual native tokenizer-fit step completed in 4.0130 seconds: all 400 inputs fit the 4,096-token cap, maximum 1,081, zero oversize inputs and zero model loads. Its aggregate representations include 986 flat parameter maps and 121 explicit object schemas. Input fit establishes neither schema fidelity nor generation correctness. Earlier synthetic review also identified nullable-union stringification and omitted additionalProperties/minItems/maxItems constraints. These synthetic findings remain distinct from the historical parsed-subset event counts.

The audit explicitly rejects a full-schema equivalence claim. Any native-interface augmentation requires a separately registered condition and interface qualification, with representational differences reported. A distinct representable-information condition may instead be studied prospectively. Unsupported fields or failed records must not be silently removed. No FunctionGemma task accuracy is reported here.

Schema-audit result commitment: SHA256 `7b589d1b625bfce6f9f1fc32f42378d069c8213e7c974a0f2239569ae0a7e4b1`. Separate tokenizer-fit result commitment: SHA256 `9329bf9cbd6fbdcc67e0990dda9eb8da0c819290f94485357b943933414e93b3`. The aggregate-only [native fit and schema inventory download](./functiongemma-native-fit-schema-20261006.json) exposes these counts and source commitments, not task-level source access or outside reproduction.

## 8. Reproducibility and access

Public aggregates support arithmetic checks. They cannot reproduce item selection, training-content overlap detection, correctness labels, model recovery or raw generations. SHA256 commitments identify fixed artifacts; they grant neither access nor redistribution permission.

| Evidence | Public location or commitment | What it supports |
| --- | --- | --- |
| Completed aggregate | [Tool-caller names comparison JSON](https://tuneharness.com/research/2026-10-06/tool-caller-names-comparison-20261006.json) | Class, paired, format and token totals |
| Scoring and methods disclosure | [Published methods appendix](https://tuneharness.com/research/2026-10-06/tool-caller-methods-appendix-20261006.md) | Historical comparator semantics and measurement limits |
| Arithmetic checker | [Standard-library checker](https://tuneharness.com/research/2026-10-06/check-tool-caller-comparison.py) | Aggregate arithmetic only; no independent correctness rescoring |
| Dated research note | [TuneHarness 6 October research note](https://tuneharness.com/research/2026-10-06/) | Published experiment context and unfinished work |
| Completed source report | SHA256 `8415cad472e44d3bbfdd7358ccb6881a769354c0af36c1fad26575cf7d40c28e` | Identity of the committed analysis, not source access |
| Frozen comparison protocol | SHA256 `713aec34daf88a4a9b154bd0ef08ac7260c4dfcdee56cf5ef5459da8b6d1111e` | Fixed protocol commitment; timing limitations remain |
| Analyzer | SHA256 `9b074aec4b9da0b997c55fb805e1ebd560a6b786d13bf1bdc0aaad37a00cf2e3` | Fixed analysis implementation commitment |

An independently runnable reproduction bundle would require permitted protocol/scorer/renderer source, model/tokenizer/configuration identities and access, data rights, environment locks, exact selection manifests, authorized predictions/outcomes and failure accounting. Such a released bundle is not established by the aggregate downloads. Raw prompts, predictions, task/item identifiers, hidden tests, weights and private paths are excluded from this draft. Dataset, model and integration-code permissions must be assessed separately; a neighboring repository's license cannot settle them.

## 9. Limitations and interpretation

This is same-owner, developer-run evidence with internal AI/agent assistance, not outside peer review or independent replication. Historical exposure, training-content matches, nonblind protocol timing, reconstructed packaging, omitted schema information and permissive comparator semantics limit interpretation. The originally selected checkpoint is missing. More informative prompts, native interfaces and reasoning/optimized serving are different conditions. Controlled latency, energy and production reliability are unmeasured.

The strongest justified statement is that, under this reconstructed names-only contract and historical cohort, the recovered arm has higher descriptive scores while still making 146 errors. A stronger statement about learning, general tool-calling skill or deployment reliability requires additional controls and independently reproducible evidence.

## 10. Publication and next-study gates

Before extending this case study with new results, complete the running full-schema collection, reconcile every registered slot and execute its frozen analysis. Complete prospectively specified scoring sensitivity before claiming robustness to scoring semantics. Collect and reconcile all 160 authored diagnostic outcomes without filtering failures. Qualify native schema fidelity before claiming matched-information comparisons. Review executable methods and artifact rights before releasing a reproduction bundle; document outside review only after it actually occurs.

The native schema inventory and subsequently appended native task comparison are measured, with information loss and invocation collapse retained; equivalent full-schema information remains unqualified. The separate H1 and CooperBench studies, source builds, native runtime/reference qualification and further acoustic experiments remain unfinished. Their preparation does not strengthen this paper's completed results. No venue acceptance, peer-reviewed status or reviewer endorsement is claimed.

## Primary-source context

The Qwen3 technical report describes thinking and non-thinking modes in the same model family. This study fixes non-thinking mode; it does not measure the reasoning-enabled condition or reproduce the report's benchmark results. [Qwen3 technical report](https://arxiv.org/abs/2505.09388v1).

BFCL evaluates serial and parallel tool calls, abstention and stateful agentic behavior across broader tasks. Our exposed historical prefix and authored structural fixture do not constitute a BFCL evaluation, leaderboard submission or comparable general capability score. The comparison here concerns this study's specific information and scoring contracts. [Patil et al., ICML 2025](https://proceedings.mlr.press/v267/patil25a.html).

Google's FunctionGemma documentation specifies native declaration/call/result markers, a developer-role tooling trigger and a function-response stop sequence. It describes single-turn and independent parallel workflows as trained capabilities. The local audit measures the pinned template's representation of our schemas; documentation alone does not establish preservation of every JSON Schema constraint. [FunctionGemma formatting and best practices](https://ai.google.dev/gemma/docs/functiongemma/formatting-and-best-practices).

These sources establish relevant model and evaluation context. They do not validate this study's measurements or establish research priority. A comprehensive related-work and independent reproduction review remains pending.

## Targeted related work

Function-call evaluation predates this case study. BFCL covers serial and parallel calls, abstention and stateful behavior; its V4 format-sensitivity study varies documentation syntax, return syntax, tags and prompt presentation. These precedents motivate separating interface presentation from correctness. Our omission of parameter schemas changes supplied information, whereas format transformations can preserve that information. Neither our exposed historical prefix nor our native-template inventory is a new comprehensive tool benchmark. [BFCL paper](https://proceedings.mlr.press/v267/patil25a.html), [BFCL format sensitivity, July 2025](https://gorilla.cs.berkeley.edu/blogs/17_bfcl_v4_prompt_variation.html).

The comparison of learned function surfaces with supplied schemas also has close prior work. Jafarzadeh Asl et al. describe matched fine-tuning of functional-token and schema-in-prompt representations, including held-out functions and out-of-scope refusal. IFEval-FC evaluates verifiable instructions carried in parameter descriptions. The vehicle study's full methods were inspected: its matched representation-training grid uses three training seeds, with unseen functions withheld as target outputs although their schemas may occur as distractors. Appendix C.6 also compares uniform and native FunctionGemma interfaces and discusses refusal trade-offs. Its “Execution” metric denotes a parsed valid call rather than vehicle actuation. Our recovered names-ablation is not its dedicated functional-token condition, and our exposed cohort does not replicate its matched held-out grid. The IFEval-FC review remains abstract and metadata based. No upstream code, dataset, checkpoints or reported results have been independently reproduced. [Vehicle function-surface study, version 2, 4 October 2026](https://arxiv.org/abs/2609.09476v2), [IFEval-FC, version 1, September 2025](https://arxiv.org/abs/2509.18420v1).

API-Bank supplies a runnable tool-augmented evaluation system. Our generated calls are never executed, so argument matching cannot establish successful API interaction. The historical comparison also uses Python equality and truth-value behavior: booleans can compare equal to integers, and empty containers are false-valued. JSON Schema boolean validation distinguishes booleans from numeric values. These semantic sources explain the scoring concern; they do not establish how many historical labels change. That measurement remains reserved for the frozen sensitivity analysis. [API-Bank, EMNLP 2023](https://aclanthology.org/2023.emnlp-main.187/), [Python boolean semantics](https://docs.python.org/3.12/library/stdtypes.html#boolean-type-bool), [JSON Schema boolean semantics](https://json-schema.org/understanding-json-schema/reference/boolean).

Oren et al. propose an ordering-based contamination test with explicit exchangeability assumptions. Our five known training matches and eight prior probes are direct exposure disclosures, not an application of that test or a bound on all contamination. Undetected additional overlap cannot be ruled out by these counts. The historical differences therefore remain descriptive case-study measurements. [Contamination study, version 2, November 2023](https://arxiv.org/abs/2310.17623v2).

This targeted review establishes close prior work and narrows our contribution to the documented reconstruction, representation-loss inventory and failure accounting. It is not a systematic literature review or independent scientific validation.

## Subsequent native comparison: invocation collapse, with class outcomes retained

A separately frozen same-lossy-native FunctionGemma comparison completed both 400-record collections. The base scored 228/360 call and 0/40 no-call correct; checkpoint-600 scored 0/360 call and 40/40 no-call. The checkpoint showed invocation collapse in this condition, not improved tool calling. Class-balanced correctness rose from 31.6667% to 50% (+18.3333 percentage points), while pooled correctness fell from 228/400 (57%) to 40/400 (10%, −47 points). This illustrates why neither macro nor pooled should replace the two class outcomes.

Paired call cells were 228 base-only and 132 both-wrong; all 40 no-call cases were checkpoint-only.

The base had 347 native-call parses and 53 malformed outputs; checkpoint-600 had 400 natural-text classifications, no native-call parses and eight output truncations. All 400 cases per arm completed with known usage; input tokens were 139,237 each, returned tokens 18,686 versus 21,026. The base stopped on token50 for all400; checkpoint had392 EOS1 stops and8 capped continuations.

No-call success measures ordinary nonempty text without invocation, not semantic refusal quality. Aggregate-only review does not explain the content or cause of the checkpoint's behavior, and no repair or retry was performed.

Both arms used the same lossy declarations; the 19 unsupported schema-audit records stayed in the denominator. Five training-content overlaps and eight prior probe exposures remain; this is not equivalent full-schema information, a directly comparable Qwen endpoint, fresh or causal generalization, frontier transfer, or outside peer review. Paired analysis SHA256 `5d690a197d9392c70c6d6cce6d1236008857ea3c30b263ddf824ec0ce4990ff1`.

## Authorship and upstream contributions

Cisco Cáceres is responsible for experiment design decisions, implementation, provenance interpretation and claims. AI assistance supported implementation, methods review, arithmetic checks and drafting. Model architectures, publisher weights, tokenizers, runtimes and established evaluation methods are upstream work. No fictional coauthor or independent reviewer attribution is added.

## References

1. Cáceres, C. (2026). [Tool-caller names comparison: downloadable aggregate and commitments](https://tuneharness.com/research/2026-10-06/tool-caller-names-comparison-20261006.json).
2. Cáceres, C. (2026). [Tool-caller methods appendix](https://tuneharness.com/research/2026-10-06/tool-caller-methods-appendix-20261006.md).
3. Cáceres, C. (2026). [Dated TuneHarness research note](https://tuneharness.com/research/2026-10-06/).

These primary experiment sources support the reported measurements. A comprehensive related-work review is a remaining paper gate; this draft makes no research-priority claim.

Editorial provenance: the original TuneHarness publication copy derived from the Echo methods draft at commit fc22997341e99723e9d847271fd99377eb8588b6, source SHA256 `ce5522b113642193677f1d1193bd5c50a6b800dfab93662734dbf03cf79c2c6a`. The targeted full-method clarification from later Echo commit 37017dab is incorporated here; the inspected current Echo source SHA256 is `531b5f94fe622f086bc7ebc62ca560271373c7f547b4cbc4123dc41c3fb859bd`. This publication also appends the newly completed FunctionGemma native comparison, separately bound to paired-analysis SHA256 `5d690a197d9392c70c6d6cce6d1236008857ea3c30b263ddf824ec0ce4990ff1`. The original draft did not contain these later outcomes. Historical product naming is rendered as "earlier internal name" under site policy. This is an editorial methods update with developer-run evidence, not outside peer review, independent reproduction or a comprehensive related-work review.


[Download the completed native comparison aggregate and commitments](https://tuneharness.com/research/2026-10-06/functiongemma-native-comparison-20261006.json).

## Subsequent completed information-condition and sensitivity addendum

A subsequent separately frozen full-schema publisher collection completed all 400 slots. Full schemas and the truthful system instruction both changed from the names-only condition; this is a descriptive information-condition comparison, not an isolated schema-only intervention.

| Condition | Call correct | No-call correct | Equal-weight macro | Pooled correct | Incorrect |
| --- | ---: | ---: | ---: | ---: | ---: |
| Recovered names-ablation | 223/360 | 31/40 | 69.7222% | 254/400 (63.5%) | 146 |
| Publisher names-only | 52/360 | 22/40 | 34.7222% | 74/400 (18.5%) | 326 |
| Publisher full-schema + instruction | 235/360 | 27/40 | 66.3889% | 262/400 (65.5%) | 138 |

The same publisher weights changed from 74/400 to 262/400 pooled correct (+47 percentage points); equal-weight class macro changed from 34.7222% to 66.3889% (+31.6667 points). Both registered descriptive summaries are retained. Full-schema publisher still made 138/400 errors (125 call errors and 13 no-call errors). Its 262 pooled correct versus reconstructed names-ablation's 254 does not imply a better overall model: their class outcomes, information conditions and model priors differ.

All 400 full-schema slots completed with known usage, no observed input/output truncation, 346 strict-format and 370 permissive-parser-valid responses. Input tokens were 171,638 versus 54,968 for names-only; returned tokens were 21,377 versus 18,456. These are token accounting, not energy/latency measurements. Full schemas and instruction changed together; reasoning-enabled, optimized and native-tools publisher controls were not run.

The separately frozen, post-collection type-stricter argument comparator reconciled all original labels across the three arms and changed **0/1,200 correctness labels**. It distinguishes boolean from numeric values, disallows false-valued argument-container coercion and avoids lossy large-integer normalization. This is argument-correctness sensitivity, not strict JSON-format scoring; the original published scorer and results remain unchanged. Zero changes weaken those specific scoring concerns for this measured cohort, not for all future outputs. Calls were not executed.

Five exact training-content matches and eight prior probe exposures remain in the historical prefix. This does not establish causal training benefit, fresh/general tool-calling superiority, novelty, frontier transfer or production readiness. The separately reported FunctionGemma lossy-native invocation-collapse diagnostic remains unchanged; its cause is unknown from aggregate review. External paid model/compute spend was $0; electricity, hardware/labor and latency economics were not measured. Developer/AI-assisted internal checks are not outside review.

[Download the aggregate information-condition and sensitivity results](https://tuneharness.com/research/2026-10-06/tool-caller-information-sensitivity-20261006.json). It contains class/format/usage/paired counts and source commitments, without outputs, per-item IDs or private paths.

## Post-hoc FunctionGemma retained-output endpoint audit: 6 October 2026

A separate audit replayed all 800 retained outputs against the frozen native parser before evaluating two explicitly named alternative JSON endpoints. All original native labels reproduced: base 228/360 call and 0/40 no-call; checkpoint-600: 0/360 call and 40/40 no-call. Those original tables and interpretations of native invocation remain unchanged.

Strict JSON on the exact stored output, stripping surrounding whitespace only, yielded zero valid parses and zero correct cases in both arms. The second endpoint removes exactly one final literal `<eos>` only when the recorded stop token is 1 and the stored output ends in that literal. It does not remove interior tokens, duplicate terminators, `<end_of_turn>`, fences or commentary. On this separately labeled post-hoc endpoint, checkpoint-600 produced 250/360 valid call-case parses and 39/40 valid no-call parses, with 159/360 calls and 36/40 no-calls correct. Pooled correctness was 195/400 (48.75%); class-balanced correctness was 67.0833%. The base remained 0/400 correct on both JSON endpoints. Typed and legacy argument comparisons agreed on every audited label; all eight checkpoint call truncations remained incorrect in the 360-case denominator. Failures, unrun cases and stop-representation mismatches were zero.

| Retained-output endpoint | Base call / no-call correct | Checkpoint-600 call / no-call correct | Checkpoint pooled | Checkpoint class macro |
| --- | --- | --- | --- | --- |
| Original native parser, unchanged | 228/360 · 0/40 | 0/360 · 40/40 | 40/400 (10%) | 50% |
| Exact stored strict JSON | 0/360 · 0/40 | 0/360 · 0/40 | 0/400 (0%) | 0% |
| One terminal-EOS removal, when stop ID 1 | 0/360 · 0/40 | 159/360 · 36/40 | 195/400 (48.75%) | 67.0833% |

This audit qualifies a blanket interpretation of native invocation collapse: the checkpoint expressed some correct function-call content under a different post-hoc JSON endpoint. It does not revise the native endpoint, show causal model improvement, establish full-schema equivalence or diagnose why the mismatch occurred. Typed equality distinguishes booleans from numbers; finite integers and floats compare numerically, with the original permissive comparator reported separately. Empty-list no-call correctness is not semantic refusal quality. Historical exposure, lossy schema and checkpoint-provenance limits remain. No new generation, retries or paid inference were used; same-owner/AI-assisted checks are not external peer review. A separate 32-fixture diagnostic is not included in these 800 results.

[Aggregate endpoint audit and source commitments](https://tuneharness.com/research/2026-10-06/functiongemma-retained-output-endpoint-audit-20261006.json). The source-bound audit first reproduced all 800 native outcomes. A separate internal metadata reconciliation verified source commitments, 1604 ordered stage events and rate arithmetic; it did not reparse raw predictions or datasets. Neither step is external peer review. Public counts/hash commitments exclude raw outputs, references, item IDs and private paths; they do not enable public item-level correctness rescoring.


## Authored structural tool-use diagnostic: matched 160-case follow-up (6 October 2026)

After the historical findings, I authored 160 deliberately simple structural fixtures: 80 single calls, 40 parallel calls and 40 no-call cases. Both the reconstructed names-ablation and publisher Qwen3-1.7B received full argument schemas and the same JSON-output instruction. This changes the information condition and task source; it does not replace any historical comparison, native FunctionGemma table or endpoint audit.

| Retained result | Reconstructed names-ablation | Publisher Qwen3-1.7B |
| --- | ---: | ---: |
| Single-call correctness | 80/80 | 64/80 |
| Parallel-call correctness | 40/40 | 28/40 |
| No-call correctness | 40/40 | 40/40 |
| Original permissively parsed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Typed parsed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Strict JSON contract AND typed correctness | 160/160 (100%) | 132/160 (82.5%) |
| Call/no-call class macro | 100% | 88.3333% |
| Three-family macro | 100% | 83.3333% |
| Strict output-contract validity | 160/160 | 137/160 |
| Completed / known usage | 160/160 | 160/160 |
| Input / returned tokens | 59,192 / 5,369 | 59,192 / 5,525 |

All 320 requests completed; neither arm had input or output truncation. Paired correctness retained 132 both-correct cases, 28 reconstructed-only cases, zero publisher-only and zero both-incorrect cases. The original permissively parsed endpoint, a typed parsed endpoint and the separate strict JSON-contract conjunct all retained the same correctness counts. Strict output-contract validity alone is a separate measure: publisher has five contract-valid but incorrect cases. These are descriptive counts on this finite fixture set, not population estimates. Zero external API spend does not mean compute has no cost.

The original selected SFT checkpoint is still unidentified. The reconstructed names-ablation is a distinct recovered artifact with unresolved training lineage. Fixtures were authored after earlier findings; unknown pretraining exposure, catalogue priors, toy structure and author selection prevent causal training-benefit, broad generalization or frontier-benchmark claims. This is not an independently blind study or outside peer review. Metadata/source/receipt arithmetic and opaque-output hashes were replayed. A separate retained-output reclassification then reconciled all 320 original labels, with zero disagreements and no generation calls or tool execution. Typed equality and strict JSON-contract-conjoined correctness agree on these fixtures; this does not prove the endpoints agree generally. EOS and token counts remain checked against recorded worker metadata, not independently tokenized. No-call correctness here concerns the fixture contract, not a general semantic refusal evaluation.

[Aggregate counts and source commitments](https://tuneharness.com/research/2026-10-06/authored-tooluse-matched160-20261006.json). Aggregates cannot let outsiders rescore individual predictions; permitted data, predictions, model lineage and environment access remain separate reproducibility requirements.
