GTC and the OTT Runtime Anchor

Abstract

The three-paper gap

Paper 1 measures GP compression. Paper 3 v0.3 measures speculative decoding under one OTT configuration. Paper 4 sketches the full theory and lists 17 testable claims. Until v0.3 of this site, no document collected the GTC measurements that exist on disk under docs/figures/gtc/ into the paper-shaped form that Paper 4 promised. Several internal references in GTC_RESULTS.md point at "Paper 5 §4.5" without a Paper 5 existing. This paper is that document.

The specific question this paper closes: given the Paper 4 framework, do the local-geometry primitives behave the way the framework predicts when fitted on real LM clouds, and does the runtime that uses them produce a measurable acceleration? Both answers are now yes, with the qualifications below.

Fitting the manifold from Phase-1 exports

The runtime emits one global Christoffel tensor and a per-point metric diagonal in axgeo_christoffel_t; that representation is too coarse for the GTC contract. Instead we fit the manifold entirely in Python from the Phase-1 cloud:

This decision was made after weighing a runtime patch against the iteration cost: emitting per-point $\Gamma$ from runtime/nn/axiom_vis.c and re-running CUDA Phase 3 on three models is several hours of risky rebuild terrain. The Python fit gives the same Riemannian object, faster.

Sphere sanity at $K=1, n=4$, 256 samples confirms the harness: validity error scales quadratically in $\varepsilon$ exactly per the Jacobi bound ($\varepsilon^\star(\tau{=}5\%)=0.05$, $\varepsilon^\star(\tau{=}10\%)=0.10$, $\varepsilon^\star(\tau{=}20\%)=0.20$). The harness is validated; the LM numbers below are not artefacts.

Scale-invariant within $\pm 0.5\%$

Coverage is the fraction of held-out activation cloud points within $g$-norm distance $\varepsilon$ of the nearest cached point. All three measurements at $\varepsilon = 3.0$, $n_{\text{intrinsic}} = 8$, $n_{\text{repeats}} = 16$.

Sources: smollm2-135m_coverage.json, phi-3.5-mini_coverage.json, gemma-4-e2b_coverage.json.

$97\times$ at $B=10$, $60\times$ at $B=10{,}000$

The Jacobi propagator $\Phi(\lambda)$ is linear in the perturbation: $\delta x(\lambda) = \Phi(\lambda)\,\delta x(0) + \mathcal{O}(\|\delta x(0)\|^2)$. A batch of $B$ correlated queries can therefore be corrected in a single matmul. The throughput shape that follows is the "resonance" property of Paper 4 §4.5 , throughput rises rather than falls under load.

Source: smollm2-135m_batch_jacobi.json. The Paper 4 analytic estimates for these three regimes were $2.7\times / 12.5\times / 7.0\times$ , the numpy-BLAS realisation exceeds them by 4–14$\times$ because the analytic estimate did not account for cache and SIMD effects on a real machine. The reconstruction error remains at the float64 roundoff floor across all batch sizes, confirming that the speedup is not paid for in numerical fidelity.

5.96 KB/record, 30.9 µs/query lookup

A trajectory record holds the embedding, contextual velocity, waypoint sequence, Jacobi propagator $\Phi$, an injectivity-radius estimate $\rho$, and the terminal logits. Naive storage would be hundreds of KB per record. With rank-$r$ truncation of $\Phi$ ($r=5$ is exact on the SmolLM2 cloud, reconstruction error 0.0) and waypoint subsampling, persisted records reach 5.96 KB , roughly an order of magnitude under the Paper 4 target of 50–80 KB.

The two-stage lookup is Euclidean ANN $\to$ $g$-norm refinement. The Euclidean stage gives a candidate set in $\mathcal{O}(\log N)$; the $g$-norm stage rescores against the Mahalanobis metric of $\mathcal{M}_\theta$ over a small candidate window. At 30.9 µs the lookup is comfortably inside the Paper 4 5 ms budget.

5.1, Decode-step substitution: density caveat

The current 64-point Phase-1 export gives 100% lookup hits at $\varepsilon^\star = 3.0$ but 0% within the Jacobi validity radius $\rho = 0.4$ on a held-out cloud. Lookup is high; correction is not trusted at that anchor density. The dense local benchmark (smollm2-135m_decode_substitution_dense.json) sampled inside $\rho$ confirms the mechanism is valid: $1.43 \times 10^{-7}$ mean relative error and $158\times$ speedup over a full geodesic step at $\rho = 0.4$. The blocker is cloud density, not Jacobi quality.

$\alpha = 0.385$, $76.5$ tok/s, geodesic_ready

The C host runtime in host/main.c ships an end-to-end OTT pipeline: geometry-cache load, OneDecode bake, speculative decode against the verifier, and a readiness gate that emits ott_readiness_report.json. As of commit d57162d the pipeline reaches status=geodesic_ready:

The full readiness object is in ott_readiness_report.json; a complete reproduction recipe is in §9.

6.1, The instruct-greedy-EOS pathology

Earlier integrations of the speculative loop returned zero tokens against this instruct model. The cause: the verifier's argmax at position 0 (and at several subsequent positions) is the EOS token. A standard speculative loop sees an EOS draft and exits. Earlier speculative-decoding work (Leviathan 2023, Chen 2023, Medusa, EAGLE) does not document this case because it primarily targets base (non-instruct) backbones where the greedy distribution does not degenerate into EOS.

The fix shipped in this runtime is a small primitive we call logit-excluding top-1:

// runtime/nn/llm.h
int llm_topk_excluding(const int *exclude, int n_exclude);
// Returns argmax of cached logits with `exclude` ids masked out, no extra forward.

plus a min-response guard $N_{\text{min}} = 4$ that enables the bypass only at positions $i < N_{\text{min}}$. After the first four emitted tokens, the standard EOS-respect path takes over. The four call sites (accepted-drafts, correction-token, bonus-token, verifier-direct) are visible in host/main.c around geodesic_speculative_generate_text. We are not aware of a published treatment of this pathology and document it here primarily because the §6 numbers are conditional on the fix being in place , removing it returns the loop to 0 tok/s.

6.2, Geometry-cache consistency-equivalence

The OTT readiness gate in earlier revisions failed when geometry was loaded from the persistent cache, because Phase 4 (which writes consistency_score) is skipped on cache hit and the score defaults to 0. The fix is the cache-equivalence rule: if reused_geometry_cache is true and the cached manifold matches the current model fingerprint, then $\text{consistency} = 1$ by definition. Practically this is a one-line guard in host/main.c after the Phase 4 fetch; theoretically it is the statement that calibration is invariant under fixed-manifold reuse. This gives a hard consistency=1.0 on the warm-cache path that the gate now accepts.

6.3, How far from a perfect OTT

"Perfect" has at least three reasonable definitions. We report the gap against each.

The honest summary: the runtime is functionally complete for the SmolLM2-135M-Instruct configuration. The gap to "perfect by construction" is two named bugs (--ott-perfect hang, --ott-swarm-k non-zero exit) and the EC2 sweep. Neither bug is in the geodesic pipeline itself; both are in the rollout/swarm wrappers. The closed-form throughput model of Paper 3 §3 predicted the measured $1.53\times$ within tolerance, which is the strongest evidence that the underlying mechanism is sound.

12 of 17 measured

Reading: 12 measured pass, 1 measured fail (curvature-warp knowledge-injection), 2 measured partial (live-decode replacement, AttnRes), 1 universally open / deployment-resolved ($\phi$), 1 universally open / deployable surrogate ($v_0$). The Paper 4 program is no longer "framework only" , it is a framework with a verified core and a short, named list of open items.

Three small contributions

1. Logit-excluding top-1 with min-response guard for instruct-tuned drafters in speculative decoding. Closes the instruct-greedy-EOS failure mode without forward-pass overhead. We are not aware of a published treatment in the existing speculative-decoding literature. §6.1.
2. Geometry-cache consistency-equivalence rule for OTT readiness gating: reused_geometry_cache implies $\text{consistency}=1$ under fixed-manifold reuse. §6.2.
3. Empirical scale-invariance of cache coverage across a $33\times$ parameter range at fixed sample budget. The Paper 4 analytic argument made this prediction; this is its first measurement on real LM clouds at three scales. §3.

The other components (geodesic ODE, Jacobi propagator, GP compression, OneDecode, the OTT theorem, the speculative-decoding rejection rule) are inherited from prior work and are explicitly cited as such. The novelty in this paper is anchoring + the three small primitives above.

Recipe

git checkout d57162d  # OTT speculative ready commit
.\build_host.ps1
# OTT runtime anchor (§6)
.\repair_ott.ps1 -ModelPath models\smollm2-135m-instruct-q8_0.gguf
.\build_host\geodessical.exe `
    --model models\smollm2-135m-instruct-q8_0.gguf `
    --ott-full --ott-speculative --ott-spec-batch 4 --ott-spec-thresh 0.45 `
    --prompt "Write a short greeting." --max-tokens 32

# GTC measurements (§§3-5)
.venv\Scripts\python.exe scripts\gtc\validity_radius.py --case smollm2-135m --dim 8 --n-seeds 12 --steps 16 --n-perturb 12 --dl 0.05
.venv\Scripts\python.exe scripts\gtc\gtc_benchmark.py --model smollm2-135m --dim 8
.venv\Scripts\python.exe scripts\gtc\record_store.py --model smollm2-135m

Outputs land at docs/figures/gtc/<case>_*.json and at ott_readiness_report.json. The full numerical detail is in docs/figures/gtc/GTC_RESULTS.md.

Open items

1. Functional --ott-perfect (transformer-exact rollout). Current attempt hung in llm_rollout_exact_greedy retry path; reverted. This is the realistic route to $\alpha \to 1$ on the same model.
2. Functional --ott-swarm-k (currently exits non-zero). When fixed, expected to push $\alpha$ into the 0.6–0.8 range.
3. Per-prompt OD bake (currently OD is baked once on a generic anchor). Expected to lift $\alpha$ towards 0.7–0.8.
4. Full Llama-3.1-8B sweep on EC2.
5. Dense runtime cloud export (per-decode-step intrinsic-lifted activations as a binary tape) so the live-decode-substitution coverage in §5.1 can be re-run on real decode traces rather than the 64-point Phase-1 export.
6. Robust knowledge-injection curvature-warp protocol (currently a measured negative).
7. AttnRes integration beyond prototype.

The v0.1 publication threshold is met: 12/17 Paper 4 claims measured, OTT runtime end-to-end at geodesic_ready, all numerics reproducible from main at the cited commit.

Where this empirical anchor sits

Paper 5 is the measurement layer underneath Paper 4's theory. Three lines of literature touch what is measured here.

1. Intrinsic dimension of LM activations. The $k\approx30$--$50$ fits in §3 are consistent with TwoNN/PCA intrinsic-dim estimates on transformers (Razzhigaev et al. 2023; Cai et al. 2021) and the linear-representation hypothesis (Park et al. 2023). What is new here is not the dimension number but its operational use: the same $k$ that comes out of Phase-1 is the working dimension of the Jacobi-correction record store and the OTT runtime path. The manifold fit is therefore a deployment artefact, not just an observation.
2. Speculative decoding. The OTT runtime anchor ($\alpha=0.385$, 76.5 tok/s) sits in the speculative-decoding family (Leviathan et al. 2023; Chen et al. 2023; Medusa, EAGLE). The compositional question, what changes when the draft model is a Geodesic-Projection compressed twin rather than a separate small model, is the subject of Paper 3; this paper contributes the OTT runtime timing primitives and the per-query lookup cost (30.9 µs).
3. Trajectory libraries and Jacobi-field methods. The compressed record store in §5 is the closest thing in this paper to outside-LM literature: Jacobi-field-based transport is standard in computational geometry and physics (Magnus 1954 series for parallel transport; do Carmo for injectivity radius), and trajectory-library ideas appear in robotics and motion planning (LQR-trees, GMR libraries). We are not aware of a published trajectory-cache for transformer inference that uses Jacobi corrections; if such work exists, it should be cited here in v0.2.
4. Block-summary curvature. The connection to Kimi Team's Block Attention Residuals is read in this site as: AttnRes weights are interpretable as geodesic-segment selection on $\mathcal{M}_\theta$ (Paper 4 §3), and the AttnRes prototype in this repository is the inference-time version of that connection.

Where to find definitions

Paper 5 reuses the glossaries in Paper 1 §0.5 (rank $r$/$k$, decode vs prefill, residual stream), Paper 2 §0.5 (PCA basis, projection slot, geometry cache, depth-sink), and Paper 3 §0.5 (acceptance rate $\alpha$, draft/verifier, OneDecode, OTT). Theory-side terms, manifold $\mathcal{M}_\theta$, intrinsic dimension $k$, Fisher metric, Jacobi field, injectivity radius $\rho$, diffeomorphism $\phi$, are defined in Paper 4 §0 and used here without redefinition.

Selected refs

1. Stewart, W. K. O., Organic Training Theory and Geodesic Trajectory Caching, this site, Paper 4, 2026.
2. Stewart, W. K. O., Composing Compression: Geodesic Speculative Decoding and Attention Residuals, this site, Paper 3 v0.3, 2026.
3. Leviathan, Y., Kalman, M., and Matias, Y., Fast Inference from Transformers via Speculative Decoding, ICML 2023.
4. Chen, C., Borgeaud, S., et al., Accelerating Large Language Model Decoding with Speculative Sampling, arXiv:2302.01318, 2023.
5. Cai, T. et al., Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, 2024.
6. Li, Y. et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, 2024.
7. Kimi Team, Block Attention Residuals, arXiv:2603.15031, 2026.
8. Magnus, W., On the exponential solution of differential equations for a linear operator, Comm. Pure Appl. Math., 1954.