Geodesic Projection

Abstract

Terms

What Paper 1 measures, what GP generalises

Paper 1 presents a deliberately narrow design: shared rank $k$ across layers, attention slots only ($Q$, $K$, $V$; $O$ skipped), no FFN compression, no quality fine-tune. The narrowness is the point, it lets the calibration-free claim and the super-baseline claim be tested without confounding from other compression knobs.

The runtime implements a more general scheme. The shape of the generalisation is summarised below. Each row is a knob in the production pipeline that Paper 1 holds fixed.

Paper 1's measurement uses the GP code path with all but the first knob held in the simplest position. The numbers reported there therefore are GP numbers in the limit where GP collapses to attention-only fixed-rank. What GP buys above and beyond is (a) coverage of the FFN, which is where the bulk of Llama-3.1-8B's parameters live (FFN ≈ 70% of weights in this architecture), and (b) the cache architecture that makes the build cost a one-time charge instead of a per-run charge.

The honest position on FFN compression is that the runtime can compress it but that Paper 1 deliberately did not because the FFN's singular-value spectrum on Llama-3.1-8B is dramatically flatter than the attention slots'. We present the spectrum evidence in §3 and the consequence for compression quality in §7.

Why a low-rank basis exists at all

The premise of GP is that the local activation manifold of a trained transformer has intrinsic dimension much smaller than the ambient model dim $d$. If that premise is false, low-rank weight projection should be uniformly destructive. The manifold-extraction pipeline checked in under legacy/axiom_vis/ runs four phases on a model, manifold sampling, symmetry detection, curvature estimation, and axiom-set extraction, and emits per-phase JSON. We summarise the Phase 1 (intrinsic-dimensionality) and Phase 4 (axiom-set) outputs below for three open-weight models.

Source data: phase1_manifold.json and phase4_axioms.json for each model under legacy/axiom_vis/<model>/. "Consistency" is the Phase 4 self-consistency score: fraction of the discovered axiom set that survives held-out rebuild on a disjoint sample.

Why FFN-down gets a different code path

The seven compressible matrices in a Llama-style transformer block, with shapes for Llama-3.1-8B ($d = 4{,}096$, $d_\text{ffn} = 14{,}336$, $h = 32$ heads):

Attention   Q          : [d, d]            =  [4096, 4096]
            K, V       : [d_kv, d]         =  [1024, 4096]   (GQA: 8 KV heads x 128)
            O          : [d, d]            =  [4096, 4096]
FFN         W_up       : [d_ffn, d]        =  [14336, 4096]
            W_gate     : [d_ffn, d]        =  [14336, 4096]
            W_down     : [d, d_ffn]        =  [4096, 14336]

Note: Llama-3.1-8B uses grouped-query attention, so $K$ and $V$ have only $d_\text{kv} = 1{,}024$ output rows (not $d = 4{,}096$). This is one reason rank $k = 1{,}024$ is a particularly natural Pareto knee: on $K$ and $V$ the projection becomes lossless (you cannot have more singular values than the smaller matrix dimension), so the only quality cost at $k = 1{,}024$ comes from $Q$ and (when included) $O$.

The attention slots, plus $W_\text{up}$ and $W_\text{gate}$, are tall in the same direction (fewer rows than columns is allowed too; what matters is that the "compressed" dimension is the one the basis is built over). For these slots GP uses the same construction as Paper 1: form the Gram matrix $K = W W^\top \in \mathbb{R}^{d \times d}$, take its top-$k$ eigenvectors $U \in \mathbb{R}^{d \times k}$, and store the projected weight $W_\text{proj} = W^\top U \in \mathbb{R}^{\text{out-dim} \times k}$.

$W_\text{down}$ is the one slot where this construction is unsatisfying. Its eigenspectrum on Llama-3.1-8B is approximately flat over the first $\sim$3,400 components, see Paper 1 §10, Figure 3, meaning that truncating to $k = 1024$ drops more than 30% of its Frobenius energy on every layer we measured. For this reason the production code path for $W_\text{down}$ uses a direct SVD rather than the eigendecomposition-of-Gram route, and the runtime exposes --axex-skip-o / --axex-attn-only flags that exclude $W_\text{down}$ (and $O$) entirely when the user judges the FFN penalty to be the dominant quality cost. Paper 1 runs in exactly this configuration.

3.1 Per-slot spectra on Llama-3.1-8B (measured)

Singular-value spectra for all seven slots, computed by full SVD on the dequantised Q4_K_M weights at layers $\{0, 7, 15, 23, 31\}$, see scripts/analysis/compute_spectra.py and the JSON output at docs/figures/spectra_summary.json. All values are layer-means; ranges across the five layers are given where they are informative.

Allocating the rank budget

A naive scheme assigns the same $k$ to every layer. Paper 1 uses this scheme. GP exposes a heuristic that allocates more rank to layers with steeper local geometry on the activation manifold. Concretely: at axiom-discovery time the pipeline emits a per-layer Manifold-Curvature-Ratio (MCR) scalar, derived from a coarse Ricci-style curvature estimate computed on the activation samples for that layer's input. Layers with high MCR are allocated rank closer to AXEX_MANIFOLD_K_MAX (currently $1{,}536$, see runtime/nn/axiom_exploit.h); layers with low MCR are allocated less. A hard floor prevents pathological collapse:

k_layer = clamp(round(k_target * mcr_layer / mcr_mean),
                k_floor,
                AXEX_MANIFOLD_K_MAX)

The $k_\text{floor}$ matters more than the upper cap. Without it, the heuristic aggressively starves shallow layers in small models, and we observed an immediate decode-quality collapse on SmolLM2-135M ($d = 576$) when $k$ for any attention layer fell below approximately $0.4 d$. With the floor at $k_\text{floor} = 0.55 d$ (≈ 320 for SmolLM2; ≈ 845 for Gemma-4-E2B; ≈ 1,690 for Phi-3.5-mini, which exceeds the cap and so falls back to it), the failure mode disappears. This is documented here as a contingency of the heuristic, not as a measurement: we have not run a quality sweep that pins down the precise floor curve.

Turning a minutes-long startup into a seconds-long startup

Building the GP bases from scratch on Llama-3.1-8B takes roughly 70 seconds on the reference RTX 4070 Laptop. This is dominated by the eigendecompositions of 32 layers × 7 slots = 224 Gram matrices. On the 70B target the same pass is extrapolated at ~70 minutes (80 layers × 7 slots, larger $d$). To make GP practical the runtime persists two artefacts:

1. ott_geometry.bin, per-layer spectra, axioms, depth-sink index, intrinsic-dim estimate, weight-blob hash, integrity hash. Small (a few MB).
2. W_proj cache (one file per slot per layer), the projected weights themselves. Large; for Llama-3.1-8B at $k=1{,}536$ this is the 1,093 MB figure cited in Paper 1.

Both caches are keyed on a hash of (model file digest, AXEX flag set, target rank). Mismatches force a full rebuild rather than risking a stale basis.

5.1 The depth-sink shortcut

Reloading the full geometry cache on a 70B model is not free, at minimum we want to verify the cache is not stale. The runtime's solution is the depth-sink layer: empirically, transformer activations saturate in effective dimensionality at a single layer roughly two-thirds of the way through the stack, and the spectrum at that one layer is enough to detect almost all weight-blob corruption. The cache integrity check therefore reads only the depth-sink layer's spectrum from ott_geometry.bin and recomputes it on the fly, which on Llama-3.1-8B takes under 200 ms.

The depth-sink is identified during the original axiom run as the layer where the cumulative-explained-variance curve flattens to within $10^{-3}$ of its plateau. On the four models we have inspected, this is layer 21/32 (SmolLM2), layer 19/26 (Gemma-4-E2B), layer 22/32 (Phi-3.5-mini), and layer 22/32 (Llama-3.1-8B). The two-thirds rule is a property of these four models, not a theorem; we flag it explicitly because we don't know yet whether it holds for substantially deeper architectures.

Numbers GP shares with Paper 1, and what changes when knobs move

The end-to-end numbers GP produces on its empirical anchor, Llama-3.1-8B-Instruct Q4_K_M, RTX 4070 Laptop, locked 30-second cooldown protocol, are the Paper 1 numbers by construction, since GP at attention-only-shared-rank is the Paper 1 configuration. We summarise them here with the GP framing rather than the calibration-free framing.

All four PPL values measured under --ppl-eval on WikiText-2 first 512 tokens; raw outputs in docs/figures/ppl_sweep/llama31_8b_ppl_sweep.json (date 2026-04-22). Two structural observations: (i) $k=1{,}024$ collapses quality (+61% PPL) because the K/V projection dimension on GQA-Llama-3.1-8B is exactly $1{,}024$ , at $k=1{,}024$ the Q matrix is rank-deficient against its full $4{,}096$-dim, while K and V are at the boundary; (ii) $k=1{,}536$ and $k=2{,}048$ produce identical PPL because once $k \ge 1{,}024$ the K and V matrices are full-rank (lossless), so additional rank only affects Q, whose PCA energy is already saturated by $k=1{,}536$. The $k=2{,}048$ disk footprint matches $k=1{,}536$ because AXEX_MANIFOLD_K_MAX in runtime/nn/axiom_exploit.h silently caps the stored basis. Operationally: $k=1{,}536$ is the Pareto rank for this model on this protocol , $k=2{,}048$ is wasted memory and compute, $k=1{,}024$ is unsafe.

6.1 What is not in this table

• FFN-included configurations. The runtime supports them; the benchmark pass is gated on a 30-second-cooldown rerun and is not in the v0.2 pack.
• MCR-driven per-layer rank vs shared rank. Same gating.
• End-to-end numbers on SmolLM2-135M, Gemma-4-E2B, Phi-3.5-mini. These models loaded and ran under GP during the manifold-extraction pass; we observed qualitatively that decode remained coherent at attn-only $k = 0.6 d$ but did not produce paired-CI throughput or PPL numbers under the locked protocol. The honest claim is "GP runs without crashing on these four models", not "GP is measured on these four models."
• Llama-3.1-70B. Compute is approved, the run is queued; no numbers yet.

Failure modes and contraindications

7.1 Small $d$ (≤ 1024)

On SmolLM2-135M ($d = 576$) we observed text-quality collapse at any attention rank below roughly $0.4 d$. The mechanism we suspect: at small $d$, attention head dim $d_h = d/h$ is already small (≈ 64), so further compression eats into the per-head rank budget in ways that the per-block PCA does not preserve. The runtime defends against this with the $k_\text{floor}$ in §4; in practice this means GP at $k_\text{floor} = 0.55 d \approx 320$ on SmolLM2 is near-lossless and at $k = 200$ it is broken. There is no useful compression to be had on this model, the floor and the ceiling are too close together.

7.2 FFN-only configurations

The FFN-down spectrum is too flat for the eigendecomposition route. The runtime's SVD path makes the math work but does not change the underlying compressibility of the matrix. Empirically, FFN-only GP at $k = 1024$ on Llama-3.1-8B is not faster than baseline (it is, in the spot checks, slightly slower, since it adds a projection without saving enough bandwidth). We have not characterised this pattern with a full benchmark sweep and so we cite it here as observation rather than measurement.

7.3 The fp32 $W_\text{proj}$ format

The persistent $W_\text{proj}$ cache is currently stored as fp32. Converting it to Q8_0 would close most of the disk-footprint gap but would re-quantise weights that have already been quantised once (Q4_K_M dequantised -> fp32 -> Q8_0), and the behaviour at the second quantisation boundary is not characterised. This is the same concern Paper 1 §11 raises and applies identically here.

7.4 Cache invalidation cost

A weight-blob hash mismatch forces full re-axiom plus full $W_\text{proj}$ rebuild. On the 70B target this would cost ~70 minutes and ~14 GB of disk. Operationally this means model upgrades are an event, not a transparent rolling deploy.

Commands and expected output

The Paper 1 reproduction recipe at repro/REPRODUCE.md reproduces the empirical-anchor row of §6. The cross-model intrinsic-dim numbers from §2 are reproducible via:

# intrinsic-dim re-run for any of the three models in legacy/axiom_vis/
.\build_host\geodessical.exe <model.gguf> \
    --axex-axiom-only --axex-export-vis <outdir>
# then read <outdir>/phase1_manifold.json

Tolerance: intrinsic-dim estimate is stable to ±1 across reseeded runs of the Phase 1 sampler at $n_\text{samples} = 64$; consistency score is stable to ±0.02. The actual axiom set is sample-dependent (random projections); only the size and the consistency are stable.

What this paper is missing before it is finished

1. Multi-model end-to-end PPL and decode tok/s under the locked protocol (SmolLM2, Gemma-4-E2B, Phi-3.5-mini, plus 70B).
2. FFN-included sweep on Llama-3.1-8B at fixed total parameter budget.
3. MCR vs shared-rank A/B at matched mean-$k$.
4. Per-slot spectra on the other three reference models (SmolLM2-135M, Gemma-4-E2B, Phi-3.5-mini); only Llama-3.1-8B is measured today (see §3.1).
5. Cache-rebuild timing on 70B (currently extrapolated, not measured).

None of the above blocks the runtime from being usable today; they block the publication-grade version of the GP claim.

Where Geodesic Projection sits in literature

Paper 1 §3.5 placed the single-slot $Q/K/V$ basis next to ASVD, SliceGPT, FWSVD, GPTQ, AWQ, and LoRA. Paper 2 inherits that placement for the attention slots and adds three more axes that do not appear in those works: multi-slot fitting across $Q/K/V/O$ plus FFN up/gate/down, persistent geometry caching, and a depth-sink shortcut that lets later layers re-use earlier bases.

1. Activation low-dimensionality. The intrinsic-dim result in §2 is consistent with the isotropy/anisotropy line of work (Cai et al. 2021; Razzhigaev et al. 2023) and the linear-representation hypothesis (Park et al. 2023). Paper 2's contribution is operational rather than theoretical: it commits to a single PCA basis per slot, ships it to a runtime, and caches it across quantisation levels.
2. FFN compression. ASVD (Yuan et al. 2023) and FWSVD (Wang et al. 2023) cover activation-aware FFN decomposition; the FFN-down SVD path in §3 is a deliberate reduction of those ideas to the smallest scheme that shares the geometry cache used by the attention slots, not a new contribution to FFN compression on its own.
3. Per-layer rank allocation. The MCR/Ricci heuristic in §4 sits next to compression-aware bit-budgeting work (GPTQ, AWQ, OmniQuant) but operates on rank rather than bits, on a fixed-precision integer budget, and against a curvature signal computed from the same Phase-1 cloud that produces the basis. The closest published precedent is Kimi Team's block-summary curvature signal in Block Attention Residuals (arXiv:2603.15031, 2026); the connection is made explicit in Paper 3 §4 and Paper 4 §3.
4. Geometry caching. The persistent $W_\text{proj}$ cache (1,093 MB on Llama-3.1-8B Q4_K_M, see §5) is closer in spirit to a compiled artefact (e.g. TensorRT plan caches, Triton kernel caches) than to existing compression work. We are not aware of a published compression pipeline that exposes the basis as a cache shared across quantisation levels for the same architecture; if there is one, this paragraph is the place we will cite it.
5. What we are not doing. We do not retrain (so this is not LoRA / DoRA territory), we do not change the attention algorithm (so this is not FlashAttention / Ring Attention territory), and we do not change the quantisation format (Q4_K_M is taken as a fixed input). Geodesic Projection is a pre-quantisation geometry pass; the papers it interacts with most are Paper 1 (the empirical anchor), Paper 3 (speculative decoding on top), and Paper 6 (adaptive layer that turns the static rank in this paper into a phase-aware schedule).

Selected refs

1. Cai, T. et al., Isotropy in the Contextual Embedding Space (ACL 2021), activation low-dimensionality on language models.
2. Razzhigaev, A. et al., The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (EACL 2023), TwoNN/PCA intrinsic-dim across architectures.
3. Eckart, C. and Young, G., The Approximation of One Matrix by Another of Lower Rank, Psychometrika (1936), optimality of truncated SVD.
4. Hu, E. et al., LoRA: Low-Rank Adaptation of Large Language Models (ICLR 2022), low-rank decomposition of weight updates.
5. Williams, S. et al., Roofline: An Insightful Visual Performance Model, CACM (2009), bandwidth/compute roofline.
6. Touvron, H. et al., Llama 3 Herd of Models (2024), reference architecture.
7. Park, K. et al., The Linear Representation Hypothesis (2023), supports the manifold premise of §2.
8. Kimi Team, Block Attention Residuals (arXiv:2603.15031, 2026), cited for the AttnRes interaction discussed in Paper 3.