No background assumed. Read this and every claim on the overview page will make complete sense.
ChatGPT runs in a data centre. A small language model — 1 to 3 billion parameters instead of trillions — can run on the phone itself. No internet, no account, completely private. Slower and less capable, but yours.
A model is billions of numbers. At full precision each takes 4 bytes, so a 1B model would be ~4 GB. Quantization compresses each number to about 4 bits, shrinking the model to ~700 MB with minor quality loss. Q4_0, Q4_K_M and Q8_0 are different compression schemes; GGUF is the file format that stores them.
Answering happens in two phases. Prefill — the model reads your prompt; parallel work, can be very fast. Decode — it writes the answer one token at a time; sequential, always slower. Both are measured in tokens per second. A token is roughly ¾ of a word.
Nearly all of an LLM's compute is multiplying huge grids of numbers together, billions of times per answer. Whoever does matmul faster, wins. Everything else here is about making matmul faster.
Arm chips include special instructions that crunch many numbers in a single step. dotprod is a good shortcut and is nearly everywhere. i8mm is a far better one for 8-bit integer matmul — exactly what quantized models need. Crucially, it isn't everywhere: two flagship phones of the same generation can differ, so the fast path can never be assumed — only detected.
Compilers default to safe, generic code that runs on any arm64 chip — and uses none of those fast instructions. -march=… is permission to emit them. Across the four phones benchmarked it was worth 1.00×, 3.88×, 4.94× and 5.27× on prefill — same lever, four different ceilings, including a chip where it buys nothing because the instructions aren't there. And the ranking is not the feature ranking: the biggest gain belongs to a chip with no i8mm at all. The right flags are the ones matching that phone — which is why they're chosen after detection, and why the size of the win has to be measured rather than predicted.
Phone CPUs mix fast, power-hungry cores with slow, efficient ones — commonly two big and six little, though some chips add a third middle tier. The counter-intuition: using every core makes generation slower than using just the big ones — the little cores become the bottleneck everyone waits for. How many big cores, and at which indices, differs per phone.
Arm's official library of hand-tuned matmul routines for dotprod, i8mm and SME2. The obvious assumption — turn it on, get speed — is exactly what we tested. On the i8mm chip, for Q4_0: no measurable gain once the compiler flags are right, because plain compiled code already saturates that hardware. On the Pixel 7a, isolating it against a KleidiAI-free control went further: KleidiAI costs ~3.6% there — llama.cpp's own repack path is simply faster. On SME2 silicon or a different quant the verdict could invert — so it stays a lever under test, per device.
llama.cpp is the open-source C++ engine that runs GGUF models on CPUs — the industry standard for local inference. llama-bench is its built-in stopwatch: load a model, run fixed workloads, report tokens per second as JSON.
adb is a USB command line to the phone: push files, run programs, read sensors. Cross-compilation is building arm64 phone binaries on an x86 laptop — the Android NDK provides the compiler that makes it possible. No Mac required.
None of these are required to use the app. Each one turns a mysterious number on the overview page into an obvious one.
Single Instruction, Multiple Data: instead of adding numbers one at a time, the CPU processes a whole row of them per instruction. dotprod and i8mm are SIMD instructions specialised for the multiply-and-add work at the heart of matmul — that's the entire mechanism behind the 4.94×.
Writing each token means reading essentially every weight in the model from RAM. A ~700 MB model at 17 tok/s streams ~12 GB every second — close to what a phone's memory can move at all. That's why decode barely improves from faster math (1.34×) while prefill leaps (4.94×), and why smaller quantizations speed up generation: less data to haul.
While writing an answer, the model must "remember" everything said so far. The KV cache stores that work so each new token only pays for itself — and it grows with the conversation, living in RAM. One of the app's levers quantizes it to 8-bit (q8_0), roughly halving its size. Whether that's free or costly is measured, per phone.
A smarter ordering of the same attention math: work in tiles small enough to stay in the CPU's fast cache, instead of building one huge intermediate matrix in slow RAM. Identical answers, less memory traffic. It helps some chips and hurts others — which is exactly why the app sweeps it instead of assuming it.
A thread is one worker; a core is the hardware that runs it. Matmul splits nicely across threads — but each step ends with everyone waiting at a barrier for the slowest worker. Add workers on slow cores and the whole team slows down. That's the mechanism behind "2 threads beats 6".
A phone has no fan. Under sustained load the chip heats up and the OS quietly slows the cores to protect it — which silently corrupts any benchmark that runs configs back-to-back. It's why the harness enforces cooldowns between variants and logs temperature before and after each one.
The app can't ship a separate build per phone. So the engine binding packs six arm64 kernel builds into one APK and picks at startup by reading the CPU's feature list — an i8mm chip loads the i8mm build, a dotprod-only chip loads the dotprod one. Detection replaces assumption at the binary level too.
Android exposes the battery's voltage and current. Sample them during a benchmark, multiply → watts; divide token rate by watts → tokens per joule. Speed per unit of energy — the number that decides whether a config is actually worth it on a battery. A config 5% faster but 30% hungrier is a bad phone config.
A phone is a noisy lab: background apps, thermal state, and battery level all nudge the numbers. Every measurement is repeated five times and the median reported, so a single hiccup — or a single lucky run — can never become a headline.
If you change the compiler flags and a library and the thread count and things get faster, you've learned nothing. The ladder changes exactly one lever per rung, so every gain has an owner. It's how we can say the 4.94× belongs to the flags — and that the library everyone credits added ≈0%.
Each layer talks only to the one below it.
v8_2_dotprod_i8mm, a dotprod-only chip loads v8_2_dotprod. The fast path ships; detection decides which one runs./proc/cpuinfo and cpufreq, never assumed.This is the product in one diagram — the pipeline the Tune tab runs, with the TypeScript module responsible for each stage.
/proc/cpuinfo + cpufreq: features, clusters, where the big cores sit.The app ships alone. Everything else in the repository exists to produce and audit the evidence:
| Component | What it is | Why this one |
|---|---|---|
| llama.cpp | C++ inference engine for GGUF models | The industry standard for CPU inference. Builds anywhere, and exposes every knob we need to test — arch flags, KleidiAI, repack, threads. |
| React Native | Cross-platform app framework | The builder writes only TypeScript and Python. RN keeps 100% of authored code in TS while llama.rn handles the native inference layer. |
| llama.rn | React Native bindings for llama.cpp | Lets the TypeScript app call the C++ engine without writing native code. Audited: its prebuilts pack six arm64 kernel builds and dispatch by CPU feature at runtime — the same detection-over-assumption story the harness proves with explicit per-arch builds. |
| Android NDK | Cross-compiler toolchain | Builds arm64 phone binaries on an x86 Windows laptop. No Mac, no emulator. |
| adb | USB command line to the phone | Push binaries, run benchmarks, read CPU features and battery state — scriptable and reproducible, with no app install needed. |
| harness/bench.py | The measurement scientist | Enforces rigour automatically: five repetitions, cooldowns between variants, battery and thermal logging, raw JSON out. Anyone reruns every number with one command. |
| GGUF models | Pre-quantized Llama 3.2 1B | Small enough to iterate quickly, permissively licensed, and available in the exact quant formats the sweep compares. |
| KleidiAI | Arm's optimized kernel library | Included as a lever under test, never an assumption. Verdict so far: neutral for Q4_0 on the i8mm chip, −3.6% on the Pixel 7a — kept in the sweep because it may matter on other quants and on SME2 chips. |
You can't attribute a speedup unless exactly one thing changes.
| Variant | -march | KleidiAI | Repack | Question it answers |
|---|---|---|---|---|
| generic | none | off | on | What do you get if you don't think about any of this? The default. |
| arch | +i8mm+dotprod | off | on | What are the compiler flags alone worth? |
| kleidiai | +i8mm+dotprod | on | on | Does Arm's library add anything on top? |
| arch-norepack | +i8mm+dotprod | off | off | Baseline with llama.cpp's own weight-repacking disabled. |
| kleidiai-norepack | +i8mm+dotprod | on | off | KleidiAI's true standalone value, unmasked by repack. |
| dp-arch | +dotprod only | off | on | Target for phones without i8mm. |
| dp-kleidiai | +dotprod only | on | on | The same, with KleidiAI. |
Physical phone only, never an emulator. Five repetitions per configuration. Fixed workloads: 128-token prefill, 64-token decode. Cooldowns of 90–120 seconds between variants, so heat from one run can't slow the next. Battery level and temperature logged before and after every variant. Raw JSON committed to results/, so every published number is auditable.
Three families. Make the machine code match the silicon. Make the memory traffic smaller. Put the work on the right cores. Everything below is one of the three, plus the energy meter that judges them all.
| Optimization | Family · when | What it actually does | Achieved (Nothing 2a) |
|---|---|---|---|
| Arch flags | machine code · build-time | -march=…+dotprod+i8mm lets the compiler emit the chip's matrix instructions instead of lowest-common-denominator arm64. | 4.94× prefill, 1.34× decode |
| Weight repacking | memory traffic · load-time | Rearranges quantized weight blocks at model load into the tile order the SIMD units consume — streaming reads instead of scattered ones. | Folded into the 4.94×; the norepack builds isolate it |
| KleidiAI | machine code · build-time | Arm's hand-written matmul microkernels, swapped in for llama.cpp's own compiled paths. | ≈0% for Q4_0 — the honest negative |
| Quantization | memory traffic · model format | Weights stored at ~4 bits instead of 16. Decode is memory-bound, so less data ≈ more tokens per second — and a 1B model fits in ~700 MB. | Enables everything; Q4_0 vs Q4_K_M vs Q8_0 sweep is next |
| Thread count | scheduling · runtime, in-app | Derives candidates from the phone's real core map instead of "all cores". Little cores stall the team; fewer, faster workers win. | +25% decode (2 threads vs 6) |
| Flash attention | memory traffic · runtime, in-app | Reorders attention into cache-sized tiles — same math, fewer trips to RAM. | Swept per device by the tuner |
| KV cache q8_0 | memory traffic · runtime, in-app | Quantizes the conversation's memory to 8-bit, roughly halving its RAM footprint as the chat grows. | Swept per device by the tuner |
| Tokens per joule | measurement · runtime, in-app | Samples the battery's power rails during every benchmark point, so efficiency ranks configs alongside speed. | Reported per config where the kernel exposes rails |
Build-time levers (arch flags, repacking, KleidiAI) are proved by the harness and arrive in the app pre-baked inside llama.rn's feature-dispatched kernels. Runtime levers (threads, flash attention, KV quant) are the ones no binary can decide in advance — they depend on the individual phone, so the app measures them there. The headline numbers came from the first family; the product is the second.
Results are per device, so here are all four. One has i8mm; two have dotprod only; one has neither. We said a chip without i8mm would tell a different story, and would get its own table only once it was actually measured. Three now have, and none of them tells the story we expected — least of all the one where the lever does nothing.
Flip between the four and the thesis is right there. The same compiler-flag lever is worth 1.00×, 3.88×, 4.94× and 5.27×. We expected i8mm to explain that spread — the matrix-multiply instructions the 2a has and the others lack. It doesn't. The Pixel 7a has no i8mm whatsoever and still posts the largest gain, comfortably ahead of the chip that has it: its Cortex-X1 cores get more out of ordinary dot-product code than the 2a's A715s get from the fancy instructions. At the other end sits the Snapdragon 710, where the lever does nothing at all — with no dotprod in the silicon, the optimized builds either crash or tie the generic one. The feature checklist told us the wrong story, and only the measurement told us the right one.
This is the most uncomfortable result in the project, and the most useful. A reasonable engineer reads the spec sheets, sees i8mm on one phone and not the others, and confidently predicts the ranking. That prediction is wrong. Silicon doesn't reduce to a feature list, and a config picked from one is really just a guess.
The thread count says the same thing even louder, because every phone wants a different answer. On the A34, the generic build decodes fastest at 6 threads, yet once the arch flags are on, 6 threads becomes the worst choice and 2 threads — just the two big A78 cores — wins. On the Pixel the winner is 4 threads: it's the only tri-cluster phone here, with exactly four fast cores (2× Cortex-X1 + 2× Cortex-A78), so the fifth and sixth threads spill onto little A55s and prefill collapses from 143.5 to 101.8 tok/s, a 29% penalty for the crime of using more of your CPU. And the Realme 5 Pro inverts the rule completely: its two A75-class big cores are weak enough that 6 threads, little cores and all, is genuinely fastest. Two phones want 2 threads, one wants 4, one wants all 6, and one changes its mind depending on the build. No static default survives that.
There's one more asymmetry sitting in every ladder above, and it's easy to miss. The same arch flag that multiplies prefill by up to 5.27× barely nudges decode: 1.34× on the 2a, 2.11× at best. That isn't a bug in the optimization; it's the two phases hitting two different walls.
Reading your whole prompt at once, the model reuses each weight across every prompt token in one batched pass — so the cores are kept busy doing arithmetic. The ceiling is how many multiply-adds per second the CPU can retire, which is exactly what the dotprod/i8mm instructions and arch flags raise. Lift that ceiling and prefill leaps — hence up to 5.27×.
Writing one token at a time, the model must stream the entire set of weights out of RAM for every single token — a lot of data moved, very little math per byte. The ceiling is memory bandwidth, and no compiler flag makes RAM faster. So decode gains stay modest (1.3–2.0×), and the real decode lever is smaller weights — quantization — not better code.
Same optimization, two ceilings. It's why a single row can read 5.27× / 2.11×, and why the tuner weighs decode more heavily when it scores a config: decode is the wall you actually feel while the answer types itself out.
Four phones across three vendors (MediaTek, Google, Qualcomm) now, which retires the earlier "these are all MediaTek chips" caveat — but it is still four chips, one model, one quant family. Three honest wrinkles. One: the Pixel's generic baseline is genuinely noisy — it swung ±9% across runs — while its arch builds repeat to within 0.3%. The published Pixel ladder is a single same-session run (16 July, corrected build), so its baseline and arch numbers shared identical conditions; the raw JSON for every run is in results/, including the ones that disagree with each other. Two: "KleidiAI adds nothing" means on the two chips where it was isolated against a KleidiAI-free control — flat on the 2a, −3.6% on the Pixel. The A34's optimized build carries both levers at once, so its 3.88× is arch flags and KleidiAI together rather than the flags alone; going by the Pixel, the flags-alone figure there would be slightly higher. On SME2 silicon or with other quants the answer may flip, and we would publish that too. Three: the Realme's 1.00× is a result about compiler flags, not about the phone being slow — the floor it marks is architectural, and any pre-dotprod chip should land on it. Nothing here is presented as a law of Arm phones. Which is precisely why the product ships a measurement loop instead of a lookup table.
Next: more devices through the same harness, each one's ladder published beside these whatever it says, plus the quantization sweep (Q4_0 vs Q4_K_M vs Q8_0), energy per token via the battery sensor, and big.LITTLE speculative decoding. The llama.rn binary audit is done: its prebuilts carry the fast kernels and dispatch by feature, which is why the app can tune runtime levers instead of recompiling.