A small contextual model, tuned to correct, not rewrite.
The selected runtime is a fine-tuned Llama 3.2 1B model merged and quantised to GGUF Q4_K_M. Fine-tuning was essential: the untuned base model produced zero clean correction-only outputs across the eight manual cases.
- Model size
- 770.3 MB
- Peak working set
- 1,327 MB
- Median load
- 1,286 ms
- Inference engine
- llama.cpp · AVX2
- Decoding
- Greedy · cached prefix
- Target hardware
- 8-core laptop CPU
What we trained, and what each experiment taught us.
Identity, rules, SymSpell
Established the “do nothing” floor and exposed the cost of dictionary-only correction. SymSpell changed 44% of cases incorrectly in the early suite.
ByT5-small
The first pilot halved CER from 10.1% to 5.0% and reached 43.8% exact match, proving the signal was learnable. Later versions improved spelling but still missed contextual grammar.
Llama 3.2 1B + LoRA
A compact language model learned correction-only behaviour while using sentence context. The untuned base comparison demonstrated that the gain came from fine-tuning, not prompting alone.
F16 → Q8 → Q4_K_M
Q4_K_M reduced model size by 67% versus F16 and produced the same eight clean outputs as Q8 in the manual test, at the lowest measured sentence latency.
Three views of quality, kept separate.
We do not collapse incompatible datasets into one headline score. The spelling subset measures the target behaviour, the full suite adds harder real-error material, and the public NSEC companion corpus stresses split and concatenated words.
Core spelling subset
650 examples
Complete mixed suite
900 examples
Public split/concatenation stress test
700 of 9,060 examples
The model improved 527 inputs, but overcorrected 128. Extreme word segmentation, such as letters separated one by one, remains a clear weakness.
Scope: this is the author’s downloadable NSEC companion corpus, not the unavailable 12-category test file used for the published LREC rankings.
Quantisation changed the product equation.
| Variant | Size | Sentence mean | Clean outputs | Decision |
|---|---|---|---|---|
| Base Q4 | 770 MB | 4,590 ms | 0 / 8 | Rejected |
| Fine-tuned F16 | 2,365 MB | 1,118 ms | 7 / 8 | Too large |
| Fine-tuned Q8 | 1,260 MB | 696 ms | 8 / 8 | Strong |
| Fine-tuned Q4_K_M | 770 MB | 444 ms | 8 / 8 | Selected |
AMD Ryzen 9 4900HS, 8 cores / 16 threads, AVX2. Greedy decoding with cached instruction prefix. Medians of three complete runs.
Competitive evidence, without inventing a shared leaderboard.
On its own spelling suite, TypoPilot’s 95.70% word-accuracy proxy is numerically above systems reported by the LREC 2020 comparison. The datasets and scorers differ, so TypoPilot remains officially unranked until it runs on the paper’s exact test inputs.
Source: Näther, LREC 2020. Apple publishes no absolute CER, WER, or false-correction benchmark for its current autocorrect, so it cannot be placed responsibly.
What still breaks, and what we should train next.
Real-world C4 errors
Exact match falls to 10.67% and overcorrection rises to 31.33%. These examples differ sharply from the synthetic training distribution.
Extreme segmentation
The public split/concatenation sample reaches only 50.14% exact, showing that letter-by-letter splits and merged spans need dedicated training.
Grammar is intentionally conservative
The model was optimized primarily for spelling and minimal edits. It may preserve awkward grammar rather than rewrite professional or personal voice.
English first
The original vision includes Hinglish, but the current recommended model and reported results are English-focused. Hinglish should not be implied as complete.
No official Apple/Gboard comparison
Neither exposes a reproducible public API and absolute shared-dataset results are unavailable. Individual examples are useful demonstrations, not a universal ranking.
CPU performance varies
All reported native timings use one Ryzen 9 4900HS laptop. Sentence length, thread count, instruction set, and thermal state affect latency.
How to read the numbers.
- Exact match
- The model output must equal the reference character for character, including capitalization and punctuation.
- CER
- Character edit distance divided by reference length. Lower is better.
- WER
- Word-token edit distance divided by reference token count. Lower is better.
- Overcorrection
- The output changed the input but ended farther from the reference than doing nothing.
- Native timing
- The model is loaded once, retained in memory, and receives a TSV batch through the C++ harness. Reported per-example latency excludes model load.
- Cross-system ranking
- Only systems evaluated on identical inputs with the same scorer should share an official position.