Results
With Grace, and alone, on the same task
Google's Gemma 4 (26B), running locally on one machine
Tested on one AMD Ryzen AI 9 HX PRO 370 machine with integrated Radeon 890M graphics and no separate GPU (Ubuntu 26.04; Gemma 4 26B, 4-bit, through Ollama). Grace ran on the same machine.
| Study |
With Grace: right first time |
Alone: right first time |
Tokens on Day 2 (with · alone) |
Time to accepted (with · alone) |
| Oct 14 |
3 of 3 |
0 of 3 |
81K · 391K |
6 min · 17 min |
| Oct 20, repeat |
3 of 3 |
0 of 3 |
172K · 319K |
10 min · 18 min |
| Oct 27, current release |
2 of 2 |
0 of 3 |
50–122K · 377K–2.0M |
4–7 min · 13–42 min |
Working alone, Gemma repeated the corrected mistake every time; with a notes file (Oct 14) it was right the first time 1 of 3. Single figures are medians.
Anthropic's Claude Code, four different tasks
Claude Code (Opus) runs in Anthropic's cloud; Grace ran on an Apple M5 Mac with 24 GB of memory. Each task had its own unstated team rule. In every run, Claude Code alone was also corrected on Day 1, kept its own settings, and was then given the same request on Day 2.
| Task |
With Grace: right first time |
Alone, corrected on Day 1 too: right first time |
Cost per Day 2 session, median (with · alone) |
| Notes app |
5 of 5 |
0 of 5 |
$0.19 · $0.48 |
| To-do app |
5 of 5 |
0 of 5 |
$0.16 · $0.35 |
| Expense ledger |
5 of 5 |
0 of 5 |
$0.24 · $0.57 |
| Bookmarks |
5 of 5 |
0 of 5 |
$0.22 · $0.45 |
| All four |
20 of 20 |
0 of 20 |
$0.21 · $0.47 |
Claude Code alone repeated the corrected mistake in all 20 runs; 20 of 20 against 0 of 20 gives p = 7 × 10−12 (Fisher's exact test, one-sided). Three of the four tasks were written independently, without access to Grace's design. With Grace, Claude used 45% fewer tokens on Day 2 (median 81K against 147K).
All studies October 2026. "Right the first time": Day 2 passed the reviewer's tests on the first review.