The Break Test Kit — I Commissioned 3 Tools From a 77,000-Star Free Coding Agent, It Reported All 3 Working, and Only 1 Was
The companion to the full video on earendil-works/pi — a free, MIT-licensed terminal coding agent with 77,527 GitHub stars (verified via the GitHub API July 25, 2026), whose top contributors are Mario Zechner (creator of libgdx, 3,449 commits) and Armin Ronacher (creator of Flask and Jinja2, 437 commits). Pi ships deliberately incomplete — no MCP, no sub-agents, no permission popups, no plan mode, no built-in to-dos, no background bash — and expects you to have the agent build what you need as a TypeScript extension. So we commissioned three tools that do NOT exist among its 79 shipped examples, then re-tested every one by hand. INSIDE THIS KIT: (1) THE THREE PROMPTS VERBATIM — copy-paste ready. (2) JOB ONE, THE PRIVACY GATE — it installed a real OCR engine on its own initiative rather than faking the hard part (genuinely impressive), then failed three ways: tesseract read '/home/' as '/nome/' so a leak shipped; the same image blocked with 'cp' shipped with 'cat' because the trigger is a verb list, not a chokepoint; and with the OCR engine removed it allowed everything silently, because the error path is catch-and-allow. Its apt-get install died with the container, so on any fresh machine that gate is loaded, running, and enforcing nothing. (3) JOB TWO, THE SPENDING CEILING — better engineered, every API real, correct per-token cost from pi's own session format. It stopped the session AND spent 6x the ceiling doing it: session entry #9 shows a fully billed $0.006674 response recorded AFTER the $0.001686 limit was active, while its summary claimed 'the request never leaves your machine.' Plus: a new session has no ceiling at all, and the warning is invisible in the unattended print mode you'd actually walk away from. (4) JOB THREE, THE ONE THAT WORKED — a claim verifier that spawns a second pi instance as a fact-checking sub-agent. Its own fixture proved nothing (Python's release date, the boiling point of water), so we fed it four deliberately wrong facts unknowable from training AND deleted its search helper first. It got all four right and live: 77,527 stars to the exact unit, v0.82.1, MIT, 2025-08-09 — routing around its own broken dependency through bash. (5) TWO MODELS COMPARED — identical prompts on a second model produced comparable code but opposite disclosure: it volunteered, unprompted, that a hard ceiling cannot stop an in-flight call because true cost is only known after the provider returns, and shipped a section headed 'Important limitations (so it doesn't guess).' (6) THE BREAK TEST — the five-step reusable method that found every failure here, plus pi's own Dockerfile so none of this touches your machine. Also inside: pi has NO permission system (its README says it runs with your full permissions), and signing in with a Claude Pro/Max subscription is NOT free — pi's docs state third-party harness usage bills per token as extra usage. All results are our own runs on July 25, 2026, reported exactly as they came out. Independent field kit from Hyperautomation Labs — not affiliated with earendil-works.
Free. No spam. Unsubscribe anytime.