I started building Praxis in early July 2026 and have been adding to it a piece at a time ever since. It logs my work and decisions across a dozen projects so I can ask about any of it later. Looking back over those three months, the AI itself was rarely the hard part. What kept tripping me up was speed, cost, tests I trusted too much, and plans that fell apart the moment they met real data. Here's what I learned.
1. Fast, cheap or local: I couldn't have all three
I noticed the speed problem through voice. If I type a question, I'll happily wait ten seconds for the answer. If I ask it out loud and nothing happens for ten seconds, it feels broken.
How long one answer took
Seconds per answer. Shorter is better. Hidden thinking was the biggest cost.
I had three places I could run the model, and each one cost me something:
- The paid API was fast, around 2–4 seconds for a spoken answer, but every question spent API credits.
- My Claude subscription, through Claude Code's headless mode (
claude -p), cost nothing extra, but the same answer took 12.6 seconds. - A local model on my laptop was free too, but it took 8.4 seconds once loaded and 27.5 seconds from cold, used 16.8 GB of my 24 GB of memory, and quietly cut off any prompt that was too long for it.
The subscription had a couple of surprises of its own. If I didn't give it my own system prompt, it loaded all of Claude Code's instructions: 6,503 tokens to answer a three-word question, compared to 398 with a short prompt of mine. And there was no way to limit how long a reply could be. When I asked it in the prompt to keep answers short, they actually got longer.
I dropped the local model and, to stop spending credits, moved questions over to the subscription, accepting answers that took around 25 seconds. That same day I found out where the time was actually going. claude -p switches on "extended thinking" by default, so the model was quietly writing thousands of hidden tokens to itself before every short answer. Once I turned that off, a standup went from 29.3 seconds to 5.6, and from 3,062 tokens to 279, with the same content. The subscription was never really the slow part.
Lesson: measure tokens, not just seconds. The slow part usually wasn't the one I suspected.
2. I was paying for the same work over and over
Every time Praxis calls a model, it writes a line to a usage log: which skill made the call, which model answered, how many tokens went in and out, and whether it was paid through the API or covered by my subscription. The cost is worked out from those numbers whenever I look at the log.
In late August I finally sat down and read it properly. 330 of 472 calls, about 85% of what I was spending, came from one skill, context-sync, re-reading projects that hadn't changed at all since the last time it ran. Every time the dashboard opened, it paid to process my quiet projects all over again.
The odd part is that Praxis already wrote down when each project was last synced. Nothing ever checked it. Now context-sync skips any project with no new log lines since its last sync, so quiet projects no longer cost anything.
Lesson: log every model call from day one. I would never have guessed where the money was going.
3. Tests that passed while things were broken
More than once, a green test suite told me less than I thought it did.
- Every skill was being tested against the wrong code. In July the suite had 97 failures. Each skill's tests imported a file called
run.py, and Python remembers modules by name, so whichever skill loaded first won and nearly every other skill was tested against its code. Once I fixed that, 220 tests passed. - Tests that went bad with the calendar. Some tests compared hard-coded dates to today's date. They passed the day I wrote them and started failing a few weeks later. Freezing the clock inside the tests fixed it.
- A bug no test could catch. Pressing Ctrl-C in the script that starts my dev servers didn't stop them. Every automated test passed. I only found it by actually using it in a terminal.
- Failures that weren't real. To check that a test really catches a bug, I'd break the code on purpose. Sometimes Python ran an old cached copy instead, and the test failed for the wrong reason. Now I clear the cache before those checks.
Lesson: when tests pass on the first try, break something on purpose and make sure they fail.
4. Real data broke my plans
I spec every feature before I build it, and some of those specs were still wrong the first time they touched real data:
- The search for unassigned GitHub work, written exactly the way I'd planned it, matched 134,036,380 open issues across all of GitHub on its first live run. GitHub search can't be limited to "repos I have access to", so the query had to list every account and organization by name.
- For voice, I assumed that running speech recognition locally with a bigger model would get technical words like project names right. It didn't: both models I tried got 0 out of 2. Giving the recognizer a short list of words to expect got 2 out of 2.
Lesson: try the real thing early. A spec is a guess until it meets real data.
What ties them together
None of these were about which model I picked or how I worded a prompt. They were about measuring before guessing and testing against the real thing.
Next in this series: From idea to merged PR in a day, a feature built end to end on top of these lessons.