A few weeks ago I demoed a small internal assistant to my team — a chat box over our runbooks and incident notes, Spring Boot service in front, retrieval over docs we already own, pinned model version behind it. People liked it. Then one senior asked the question that ended the celebration early: “Is it registered?” He did not mean the demo URL. He meant the model inventory.

So I spent the next two weeks doing something I had never done for any CRUD service: filing my LLM app with the model-risk board. Forms, diagrams, eval sheets, a meeting where serious people asked unserious-sounding questions like “what stops it from making things up about a production database?” I walked in annoyed at the paperwork. I walked out thinking every team building LLM apps in a bank should do this on purpose.

Our Spring Boot services build on GitHub Actions ten, fifteen times a day, and for a month the pipeline had a habit I hated: a different integration test failed every other run, somebody hit re-run failed jobs, it went green, everybody moved on. The suite was green on paper and nobody believed a word it said.

Last month it bit us properly. An OrderStatusIT test failed on my PR, passed on rerun, failed again on a teammate’s completely unrelated PR, and blocked three merges in one afternoon. The failure had nothing to do with any of our changes. That evening I stopped asking “whose change broke this” and started asking the better question: why does one unreliable test get to hold the whole team hostage?

During one recent sprint I ran a small experiment. Instead of assigning a routine bug ticket to a teammate, I assigned it to our coding agent. Twenty minutes later, while I was still in standup, a draft pull request opened itself: commits pushed, checks running, a tidy checklist of what it intended to do. Nobody typed anything. The backlog had effectively started opening its own pull requests.

I have written before about reading AI-generated diffs and about repo instructions for coding agents, but those were about the inner loop, me at the keyboard with an agent beside me. Background agents are a different animal. They change not how I code, but how work flows through the whole team. Somewhere along the way my issue tracker quietly turned into a job queue.

It is hiring season again, and I have been reading a stack of CVs that all say some version of the same thing: “proficient in AI-assisted development”. In interviews, candidates live-code impressively fast. The autocomplete hums, the boilerplate appears, the endpoint compiles. Then I ask them to trace a null pointer through three layers of a Spring service they did not write, and the room goes quiet.

I am not complaining about the tools. I use them every day and I am not giving them up. But somewhere between the demos and the hiring loops, we seem to have confused typing code with understanding systems. Those were never the same skill, and the gap between them is now the whole game.

The numbers back up the unease. Last summer METR ran a randomized trial with sixteen experienced open-source developers on their own mature repositories, mostly using Cursor with Claude 3.5 and 3.7 Sonnet. Developers expected a 24% speedup. They took 19% longer with AI enabled — and afterwards still believed the tools had made them 20% faster. They accepted less than 44% of the generated code untouched. And in last year’s Stack Overflow survey of over 49,000 developers, the top frustration, cited by 66%, was “AI solutions that are almost right, but not quite”, with 45% saying debugging AI-generated code takes more time. Usage keeps climbing while trust keeps falling.

The first planning week of the year is when everybody is brave. Roadmaps are fresh, the release calendar has that new-calendar smell, and someone always puts “clean up the user table” on the board like it is a half-day task. Renaming a column sounds trivial. It is one line of SQL. What could it possibly do?

Quite a lot, as I learned on an otherwise ordinary release train. The rename itself took milliseconds. The rolling deploy around it took down nothing and broke something subtler: for about ten minutes, old pods and new pods were serving traffic against the same database, disagreeing about what a column was called. Nobody paged. The errors just quietly piled up in the logs until support started forwarding screenshots. That morning taught me the rule I have followed ever since: never change the schema in one step when two versions of your code will ever meet it.

Not long ago, I watched a teammate’s agent loop burn through $2,300 in API calls overnight. The task? Classifying support tickets into three buckets. The model? GPT-4o. Every single ticket — even the “reset my password” ones — got the full frontier treatment.

That’s the trap. When you first get API access to a reasoning model, it feels like a superpower. You start reaching for it everywhere. Why wouldn’t you? It’s smarter, it reasons, it gets the hard stuff right. But somewhere between the demo and the production bill, the economics invert. You’re paying premium prices for commodity work.