One Saturday morning I opened Task Manager before I opened my mail. There it was: a model server holding nine-odd gigabytes of RAM, alive since Friday night, with a couple of things on the machine quietly pointing HTTP requests at it. A summarisation job had run overnight against my local model, on schedule, nobody watching. Somewhere in the past year or so, without any ceremony, my laptop stopped being a machine that occasionally plays with AI and became a small server that other software depends on.

I did not plan this. When I first ran models locally back in late 2023, it was a toy: Mistral 7B in a terminal, mostly to see whether it could hold a conversation at all. Fun toy. But a toy.

Somewhere in our account there is an S3 bucket that nobody owns and everybody uses. Our Spring Boot services dump daily statements, audit exports, and application logs into it, versioning switched on years ago by someone careful, lifecycle rules configured by nobody. Last quarter I opened Storage Lens out of curiosity and just stared. One prefix held three years of files nobody had opened since the week they were written, all sitting in Standard storage, all billed every single month.

Storing data feels free because the per-GB number is tiny. Keeping it there forever, and then moving it around carelessly, is where the real money goes.

A few weeks ago I demoed a small internal assistant to my team — a chat box over our runbooks and incident notes, Spring Boot service in front, retrieval over docs we already own, pinned model version behind it. People liked it. Then one senior asked the question that ended the celebration early: “Is it registered?” He did not mean the demo URL. He meant the model inventory.

So I spent the next two weeks doing something I had never done for any CRUD service: filing my LLM app with the model-risk board. Forms, diagrams, eval sheets, a meeting where serious people asked unserious-sounding questions like “what stops it from making things up about a production database?” I walked in annoyed at the paperwork. I walked out thinking every team building LLM apps in a bank should do this on purpose.

Our Spring Boot services build on GitHub Actions ten, fifteen times a day, and for a month the pipeline had a habit I hated: a different integration test failed every other run, somebody hit re-run failed jobs, it went green, everybody moved on. The suite was green on paper and nobody believed a word it said.

Last month it bit us properly. An OrderStatusIT test failed on my PR, passed on rerun, failed again on a teammate’s completely unrelated PR, and blocked three merges in one afternoon. The failure had nothing to do with any of our changes. That evening I stopped asking “whose change broke this” and started asking the better question: why does one unreliable test get to hold the whole team hostage?

During one recent sprint I ran a small experiment. Instead of assigning a routine bug ticket to a teammate, I assigned it to our coding agent. Twenty minutes later, while I was still in standup, a draft pull request opened itself: commits pushed, checks running, a tidy checklist of what it intended to do. Nobody typed anything. The backlog had effectively started opening its own pull requests.

I have written before about reading AI-generated diffs and about repo instructions for coding agents, but those were about the inner loop, me at the keyboard with an agent beside me. Background agents are a different animal. They change not how I code, but how work flows through the whole team. Somewhere along the way my issue tracker quietly turned into a job queue.

It is hiring season again, and I have been reading a stack of CVs that all say some version of the same thing: “proficient in AI-assisted development”. In interviews, candidates live-code impressively fast. The autocomplete hums, the boilerplate appears, the endpoint compiles. Then I ask them to trace a null pointer through three layers of a Spring service they did not write, and the room goes quiet.

I am not complaining about the tools. I use them every day and I am not giving them up. But somewhere between the demos and the hiring loops, we seem to have confused typing code with understanding systems. Those were never the same skill, and the gap between them is now the whole game.

The numbers back up the unease. Last summer METR ran a randomized trial with sixteen experienced open-source developers on their own mature repositories, mostly using Cursor with Claude 3.5 and 3.7 Sonnet. Developers expected a 24% speedup. They took 19% longer with AI enabled — and afterwards still believed the tools had made them 20% faster. They accepted less than 44% of the generated code untouched. And in last year’s Stack Overflow survey of over 49,000 developers, the top frustration, cited by 66%, was “AI solutions that are almost right, but not quite”, with 45% saying debugging AI-generated code takes more time. Usage keeps climbing while trust keeps falling.