OpenAI released GPT-6 Astra yesterday (September 3), and president Greg Brockman called it a “generational leap” — reports from the briefing have him closing with “Welcome to the AGI era.” I have been doing this long enough to discount launch-day adjectives, so let me skip the AGI debate and tell you what actually caught my attention as someone who builds AI systems inside a bank: the rollout is gated, the cyber capabilities are split into a restricted track, and OpenAI’s own chief scientist spent part of the launch call explaining why this model is harder to monitor than the last one.

That combination matters more to practitioners than the headlines.

Last quarter, a prompt tweak landed on a Friday. By Monday, 8% of our collateral-margin advisory bot’s responses were hallucinating policy clauses that had not existed since the 2021 Basel III update. The change looked harmless in review — just “make the tone more helpful” — but the model took that as permission to improvise. Nobody caught it until a relationship manager flagged a client email. The rollback took twenty minutes. The post-mortem took three days.

That incident is why I no longer ship agent changes without an eval gate. Not “we should really add evals.” Not “evals are on the roadmap.” The gate is the only way I sleep at night.

It was a regular Tuesday morning when our settlement-status service fell over, and nothing about the traffic looked unusual at first. Volumes were maybe thirty percent above normal, the kind of spike we had survived before. But an upstream job had started retrying aggressively on timeouts, so every slow response came back as two more requests, and within twenty minutes the Tomcat thread pool was fully occupied with calls waiting on a downstream that was itself waiting. Health checks timed out. Kubernetes restarted pods that were not broken, just busy. The restart dropped the in-flight work, the retries noticed, and the whole thing repeated.

I spent that afternoon staring at a dashboard showing a queue depth climbing like a staircase and thinking about how polite our service had been through all of it. It accepted every request. It queued every request. And then it served none of them well. A bouncer who lets everyone into a packed room is not being kind. He is starting a stampede.

The other day, a colleague asked our internal coding agent to “refactor the fee calculation module like we discussed last month.” The agent cheerfully rewrote the whole thing — except we’d never discussed it. It had hallucinated a conversation from thin air, stitched together fragments from three unrelated PR reviews, and produced code that compiled but violated a regulatory constraint we’d explicitly documented six months ago.

That incident crystallized something I’ve been wrestling with: memory in agent systems is not a solved problem. It’s the new capacity planning.

Recently, one of our internal chatbots gave a confidently wrong answer about a collateral margin rule, and the incident thread quietly died with the words “looks like a hallucination, closing as won’t fix”. I have started hearing that phrase the way I used to hear “it’s flaky”. Technically true, completely useless, and usually a polite way of saying nobody looked.

Six years ago I wrote about correlation IDs because we refused to accept “it’s flaky” as a root cause for a request crossing five .NET services. We widened log windows, followed IDs, found the rolled-back transaction hiding in the noise. The same fight has come back, except this time the misbehaving component bills by the token and improvises its own control flow. The old excuse was that you simply could not see inside these systems. That excuse is expiring, and this post is about what replaced it.

The other week a flaky integration test in one of our Spring Boot services started failing on the build, and I did what I do these days: I pasted the stack trace into the coding agent and went to make coffee. By the time I was back, the fix was in — a test ordering dependency, one shared fixture leaking state between two test classes. Green build. Merged by lunch.

And here is the embarrassing part: if you asked me today exactly which fixture leaked what, I would have to look it up. The agent found the bug. I just carried the coffee.