Share this post
At work, I've been asked to build things using Kanban and sprint planning. And it made me wonder: is any of this still relevant?
Sprints, story points, T-shirt sizes, Kanban boards. Planning has been the ritual of software teams for as long as I've been in this industry. Two-week sprints, epics broken into features broken into stories, boards full of sticky notes. A lot of it grew up around one fact: writing code was slow and expensive, so you had better know what you were building before anyone started typing.
That fact is gone. An agent will build you a working app in under an hour. So is sprint planning even relevant anymore, with models this capable? Does breaking work into stories still help, or is it ceremony we carried over from the old world? And if some planning does help, which kind? Those questions are why I started this experiment.
I could argue about it, or I could measure it. So I took one app idea, had two AI agents, Claude Code and Codex, each turn it into five different plans, from a bare goal all the way to a full sprint, and had them build every plan five times. Fifty builds. Here's what came out, what surprised me, and what I'm testing next.
The test: five plans, two agents, fifty builds
It started with my own bank account. I wanted to see where my money went over a few periods, so I exported my statements and told an agent to show me. No plan, no stories, just a conversation. I iterated a few times until the breakdown actually made sense. Pure vibe coding, and it worked.
Then the obvious question: what if I planned the rebuild properly, the way a team would, more carefully and more securely? Which planning method gets you the best result?
So the idea became a short brief, and each agent turned it into five plans. Claude's five looked like this:
- Goal only. The brief and nothing else.
- Epics. The brief plus a handful of high-level user outcomes.
- Stories. A flat list of user stories.
- Epics, features and stories. The full hierarchy.
- Sprint. The hierarchy plus story points and a two-week commitment.
Codex's five climbed the same ladder, each in its own way. In Claude's plans, only the sprint is timeboxed. The other four are closer to Kanban-style flow: no sprint, no commitment, just work to pull. Kanban wasn't its own plan; it's the shape the other four share.
Two agents ran it, Claude Code and Codex. Same idea, written up separately for each. Every plan was built five times, from scratch, on made-up bank statements rather than my real data. Twenty-five builds per agent, fifty in total. And each run was scored by a hidden exam the builders never saw, written by the other AI: Codex examined Claude's builds, Claude examined Codex's.
Kickoff: Codex sprinted, Claude measured
This is where it got interesting. I kicked both runs off, and the two agents went in completely different directions.
Codex just started building. Claude stopped, went off for about three hours, and wrote a testing harness before it built a single app.
In an experiment about planning, the agents had their own planning styles. Codex sprinted. Claude measured first.
When I checked back, Codex was stuck: its builds kept hitting the time limit. Claude was ready, and wanted to run one build first and check the loop with me before doing all 25. That was nice.
Codex got going again with more time per build and finished. Both got there in the end, just by very different roads.
So which plan won?
None of them.
Claude's report said it best: how much you plan barely changes whether the app works. It changes what gets built. All 25 of Claude's apps started up, and four of the five plans landed within a few points of each other, 82 to 86 percent.
Each agent was graded on its own test, so compare the direction inside each column, not Claude against Codex.
Then came the sprint, and this is where it got funny. And very human.
I asked for a sprint, whatever that means to an AI, and it went full human on me. Claude found its own way to represent the timebox: a hard limit on what fits, even on its own tasks. Then it simulated exactly what happens in a real sprint. Cut the scope, under-delivered. Budgets got pushed out of "this sprint", so three of the five builds were budget apps with no budgets, and the sprint scored 67. The brief asked for budgets, and the plan said they could wait. Sounds familiar? ;)
Now isn't that funny? An AI and a human team, same outcome.
Codex landed in the same place: 25 builds, no winning plan. Its scoring didn't get the same scrutiny as Claude's, so I wouldn't lean on the numbers, but the headline was the same.
And the AI testing Claude's apps? It was doing their work for them, math the apps never did, so they passed anyway. Once I fixed that, 29 passes disappeared, and every number above is after the fix. Check the checker.
Full breakdown, same app every time
Here's the part I did find interesting. In Claude's run, breaking it all the way down gave me the same app almost every time.
The full plan, epics broken into features broken into stories, was rock steady: all five builds scored about 85 to 88 percent. Give it just the goal and the scores swung from 75 to 94. The sprint swung from 52 to 92.
So the full breakdown buys you consistency. In Claude's run the detailed plans also built smaller and faster. But they didn't really get better. And you lose something along the way: the agent's wider system thinking, the shared context, its ability to come up with features and make the thing really nice.
Same app every time is great when you need the same app every time. It's not the same as the right app.
They mostly did well. None of them worked for me.
Most of these apps did well on the tests. And for what I actually needed, every single one was unusable.
None of them really thought about statements. How I bring them in, how I reconcile them, how I take one back out when it's wrong, along with its data. One of Claude's builds, from the full epics, features and stories plan, didn't even have an upload button. That's the core of the whole app. It should have been in every plan.
Now, that's probably on me. I never spelled that workflow out properly in the requirements. But that's exactly the point: it shows you how much planning correctly matters. Not stories versus sprints. Getting the right things into the plan in the first place.
Codex, for what it's worth, designed the much nicer interface, at least to my eye. Its apps just looked better.
Left: a Claude Code build. Right: a Codex build. All bank data is made up.
And the app that actually worked for me from the start? The one I vibe coded. No plan, just me and the agent going back and forth until it showed me what I wanted to see. The app looks better because it's what you asked for.
When you can't plan everything in detail, and you don't have tests you can truly trust, the best verification is you. A human in the loop, looking at the thing and saying "no, not that, this."
Same thing at work
This isn't just an AI thing. I've seen it at work too.
I spent days designing a model service that routes requests to LLMs. Epics, features, stories, all of it, and then we built it out meticulously. And still, I had to vibe code over the UI and make fixes. Then more input came in, and later reviews re-opened the architecture.
It's really hard for people to consume all of that, understand it, and actually tie things down. I spent hours reading everything the AI put out. It's possible, it's just hard, and even then I'm not sure you end up with the product you wanted.
On the flip side, my cousin, who's a VP, took a transcript of a conversation with HR about how to run people's reviews, uploaded a bunch of Excel files, and wrote one simple prompt. Bang, one shot, and he got exactly what he wanted. Now the whole company uses it.
So there's a lot to think about in how we build products these days, and what we actually plan.
What I learned
So where does that leave planning? Here's my take, for now.
Vibe code to get aligned. When you don't know exactly what you want yet, and most of the time you don't, vibe coding is the fastest way to find out. Build it, look at it, push back, build it again. That's how my little bank app went from "show me where the money went" to something that actually worked for me.
Use stories and features when you need consistency. If something has to come out the same way every time, break it down. The full epics, features and stories plan was the steadiest one in Claude's run.
Keep the rest loose. Not every part of an app deserves a detailed plan. Plenty of it can be vibe coded and fixed as you go.
Rough guidelines, not slam dunks
This experiment didn't give me a super conclusive answer. What it gave me is some guidelines, a few things that work, and a handful of ideas and concepts. They're rough, I know. That's part of learning. I'm not promising any of it is a slam dunk.
On the flip side, it opened a lot of doors. It generated a pile of new ideas and experiments I can run to figure out how to plan better and how to get better alignment from AI. So I'm going to keep going down that path, and compare it to the things that already work really well for me.
Next: planning by module
One thing that has worked really well for me is how I plan modules. I define a module by its intent, its boundaries and its user flows, using plan-html, the planning tool I built and wrote about in Stop Approving Plans You Didn't Read (there's a video too). It's been really helpful for designing certain software components.
It's a lot more abstract than detailed planning. No story points, no sprint. Just what the module is for, where it stops, and how people move through it.
So that's the next experiment: plan by module that way, and put it head to head with the planning methods from this one, and with the planning skills people are using. Let's see what that gives me.
Watch the full video for the whole story. The experiment page has the details, and I'll share what comes next in my newsletter.
Related:
- Casino Code: AI Pair Programming with Luck and Hope, on why the same prompt gives you a different app every time
- Workflows That Ship: 80 Experiments in Agent Orchestration (video), on plans, ground truth and verification
- The 10x AI Engineer (video), on why understanding, not output, is the bottleneck
