Share this post
AI didn't only break how we code. It also broke how we interview for AI engineering roles.
Try a quick experiment: ask five engineering leaders whether candidates should use AI in interviews. You'll get five different answers, each delivered with total confidence.
Don't take my word for it. Look at what the biggest names in the industry are actually doing:
- Canva: "In fact, we insist." Backend, frontend, and ML candidates are expected to use Copilot, Cursor, or Claude, and the questions were redesigned to be "more complex, ambiguous, and realistic" (canva.dev, June 2025).
- Meta: added a 60-minute AI-enabled coding round in CoderPad with a built-in AI assistant candidates are expected to use (late 2025, expanding through 2026).
- Shopify: Cursor and Copilot allowed; hiring managers grade how you drive the architecture, explain your choices, and verify the AI's work.
- xAI: the take-home openly assumes you'll use Cursor or Claude Code, and the loop reportedly has zero LeetCode-style problems.
- Google: Pichai says they're reintroducing at least one in-person round "to make sure the fundamentals are there" (June 2025).
- Anthropic: live interviews are "all you," no AI assistance unless they say otherwise.
- Google DeepMind: "Please don't use any AI tools during live interviews."
- Amazon: AI use in interviews can reportedly disqualify you as an "unfair advantage" (per Business Insider).
Insist, expect, allow, assume, restrict, forbid, disqualify. Everyone is doing something different, because nobody actually knows.
And here's my favorite pair of numbers on the whole subject. In a January 2026 Karat survey of 400 engineering leaders, 71% said AI is making technical skills harder to assess. In the same report, 62% of organizations still prohibit AI use in technical interviews. Read that again. The industry admits the signal broke, then bans the tool that broke it.
I've been living this from both sides of the table. I've hired, I've interviewed, I've compared notes with a lot of friends doing the same. So instead of adding another hot take to the pile, I tried to curate something practical: a set of questions, exercises, and one big build that actually tell you whether someone can engineer with AI. This post is the written companion to the video, same arc, more receipts.
▶ Video walkthrough coming soon — I walk through every question and exercise on YouTube; the link lands here the moment it's live.
And everything below lives in an open-source repo: github.com/DoryZi/ai-engineer-interviews. Browse it, steal it, spin it for your own stack, or best of all, send PRs so it grows.
Why the Old Interview Broke
Every classic interview format measures the same thing: can you produce code. LeetCode measures whether you can recall algorithms under pressure. Take-homes measure whether you can build a small thing over a weekend. And production just went free. An agent passes any take-home. AI already writes 42% of committed code, per Sonar's 2026 developer survey, and that number is climbing. That's the industry average; from my own experience, on my own projects, it's closer to 99%.
The receipt that should end the argument comes from Anthropic itself. Their engineering blog admitted that by May 2025, over 50% of their candidates would have been better off delegating the AI-allowed take-home to Claude Code entirely. Sit with that. The company that makes Claude cannot keep its own interview ahead of its own model. If they can't, your take-home from 2023 definitely isn't holding the line.
So what's scarce, if production isn't? Not just judgment. It's the builder mentality. As Paul Graham put it in early 2026: "When anyone can make anything, the big differentiator is what you choose to make." It's accountability: what are we shipping, and how are we shipping it. We keep hearing that the bottleneck moved, so let's name where it moved to: PRs, reviews, verification, understanding the intent behind the work and being accountable for it. That's the actual job now, and it's the thing almost no interview measures.
The kit is built around one rule that fixes most of this at once:
The candidate uses AI in every exercise, and you grade the interaction.
This rule also dissolves the cheating problem instead of policing it. Interview cheating with AI more than doubled in six months of 2025, from 15% to 35% per interview platform Fabric's own data. But cheating only works on questions whose answers are producible. When AI use is expected and the interaction itself is the grade, there's no secret channel left. You can't cheat a driving test by driving.
The Nine Questions
The first layer is verbal questions, and no, questions are not the innovation here. We've all used them on phone screens and on-sites forever. Don't you just love "where do you see yourself in five years?" The format stays; the target changes. These questions are built to expose a different mode of thinking: whether the candidate actually adjusted their approach and shifted from the old school of engineering to the new one, or is still narrating 2022 with new vocabulary. They're cheap to run (no setup, no artifacts), they ask for practical knowledge, and each one opens a deep dive into how agents actually work and how deep the candidate's technical understanding of them really goes.
Those are the lines of questioning I try to come up with. Here are four examples out of the full bank (the rest lives in the repo), and what each one actually catches.
"What does your CLAUDE.md or AGENTS.md contain today?" Then follow up: how long should it be, and what have you deleted from it? The config file is a fingerprint of real usage. People who work with agents daily have opinions written in scar tissue: behavioral rules like "plan before implementing" and "verify before claiming done," a real position on length (every line is context the model reads on every turn), and actual deletions, because rules stop earning their keep as models improve. People who don't work with agents have a blank file or a pasted mega-template they can't explain line by line. If the answer is "best practices and clean code," that's vapor.
"What guardrails make AI code safe to ship?" Here's the landmine: if the candidate thinks their CLAUDE.md is a guardrail, that's a problem. Config files are suggestions the model can ignore. Real answers involve deterministic gates the model can't talk its way past: pre-commit hooks, semgrep rules, type checks, verification gates, CI as one layer rather than the whole answer. Bonus signal: they've turned a caught failure into a new permanent rule. "I review it carefully" is a wish, not a gate.
"How did AI change how you approach engineering, and where does your time actually go now?" The naive answer is a story about generating code faster: prompt, accept, repeat. The experienced answer describes a reallocation. The majority of the time now goes to planning, aligning with the model, setting up the loops correctly, and making sure verification actually happens. Strong candidates talk about what they automate, how they classify work, and how that's what makes them faster. How we review PRs now is a perfect example of that shift; I wrote up exactly that in reviewing AI PRs at scale. Bonus points if they tell you how planning itself changed, because it did.
"Walk me through using AI to debug a production issue." This is the reality check: observability, patterns, and intent, with real stakes attached. Do they connect the agent to actual evidence (MCP tools, logs, error trackers)? Do they know what those MCP tools cost in context, or do they just let the AI reach again and again with no budget? It's also the natural place to ask about agent failure modes: which ones have they actually seen? And "when did the agent last fail you?" is a really nice segue into the follow-up that cuts deepest: an agent-written change you approved broke production. Walk me through the postmortem. What was your failure in it? Strong candidates own the approval as the failure point and answer with a new gate, not a resolution to be more careful. If the villain of the story is the model, that is the answer.
That's four of nine. The others live in the repo, each with answer keys, listen-fors, and red flags, and I walk through all nine in the video (coming soon).
One defense makes all nine cheat-proof: whatever they claim, drill two levels down. Lived practice has a repo, a bug, a rule with a reason behind it. Fabricated practice collapses on the second level.
Three Live Exercises
After the questions, I move to small, contained exercises. They're easy to run under time constraints: a tech screen, or that one on-site hour where you want to see how a person actually operates. The candidate gets full agent access in every one, because that's how I work today (correction: how we all work today), and there's no point simulating a reality nobody ships in. The critical part is what I call the paste test: if you can copy the exercise into a frontier model, today or in a few months, and get the right answer back, the exercise is dead. So you have to get creative and test judgment, accountability, and orchestration, because the code itself is now the minimal, least interesting part; the exercise is everything around it. These three survive because the ground truth lives where the agent can't reach it.
Intent Outside the Repo. A ticket with five acceptance criteria and a PR that satisfies all five. The code is right, the tests pass, and it's still wrong, because it filters on the wrong financial column and the information that reveals it was never written down. It's in your head, and you answer only what you're asked. You can also play this as a mock conversation with a business owner. It tests whether "review" means matches the spec or does the job.
Which One Ships? The same feature built three times: 29 lines, 218 lines, 47 lines. Identical behavior, one shared test suite, nothing hidden. Which one ships? Kill the other two, out loud, with reasons. Anyone can generate three options now; the scarce skill is killing two and defending the kill. It tests judgment about carrying cost, and whether they know the deciding facts live in the product, not the code.
The Big PR. 2,400 lines across 29 files, all green, nothing planted, in a slot far too short to read it. That's the point. The agent can find things fine, but it cannot rank them, because which change can hurt this business isn't in the diff. Watch them work backwards, prioritize, and talk about blast radius. It tests allocation, not reading.
The Long Task: Think Big and Audacious
The final layer is the closest thing to the actual job: give them something genuinely big. Run it as a four-to-five-hour onsite working with an agent, or as a take-home.
Whiteboard the architecture first if you want to see design thinking, or skip straight to building. Then layer it: a CRM, then add deal pipelines. An order and inventory system with reservations that expire. A help desk with SLAs. Or, even better, have them build your own app, the whole thing, and present it back. That one is a triple win: the presentation tests communication skills, you get fresh eyes and a new perspective on the thing you build every day, and you see exactly how much a person can actually do. Let them do everything, including deploy, on a trial cloud account if you like.
The trick, which the repo brief leans into, is that the brief is deliberately bigger than the slot. What they cut is the grade. And when they're done, a hidden suite runs against whatever shipped. Watching what a person can actually push through and deliver with AI in a few hours is the single best indication I've found of how good they are at this job.
What You're Actually Hiring For
Nine questions, three exercises, and one big build. You don't have to use any of them as-is. Run the questions alone, tailor the exercises to your stack, mix and match.
The underlying shift is what matters: the bottleneck moved away from creating code, onto PRs, reviews, verification, and intent, and the interviews have to move with it. What you're screening for now is the builder mentality, accountability, and judgment: people who have taste, who care about what you ship and how you ship it, who take ownership of what the agent produces, and who chase the business intent behind the ticket instead of the letter of it.
And an honest timestamp: this is where we are in mid-2026. It might change some more, and probably will. I'm going to keep running these in real loops and report back on what actually predicted performance. That's the build-in-public deal.
Key Takeaways
- Nobody agrees on AI in interviews. Canva insists, Meta expects, Shopify allows, Anthropic forbids. And per Karat, 71% of leaders say AI broke skill assessment while 62% still ban it in interviews.
- Production went free. Anthropic's own take-home lost to its own model: by May 2025, most candidates would've done better delegating it to Claude Code entirely.
- One rule fixes most of it: the candidate uses AI in every exercise, and you grade the interaction. It also dissolves cheating by design.
- Ask about practice, then drill two levels down. CLAUDE.md contents, guardrails, where your time goes now, production postmortems. Fabricated practice collapses on the second level.
- Ground truth must live where the agent can't reach it: the interviewer's head, tradeoffs with no right answer, or a brief bigger than the slot.
Steal the kit, run it in a real loop this week, and tell me what you find: github.com/DoryZi/ai-engineer-interviews. PRs welcome, especially reports from real interviews.
Stop wasting time on AI. I run practical experiments — real lessons you can use tomorrow, biweekly.
Subscribe to AI Will Replace Your Engineers
Watch, read, and build
- ▶ Video walkthrough: coming soon — every question and exercise, walked through live.
- 🔧 Grab the interview kit: questions with answer keys, three scripted exercises, and the long-task briefs. MIT licensed.
- 📖 Reviewing AI PRs at Scale: the deep-dive behind the review questions.
- 📬 Subscribe to AI Will Replace Your Engineers: real experiments, biweekly.
- 🔗 More at doryzidon.com.
