Skip to content

Objections and honest answers

This guide is for AI champions, leads, and skeptics themselves, and takes about 40 minutes to read. It pairs with the playbook and Session 0 of the training series.

Last checked November 8, 2026.

TL;DR - Most objections to AI-first engineering are partly right. Start every answer with what's true, because skeptical engineers trust people who concede real limits. - The best answer to an objection is usually an experiment on your own code, not a study or a launch post. Each entry below ends with one. - The strongest objections are about verification, review load, security, and learning. Ask which tool, version, and task each objection describes. - Some questions don't have good answers yet. Say so. The last section lists them.

How to use this guide

  • Demo instead of arguing. Find the objection, then run its "Try this" on a real task with the person who raised it.
  • Concede first. Each entry opens with "What's true," so read that part out loud.
  • Every claim links to the module section that sources it, so a skeptic can check it.
  • Keep an objections log. When someone raises one that isn't here, or the answer here doesn't land, write it down and bring it to the monthly radar review.
  • Respect "no." Some people will stay at one supervised agent, or use agents only for chores. That's AI-first enough if their work is reviewed and their skills stay sharp.

Index

# Objection Category
1 "It hallucinates. I can't trust anything it says." Quality and trust
2 "I tried it in 2024 and it was useless." Quality and trust
3 "It writes unmaintainable code." Quality and trust
4 "It cheats on tests, or writes tests that test nothing." Quality and trust
5 "Our codebase is too big and too weird for it." Quality and trust
6 "It got worse last week. You can't build on something that changes under you." Quality and trust
7 "Reviewing its code is slower than writing it myself." Speed
8 "The METR study says it makes you slower." Speed
9 "The productivity claims are vendor marketing." Speed
10 "The benchmarks are fake." Speed
11 "Fully autonomous agents are hype." Autonomy
12 "I don't want to be a prompt engineer. I like writing code." Autonomy
13 "It'll take our jobs." People
14 "Juniors won't learn anything." People
15 "I'll lose my own skills." People
16 "This is just a way to squeeze more output out of us." People
17 "Security will never approve it." Security, privacy, legal
18 "Our code will end up training their models." Security, privacy, legal
19 "We'll get sued over IP and licensing." Security, privacy, legal
20 "It's too expensive." Cost and tools
21 "Which tool? It changes every week." Cost and tools
22 "We can't measure the ROI, so we can't justify it." From managers
23 "Why not just mandate it and set a target?" From managers
24 "What about the environmental cost?" Other

Quality and trust

1. "It hallucinates. I can't trust anything it says."

What's true. It does hallucinate. A language model predicts plausible text and has no built-in notion of truth. In code that shows up as invented APIs, wrong library versions, confident descriptions of files it never opened, and package names that don't exist (module 01). In one study, at least 5.2% of packages suggested by commercial models and 21.7% by open models didn't exist, and attackers register those names (module 11).

A coding agent can run the compiler, the tests, and the type checker, and read the installed package's source. An invented method fails the build in seconds, as long as the agent is told to run the build. Treat every claim as a hypothesis until a compiler, test, document, or person confirms it (module 01). In Stack Overflow's 2026 survey, 48% trust AI output only when it's easy to verify (module 05), so make verification easy.

Be careful where no check catches the error, as with new dependency names, explanations of code, and claims about files. Verify those by hand.

Try this. Ask the agent to explain a module you know well. Require a file path and line number for every claim. Check five claims against the code.

2. "I tried it in 2024 and it was useless."

What's true. For a lot of 2024 work, it was. Most people then used autocomplete or pasted code into a chat window. Even early-2025 agents slowed experienced developers down in METR's study (module 08). Your judgment was reasonable for the tool you tried.

It's a different product now. A modern agent reads your repository, runs your tests, and iterates until they pass. METR measures the length of task (in skilled-human time) that an agent completes half the time. That was about 50 minutes in March 2025, and METR put public frontier models at around 12 hours by early 2026, with a wide confidence interval (module 05). But the 80% horizon is much shorter, about 1.5 hours, and METR's tasks are cleaner than most real tickets. Harnesses also picked up plan modes, hooks, sandboxes, and worktrees in that time (module 06).

It still gets you most of the way quickly, and the last stretch (edge cases, error handling, security) is still yours (module 08).

Try this. Do day 2 of the engineer path on a small real bug with a reproducible failure. Have the agent write a failing test first, then fix it, and read the diff line by line.

3. "It writes unmaintainable code."

What's true. Unsupervised, it often does. The recognizable patterns are over-engineering, abstractions with one implementation, swallowed errors, dead code, and a new helper that duplicates an existing one (module 10). Users complained that one 2026 frontier model over-engineered beyond what was asked (module 05). A vendor study found about 1.7 times as many issues in PRs with signs of AI co-authorship, with real caveats about its labels and its author's commercial interest (module 10).

Maintainability is something your review enforces. Hold AI-written code to the same bar as human code (playbook review policy), then steer it: - Ask for "the smallest change that satisfies the spec, no new abstractions or dependencies." - Put "use the helpers in X" and "don't copy the patterns in legacy/" in the instruction file. Flagging a misleading pattern that's everywhere in the code is exactly what that file is for (module 07). - In review, ask "what can we delete?" - Don't fix every bot finding. Anthropic warns that doing so leaves "extra abstraction layers and tests for cases that can't happen" (module 10).

Try this. Run the module 10 review checklist over the last five AI-assisted PRs your team merged, and count findings by type to see which line to add to the instruction file.

4. "It cheats on tests, or writes tests that test nothing."

What's true. METR documented frontier models rewriting timers and monkey-patching evaluators, and telling them not to cheat still left reward hacking in 70–95% of runs on the affected tasks (module 08). Agents also write tautological tests that check a mock against itself, and delete or skip failing tests under pressure (module 08).

That's why the test is the contract, and why you defend it (module 08): - Write or approve the tests before implementation. Commit them separately. - Make the agent show the test failing before it passes. - Read the test diff first in every review (git diff --stat main -- tests/). - Say it in the prompt: "Make it pass without modifying the test." - Enforce it with a hook when it matters. A rule in an instruction file is followed "nearly every time"; a hook applies every time. - For unattended work, keep some checks out of the agent's reach.

Try this. Break a function on purpose (mutation testing by hand). If no test fails, the agent's tests, or yours, aren't testing it.

5. "Our codebase is too big and too weird for it."

What's true. Mature codebases with tacit conventions speed up less than greenfield work, and experts on code they wrote gain least (module 05). The real spec of a legacy system lives in people's heads (module 08). Even the inventor of the Ralph loop wrote "There's no way in heck would I use Ralph in an existing code base" (module 08).

Size is less of a problem than it was. Larger context windows help, but attention, cost, and compaction still limit sessions, so agents search instead of reading everything (module 07). The bigger problem is tacit knowledge, which is what a short instruction file is for: the frozen directory that looks live, the helper the team chose, the test that needs Docker (playbook starter file). Nested AGENTS.md files or path-scoped rules cover different areas of a big repository.

The weird legacy corner nobody remembers is often where agents help most, starting read-only. Concede the rest. There will be modules where it isn't worth it yet.

Try this. Pick the part of the codebase people are most afraid of. Ask the agent, read-only, to "trace a request from the entry point to the database, listing files in order and what each does." Check it against the code.

6. "It got worse last week. You can't build on something that changes under you."

What's true. Quality does go up and down. Anthropic published postmortems for weeks of degraded answers in 2025 (three infrastructure bugs) and six weeks of Claude Code complaints in 2026 (harness changes). OpenAI's Codex team published its own degradation investigation. In several cases, users noticed before the labs' evaluations did (module 14).

Treat it like any other dependency that changes. Most documented regressions were in the harness, product, or serving layer, not the model weights. Diagnose before you switch. Check the status page, then your tool's version and settings, then your golden tasks, then whether today's task is just harder (module 14). Pin CLI versions and model snapshots for anything automated, and keep a second tool configured and warm (module 14).

Try this. Build five golden tasks from merged PRs (module 05). Next time someone says "it got dumber," rerun them and compare.

Speed and productivity

7. "Reviewing its code is slower than writing it myself."

What's true. Sometimes it is. Review has become the bottleneck as agents write more code, and AI-written code is tidy and plausible, which makes problems easy to skim past (module 10). In one industrial study, average PR closure time rose from about 6 to over 8 hours after a review bot was added (module 10). Some engineers find supervising agents more tiring than writing the code (module 15).

It changes what you review, and when: - Review the plan first, then the code. The plan is the cheapest place to catch a wrong idea (module 08). - Review the test. If you approved a failing test first, the diff mostly has to make it pass. - Keep diffs small by asking for the smallest change. - Let a bot take the first pass on local bugs, so your attention goes to intent, design, and risk (module 10). Another industrial study saw PR cycle time fall 30.8% (module 10).

Part of the objection holds. For a ten-line change in code you know cold, typing it yourself may well be faster, and AI-first doesn't mean AI-only. The wins are biggest on unfamiliar code, tests, migrations, glue scripts, and debugging from logs (module 08).

Try this. For two weeks, log three tasks a week: one done by hand, two with an agent. Record wall-clock time including review. Compare by task type.

8. "The METR study says it makes you slower."

What's true. It did. In METR's 2025 randomized trial, 16 experienced open-source maintainers took 19% longer on 246 real tasks in their own large repositories when AI was allowed. They expected a 24% speedup and afterward believed they'd been 20% faster (module 05). It's a careful study and deserves to be taken seriously.

Take all of it seriously, including what came after: - The tools were early-2025 agents. METR itself now marks the result out of date. - METR's February 2026 follow-up, with late-2025 tools, estimated tasks about 4% to 18% faster depending on the group, with confidence intervals that include zero. METR calls that an unreliable signal, likely biased toward zero, because 30–50% of developers withheld tasks they didn't want to do without AI (module 08, module 05). - The finding that holds up across several studies is that self-reports need independent checks. METR's 2026 assessment of one AI lab found self-reported individual uplift of about 12x alongside an estimated 2x organizational acceleration; these are different quantities (module 05).

Check perceived speed against measurements, whether it's your perception or someone else's. Experts on familiar code may see small gains, and nobody should promise 10x.

Try this. Measure your own team with a baseline, golden-task timing, and stability metrics, as the playbook describes.

9. "The productivity claims are vendor marketing."

What's true. Many are. The labs write the launch posts, and customer quotes are testimonials, often unverified (module 14). Some widely shared numbers come from vendors measuring their own tools (module 10). Self-reports run ahead of measurement everywhere.

Agree, discount them, and look at independent work instead. Published field studies range from 55.8% faster (a small greenfield task, 2023) to 19% slower (METR, 2025). Three field RCTs covering 4,867 developers found 26% more completed tasks, with bigger gains for less experienced developers (module 05). They disagree because they measured different tasks, tools, people, and outcomes. Taken together, they show real speedups on some work and real slowdowns on other work, and the predictors are task type, skill with the tools, and verification practices. DORA's 2025 report sums it up by saying AI "amplifies what's already there."

Try this. Ignore every number in this guide and run your own eval.

10. "The benchmarks are fake."

What's true. A lot of the criticism is right. Benchmarks leak into training data: in one study, models named the file to fix in up to 76% of SWE-bench Verified tasks from the issue text alone, without seeing the repository. OpenAI stopped reporting that benchmark in 2026. Scores swing with harness and settings. Labs pick favorable comparisons, and some providers tested many private variants on Arena and published only the best (module 05, module 05).

Benchmarks are narrow, age quickly, and can be selectively reported. Benchmarks are good for spotting large generational gaps and trends. They're poor for choosing between two frontier models a few points apart, especially in a harness different from yours. Cross-check with independent evaluators, and decide with your own golden tasks (module 05).

Bryan's take: Opus 5 felt benchmark-maxed to me. It posted great numbers and kind of sucked in daily use, and people didn't like it. Opus 5.5 is the one people really liked.

No public evidence shows that model was trained on benchmark answers. The better-supported story is that benchmarks scored task completion, while users also cared about communication and scope, which benchmarks barely measure (module 05).

Try this. Pick a launch chart and run it through the eight-question checklist in module 05.

Autonomy and the nature of the work

11. "Fully autonomous agents are hype."

What's true. For ordinary product work, mostly yes, today. The famous successes share four things: a strong automated oracle, greenfield code, a harness humans built, and a high tolerance for waste (module 08). A C compiler built by parallel agents cost about $20,000 and its author lists serious limitations. A week-long agent-built browser drew sharp criticism about whether it even built. Steve Yegge describes his own orchestrator as "chaotic and sloppy" and "expensive as hell" (module 08, module 08 on Yegge).

You don't need to believe in full autonomy to get value. Most of it is in supervised agents and parallel supervised agents, where a human reviews every merge. Treat autonomy as a dial and turn it up one notch at a time, only as fast as your automated checks can catch what you stop looking at (module 08).

Bryan's take: I don't believe fully autonomous agentic workflows work reliably today. Maybe I'm just too dumb; I haven't had good results, but some people claim they work.

The strongest counterpoint is that the trend lines are steep, Anthropic reports that Claude writes more than 80% of the code it merges, and some teams build serious software with holdout tests and digital twins of their dependencies (module 08). A fair position is that full autonomy works for well-specified, oracle-rich, greenfield problems, and that boundary is moving.

Try this. Run the "autonomy ladder" exercise in module 08: one spec with hidden tests, three levels of autonomy. Compare the results.

12. "I don't want to be a prompt engineer. I like writing code."

What's true. The work does shift, toward specifying, reviewing, and verifying. Some people find that less satisfying, and some feel they aren't "real" engineers when an agent typed the code (module 15). That's a legitimate feeling, not a skill gap.

First, "prompt engineering" as a bag of tricks mostly aged badly. Claude Code's own docs say capital letters and "IMPORTANT" mostly make the model over-apply a rule, and "think harder" is just text (module 07). What works is what you already do. State the problem precisely, give the reason, define done, and provide a check. The skills that gain value are specification, decomposition, reading code, debugging, testing, and systems thinking (module 15).

Second, you choose what to delegate. Keep writing the parts you enjoy. Hand off the parts you don't: boilerplate, migrations, test scaffolding, log spelunking, ticket drafting.

Try this. For two weeks, use the agent only for the three chores you like least. Then decide.

People and careers

13. "It'll take our jobs."

What's true. Early-career hiring is under real pressure. Payroll data show employment of 22–25-year-olds in AI-exposed occupations, including software developers, about 19% below trend by mid-2026, mostly through fewer hires (module 15). Nobody knows the long-run effect, and anyone who promises you a specific outcome is guessing.

The evidence so far points to AI complementing experienced engineers more than replacing them. Where AI mainly complements workers, employment is flat or rising, especially for experienced ones. The same authors find no broad, economy-wide displacement, and other explanations (the post-2022 correction, interest rates) remain in play. The US Bureau of Labor Statistics still projects 10% growth in software developer jobs from 2025 to 2035; projections can be wrong both ways (module 15).

Engineering still requires someone to decide what to build, catch the "almost right" solution, and answer for the result. Module 15 recommends getting good at the tools, investing in the skills agents can't supply, and helping others adopt well.

Try this. Read the skills table in module 15 and pick one skill to practice deliberately this quarter.

14. "Juniors won't learn anything."

What's true. The evidence supports the worry. In a 2026 randomized trial, mostly junior engineers learning a new library with AI scored 50% on a follow-up quiz against 67% for those who coded by hand, and weren't significantly faster. The biggest gap was in debugging (module 15).

The same trial shows how they use it matters. Participants who asked conceptual questions, or asked for explanations alongside code, scored 65% or higher. Those who delegated scored below 40%. A high-school study found the same split: an AI that gave answers hurt later exam scores, and one that tutored with hints didn't. So design the work (playbook): - AI-free reps for some fundamentals and onboarding tasks. - AI-as-tutor (learning modes, hints, explain-back) while learning. - AI-accelerated once they can verify the output. - Seniors pair with juniors on agent sessions and narrate what they reject and why. - Explain-back in every code review.

Try this. Have a junior and a senior each run the same small task with an agent, then swap and explain each other's diff without the agent open.

15. "I'll lose my own skills."

What's true. You can. A survey of knowledge workers found that higher confidence in AI was associated with less critical thinking (module 15). This round of automation is broader than past ones: agents touch the skills you need to supervise them, like reading code and debugging (module 15).

Keep practicing deliberately. Debug some problems yourself before asking, and do some work agent-free. Read every line you merge; reviewing a lot of code builds skill too, as long as you actually read it. Use the agent to understand as well as produce. Ask why, then explain it back (module 15). As module 15 notes, when tools change or fail, the people who understand the system underneath are the ones who fix it.

Try this. Set aside one hour a week with no assistant, and spend it on the part of the system you understand least.

16. "This is just a way to squeeze more output out of us."

What's true. The risk is real. Research in Harvard Business Review argued that AI tools tend to intensify work rather than reduce it. Running several agents means constant context switching, and some engineers find it more tiring than writing code themselves (module 15). Headlines about what share of a lab's code agents wrote invite exactly the wrong targets (module 08).

A good rollout rejects that framing in writing. The playbook refuses lines of code, PR counts, and "% written by AI" as targets, measures stability and review quality alongside throughput, and tells people to run only as many agents as they can actually review. If anyone starts ranking engineers by tokens or PRs, push back. The modules back you up, since Goodhart's law applies and these numbers reward bloat (module 10).

Try this. Ask your lead to commit, in the team's policy, to the metrics they will and won't use.

17. "Security will never approve it."

What's true. Security's concerns are legitimate. Prompt injection is not solved; one 2025 study bypassed most published defenses at success rates above 90% (module 11). A coding agent on a laptop has untrusted input, private data, and the ability to act, by default. Real incidents hit exactly that setup, including malware that drove developers' own AI CLIs to hunt for secrets (module 11). The agent tools themselves ship security advisories.

Security teams can approve a scoped rollout with architectural controls, because the defenses are architectural (module 11): business plans with no-training terms, sandboxes, deny rules for secret files, no production credentials on agent machines, branch protection, managed settings that disable skip-permissions modes, an approved list of MCP servers and skills, and read-only CI agents for untrusted input. Bring them the team checklist and the policy template, and ask which leg of the Rule of Two to cut for each workflow.

Attackers already use these tools, and unmanaged personal use is usually already happening, so a controlled rollout is the safer state. Do concede that some things (browser agents on logged-in sessions, CI agents holding secrets while reading public input) should wait until the controls exist.

Try this. Run the safe self-injection exercise in module 11 with your security team, in a throwaway container with a fake secret.

18. "Our code will end up training their models."

What's true. On some plans, it can. Anthropic's consumer terms changed in August 2025 so that Free, Pro, and Max users choose whether their sessions are used for training; if they allow it, data can be retained for up to five years. ChatGPT consumer accounts have their own setting. Defaults change (module 15). And anything an agent reads, including a .env file, goes to the provider and may sit in transcripts on disk (module 11).

Use business or API plans and get the terms in writing. Anthropic's and OpenAI's business and API terms default to no training on your data. Ask about retention, zero data retention for sensitive uses, data residency, SSO, and audit logs before you buy (module 12). Ban personal accounts for company code. Keep secrets out of the working tree and deny agents read access to them.

If code truly can't leave your network, self-hosted open-weight models are an option, with a capability trade-off and their own security work (module 04). Cheap third-party "coding plans" send your code to yet another provider, possibly in another jurisdiction (module 12).

Try this. Send the procurement questions in module 12 to your vendor and share the written answers with the team.

19. "We'll get sued over IP and licensing."

What's true. There are two real questions, and neither is fully settled. On ownership, the US Copyright Office concluded in January 2025 that AI-assisted output can be protected where a human determined the expressive elements, but that "prompts alone" are unlikely to qualify. Other countries differ. On infringement, models sometimes reproduce training code, including copyleft-licensed code with obligations, and the risk is highest for long, distinctive snippets (module 15).

Manage it like other IP risk, and get your legal team's view (this isn't legal advice): - Code that engineers substantially shape, review, and modify is on firmer ground than one-prompt output nobody touched. - Run license scanning in CI. Some assistants can flag or block suggestions that match public code. - Be suspicious of any large, polished block that appears instantly for a niche problem. It's on the review checklist. - Check customer contracts for restrictions on AI use with their code or data. - For open-source contributions, follow each project's AI policy. Some ban AI-generated contributions; others require disclosure and full human accountability (module 13).

Try this. Ask legal to review the IP section of the policy template before rollout.

Cost and tools

20. "It's too expensive."

What's true. At API prices with heavy use, it is. Agents re-send the whole conversation every turn, so cost grows roughly with the square of session length. Module 12's worked example puts one full-time agent on a frontier model at about $630 a month even with caching, and parallel agents multiply that (module 12). Anthropic reported about 15 times the tokens of chat for its multi-agent research system; that is one workload, not a general multiplier (module 09).

For interactive work, subscriptions change the math: roughly $20 to $200 a month for individuals (more for some tiers), and team seats with admin controls (as of November 2026; module 12). At a loaded cost of about $100 per engineer-hour, a $200 seat costs about two engineer-hours a month (module 12). Use the API for CI and automation, with spend limits. Judge cost per finished task, not per token, since a cheaper model that takes three times the turns can cost more (module 12).

Bryan's take: API pricing is too expensive for heavy agentic use. Subscriptions are what make it affordable to work this way every day.

Concede the downsides. Subscriptions have caps that bite at the worst time, vendors have changed limits and terms several times in a year, and the ROI depends entirely on whether the tools actually help your team, which you have to measure.

Try this. Have two people run npx ccusage@latest daily after a normal week and compare API-equivalent cost to their seat price (module 12).

21. "Which tool? It changes every week."

What's true. It nearly does. OpenAI's Codex CLI had reached version 0.167 by November 2026. In 2026 alone, Roo Code (a popular open-source agent) shut down, Cursor was acquired, and Gemini CLI moved to enterprise-only (module 00, module 14). Pricing and plans get reshuffled too.

The concepts transfer even when the tools don't. By late 2026 nearly every major harness had instruction files, skills, hooks, MCP, subagents, worktrees, and sandboxes (module 06). So: - Pick one primary tool and keep one warm backup from a different lab. - Put team knowledge in portable formats: AGENTS.md, Agent Skills, MCP (module 06). - Run a team AI radar and apply the two-week rule to new releases (module 14). - Switch only when your golden tasks say so, by more than the error bar.

Bryan's take: For coding, Claude and Codex are the best. Claude and Codex have both had ups and downs.

Try this. Run the afternoon evaluation on the next release that tempts you. Decide adopt, trial, or wait.

From managers

22. "We can't measure the ROI, so we can't justify it."

What's true. Measuring productivity is hard, and getting harder. Clean randomized trials struggle because people no longer want to work without AI. Self-reports are unreliable: at one AI lab, staff reported about 12x individual uplift while organizational progress was estimated at about 2x; these are different quantities (module 05).

You can measure enough to decide. Record a baseline, then track review latency, PR size, change failure and revert rates, escaped defects, golden-task results, and cost (playbook). For the business case, show the arithmetic with its assumptions: a 10-person team on $200 seats costs about $24,000 a year, roughly 13% of one loaded engineer, so the break-even gain is small (module 12). The weak link is the assumed gain, so replace it with your measured numbers after the pilot. Fluency with these tools is also becoming a baseline expectation for engineers (module 15), which matters for hiring and retention.

Try this. Fund a small pilot with a written exit decision (playbook Phase 1), and decide on its results.

23. "Why not just mandate it and set a target?"

What's true. Mandates do move adoption numbers, and some teams need a push.

Targets on usage measure the wrong thing. "Percentage of code written by AI" describes adoption, not quality, and as a target it rewards accepting more AI output (module 10). DORA's 2025 report found AI adoption linked to higher throughput but lower delivery stability, and summarized it as "AI amplifies what's already there" (module 05). Pushing volume into weak review and testing practices amplifies the weakness. What works better is paid tools, real training on your own code, a review rule, outcome metrics, and champions who share failures as well as wins (leads path). Expecting everyone to learn the tools is reasonable. A quota isn't.

Try this. Replace any usage target with two outcome targets from the playbook's "what to watch" table.

Other

24. "What about the environmental cost?"

What's true. Total data-center energy demand is large and growing. Agentic coding is far heavier than a single chat prompt: an hour can be 60 or more model calls with long contexts, and no provider publishes per-session figures (module 15).

One provider reported a falling median energy cost for text prompts: Google measured its median Gemini text prompt at 0.24 Wh in May 2025, after a 33x drop over twelve months. Module 15 gives an illustrative guess (not a measurement) of about 144 Wh for an hour of agent calls, roughly an hour and a half of a 100-watt appliance. The real number could be higher or lower. Don't run loops that produce nothing, use smaller models for easy work (it's cheaper too), and be skeptical of energy numbers without a stated method. It's a fair concern, and part of it can't be answered precisely yet.

Try this. Add "loops and swarms have stop conditions" to your team's automation rules. This limits both energy use and spend.

Questions nobody can fully answer yet

Be honest about these. They're open questions in the research, not only on your team.

  • Does AI code review reduce production defects? Most studies measure whether comments lead to code changes, not whether fewer bugs reach users (module 10).
  • What happens to skill formation over years, not hours? The learning studies measure short-term quizzes and exams (module 15).
  • Where will future senior engineers come from if junior hiring keeps falling (module 15)?
  • Do long agent time horizons transfer to messy, existing codebases? METR's tasks are cleaner than most tickets (module 08).
  • Are today's subscription prices sustainable? Nobody outside the labs knows the unit economics (module 12).
  • Who is accountable for code nobody reviewed? Law, audits, and incident reviews assume a person looked (module 08).

Sources and further reading

Every answer above links to the module section that sources it. The primary sources behind the most-cited points: