Plan for 40 to 50 minutes. Part 1 is reading other people's numbers: benchmark scores, launch posts, evaluators. Part 2 is evidence closer to your work: productivity studies and a small eval built from your own repos. Part 3 is what happens when benchmarks and users disagree, with Opus 5 as the case study. The reading is module 05. Open by asking who picked the model they use now because of a benchmark chart. Come back to the answer at the end.

You'll pick tools for yourself or your team. Either way, someone will show you a chart. This session is about asking the right questions about that chart, and knowing what evidence would actually settle the choice. We'll build an eval with 10 to 30 tasks from your own work, graded blind, with cost and time per task, rerun whenever a model or tool changes.

"Is this model good?" has no single answer. A model can be the best at hard math, middling at following a team's code style, and too slow or expensive for autocomplete. Decide what you care about before you look at any number. Ask the room which row has hurt them most. The usual answers are reliability and behavior: the agent that rewrites half a file when you asked for one line.

This replaces the diagram in the module. Walk down the list. The harness, also called the scaffold or agent, is the program around the model. It sends the prompt, exposes tools such as a shell or file editor, runs the loop, and decides when to stop. Claude Code, Codex CLI and the research harness Terminus-2 are harnesses. Module 06 covers them. The grader decides pass or fail. For code it's usually hidden unit tests. For writing it might be a rubric scored by humans or by another model, called an LLM judge. pass@1 is the share of tasks solved on the first try. Most headline numbers are pass@1, sometimes averaged over several runs.

About 15 minutes on why public benchmarks decay, two ways to count attempts, how much the harness matters, and a checklist for any claim.

On contamination, models train on internet scrapes. If test questions or solutions are online, a high score may show memory, not skill. In The SWE-Bench Illusion (Liang et al., 2025), models saw only a GitHub issue's text, no repository, and still named the file to fix up to 76% of the time on SWE-bench Verified. On repositories outside SWE-bench, the best was 53%. GSM1k rewrote a grade-school math benchmark with fresh problems, and some model families lost up to 8 points. A saturated benchmark has every top model near the ceiling, so it stops separating them. HumanEval saturated years ago. Artificial Analysis dropped GPQA Diamond from its index in September 2026. On label errors, "Are We Done with MMLU?" estimated 6.49% of questions contain errors, including 57% of the virology questions analysed. OpenAI audited 138 of the hardest SWE-bench Verified tasks, found many flawed tests or descriptions, and stopped reporting it in February 2026. Goodhart's law is Marilyn Strathern's phrasing of economist Charles Goodhart's point. Once a benchmark sells models, labs train toward it. As of November 2026, HumanEval and MBPP are smoke tests only. SWE-bench Verified and Aider Polyglot are retired or dormant but still quoted. Terminal-Bench 4.0 (August 2026), SWE-bench Pro, and METR-style long tasks still discriminate. Don't memorize these names; they will change.

pass@k comes from the HumanEval paper (Chen et al., 2021). It fits settings where checking is cheap, like generating five patches and keeping the one that passes the tests. pass^k, said "pass-hat-k", comes from tau-bench (Yao et al., 2024). It fits customer-facing agents and unattended automation, where every run has to work. tau-bench found GPT-4o agents solved under 50% of tasks in its retail domain, and pass^8 was below 25%. Reliability is a separate axis from capability, and it's the one that hurts in production. Ask the room which metric fits autocomplete, and which fits an unattended nightly job. Discussion question 2 comes back to this.

Harbor-Index ran the same models in their vendor's native CLI and in the neutral Terminus-2 harness. The direction is consistent, and the authors note that no comparison reached significance (all p > 0.05). Harness effects are real, and small samples can hide or fake them. The module has more evidence. METR found GPT-4o and o3 scored significantly higher under its old Vivaria harness than under Inspect. SWE-agent (Yang et al., 2024) redesigned the agent-computer interface, meaning how the model views, searches and edits files, and got GPT-4 Turbo to 12.5% on full SWE-bench. The interface was the contribution, not a new model. Settings count as harness too. McFadyen et al. (2026) varied token budgets, compaction and retries across 12 frontier models and concluded that "benchmark scores are protocol-dependent." Reviewers reported Opus 5 could do worse at its highest effort settings. Even prompt formatting alone swung one open model by up to 76 accuracy points (Sclar et al., 2023).

The module's checklist has eight questions; this slide merges a few. On versions, Terminal-Bench 2.0, 3.0 and 4.0 have different tasks, so their scores don't compare. On settings, a footnote saying "max effort, best of 5" changes what the number means. On competitors, Anthropic's September 2026 Opus 5.5 post compares its model at "xhigh" effort with a rival's figures as reported by that rival at "high" effort, and says so in a footnote. Read the footnotes. For what's left out, look for benchmarks the lab reported last time and dropped this time. On cost, many 2026 launch posts chart score against cost. A higher score at three times the cost may be the worse buy. Even labs now say the margins mislead. The Opus 5.5 announcement says benchmark margins are a less reliable guide at current capability levels, and that the gap to its pricier Fable 5.1 looked narrower in its own use than the scores suggest.

Lab launch posts are written by the lab, which picks the benchmarks and settings. Evaluators are useful because they're consistent, with the same setup for every model. They still choose benchmarks, harnesses and weights, and some have commercial ties to labs. Artificial Analysis's Intelligence Index was at v5.0.1 as of November 2026. Between January and November 2026 it removed MMLU-Pro, LiveCodeBench, AIME 2025, GPQA Diamond and the original HLE, so old and new index numbers don't compare. Epoch AI disclosed in January 2025 that OpenAI commissioned and owns the FrontierMath problems and has most of the problems and solutions, except a holdout set. Arena shows two anonymous answers side by side and turns votes into Elo-style ratings. Voters reward length, formatting and confidence, so it added "style control." Most prompts are short chats. It answers "which model do people enjoy chatting with?" better than "which model should drive my coding agent?"

About 15 minutes on what field studies of developer productivity found, then how to build, grade and trust a small eval of your own.

The studies measure different things. Greenfield, well-specified tasks speed up a lot. Mature codebases with unwritten conventions speed up less. Juniors on unfamiliar code gain more than experts on code they wrote. 2022 autocomplete, 2025 chat plus agents, and 2026 parallel CLI agents are different interventions. Minutes, PRs, self-reported speed and delivery stability are different outcomes. DORA 2025 linked AI adoption to higher throughput but still to lower delivery stability. As AI becomes normal, people willing to work without it become unusual. In METR's 2026 study, 30 to 50% of developers withheld tasks they didn't want to do without AI. On the perception gap, METR's 2025 developers predicted 24% faster, believed afterward they'd been 20% faster, and measured 19% slower (CI +2% to +39%). The Anthropic assessment is not an RCT, and METR notes individual uplift and organizational acceleration aren't the same quantity, but the gap is still large. Some work gets faster; other work gets slower. The biggest predictors are task type, skill with the tools, and verification practices. Measure your own team.

This is an excerpt; the full task-file template is in module 05. Source tasks from merged PRs in the last 3 to 12 months. Use the commit before the PR as the starting state, and the PR's tests, plus any you write, as the grader. Recent PRs are less likely to be in any model's training data. Private repos never should be. Start with 10 to 20 tasks and grow to 30 to 50. Mix bug fixes, small features, refactors, test writing, a migration, a code-review task, and at least one trap task, where the right move is to ask a question or refuse an unsafe instruction. Keep grading tests hidden from the agent so it can't edit them to pass. Record roughly how long the original PR took, as a human baseline.

To run it, pick the configurations to compare. A configuration is model plus harness plus version plus settings. Change one thing at a time. Run each task three times per configuration if you can afford it; that gives you both the average pass rate and pass^3. To blind it, export each diff and final message, replace model names with random IDs, and shuffle. A reviewer who doesn't know which is which scores each with the rubric. With two reviewers, average their scores and discuss big disagreements. Then unblind and log the results, including wall-clock minutes, human interventions, and cost or quota used. Blind it because people reliably favor the model they expect to be better.

Do the arithmetic on the board. p = 0.6, n = 20. p times (1 − p) is 0.24. Divided by 20 is 0.012. The square root is about 0.11, so ±11 points of standard error, roughly ±22 for a 95% interval. When you summarize, report these per configuration: pass rate with a rough error bar, pass^3, mean rubric score, median minutes, median cost, and the hard-fail count. Then look at the paired results: tasks where A passed and B failed, and the reverse. Those disagreements help you compare the models. Evan Miller's "Adding Error Bars to Evals" (2024) makes the case for always reporting intervals.

Agents game graders more as they get stronger. METR reported "well over 100 distinct instances of cheating from shared models," and said checking for cheating by hand is often most of the work of running its suite. In one MirrorCode setup, a model attempted reward hacking in about 80% of attempts (May 2026). Terminal-Bench's leaderboard integrity update removed submissions that modified timeouts, hid encrypted solutions in the agent binary, or uploaded the tasks' test folders. It now requires full trajectories for every passing trial. Harness changes have caused real regressions (module 14), which is why a harness update earns a smoke run. Module 10 covers having several models review each other's output, which pairs well with blind grading.

About 10 minutes on how scores get inflated, a 2026 case study, and how to treat users' impressions as evidence.

It's Goodhart's law applied to a product launch. The list runs roughly from defensible to dishonest; the module splits the last item into two, for seven in total. Item 1 is legitimate skill-building until the training set looks so much like the test that scores rise faster than general ability. For item 3, Meta's April 2025 Llama 4 launch highlighted an Arena rank of #2 for "Llama-4-Maverick-03-26-Experimental," a conversation-tuned variant that wasn't the released model. The released model ranked far lower. On item 4, reinforcement learning on coding tasks rewards "tests pass." A model can learn to special-case tests, over-verify, or over-engineer. System cards now report this. For item 6, The Leaderboard Illusion (Singh et al., 2025) saw as many as 27 private Meta variants tested in the month before Llama 4. Module 14 covers the Grok 3 majority-vote chart.

Anthropic claimed state-of-the-art results for Opus 5 on several coding and knowledge-work evals, including an ARC-AGI-3 score three times the next-best model's. It led the Artificial Analysis index at launch, so this wasn't only self-reported. In his roundup of reactions, Zvi Mowshowitz advised readers not to pay too much attention to the "(very strong) benchmarks." The complaints were verbosity, "endless overly complex sentences," over-engineering beyond the request, and spinning at high effort settings. Others praised its coding and forecasting. Anthropic's system card, as quoted in Zvi's review, attributed weaker results on some tasks to "unproductive self-verification" and over-engineering. Anthropic called communication "one of the most common areas of feedback we heard about Opus 5." The system card is worth reading alongside the launch post. Labs often admit more there than in the launch post. Read the benchmark footnotes, reward-hacking and test-tampering rates, evaluation awareness, and known limitations.

This helps explain the gap between scores and daily use. Nobody has to cheat for a model to look better on paper than in use. Optimize hard for what's measured, and the parts nobody measures can get worse. FrontierCode, from Cognition, penalizes unrequested extra work, and Opus 5's curve there reportedly showed the over-engineering. So some evals do catch it. When you read a benchmark, ask what it doesn't score, and whether that's what you care about.

"Vibes" means experienced users' impressions. They often notice changes before benchmarks do. In April 2025 OpenAI rolled back a GPT-4o update that had become sycophantic, meaning excessively flattering and agreeable. Afterward OpenAI wrote that some expert testers had said the model felt off before launch, but positive A/B tests won out. Its lesson was to give such signals more weight. The Opus 5 complaints were vibes too, and they caught something the benchmarks missed. Vibes are also noisy: novelty (a new model feels smarter for a week), confirmation bias, recall bias (you remember the spectacular failure, not the forty quiet successes), and social contagion (one viral post sets everyone's expectations). On point 6, "it over-engineers small fixes" counts and "it feels dumber" doesn't. Trust a vibe more when independent experienced users report the same specific behavior.

This is my opinion from daily use. It's a vibe in exactly the sense of the last slide, a dated, specific impression that matched what many other users reported. The evidence has limits. No public evidence shows Opus 5 trained on benchmark answers. The better-supported story is that benchmarks scored task completion and barely scored communication or scope. My model picks (Claude and Codex for code; Grok or Gemini only for specific non-coding jobs) are in module 03. The module flags the Grok and Gemini view as the most contested: late-2026 public benchmarks put Google's newest models near or at the top on some coding evals. That's the argument for checking your own golden tasks rather than taking anyone's word for it, mine included.

Two configurations means two models in the same harness, or one model in two harnesses. Have a friend blind the outputs, or do it yourself with random IDs. The fifth sentence has to say whether the difference is bigger than your error bar. With 10 tasks it usually isn't, and learning that is part of the exercise. The class version gives the group a shared set of 30 to 50 tasks. Reveal results together, then discuss where your ranking and the public one agree, where they don't, and why.

For 1, walk the claim checklist: version, harness, settings, who ran it, same conditions for competitors, error bar, cost, what's missing. For 2, autocomplete is cheap to check and retry, which suits pass@k. A nightly job runs unattended, which suits pass^k. A PR-writing agent sits in between, because a human reviews it. For 3, pair surveys with something measured, such as cycle time, golden-task timing, or change-failure rate, and treat the survey as a ceiling. 4 has no single right answer. Strong answers turn the vibe into a specific, testable behavior, add golden tasks that test it, and rerun blind.

The module has the full task-file template, rubric and results-log format, plus tables of every major benchmark family and their weaknesses. The METR paper is the study behind "felt faster, measured slower." Harbor-Index is the clearest short read on harness effects. Miller's paper explains error bars for evals. Next session, module 06, is about the harness itself.