Were we right?
Four bets we made in early 2025, scored by the person who placed them, receipts attached.
Volume II · written from October 2026, two years after Volume I
Volume IIWhy keep score
Marking my own exam.
Marking your own exam is frowned upon in most schools. This page does exactly that, so the marking comes with the answers.
In early 2025, André-Benoît De Jaegere and I closed the first Generative Engineering study with four bets about how AI would change engineering: from compromise to coherence, from rigidity to resilience, from tools to teammates, and from execution to ethics. They were not the findings of a laboratory. They were a leap of faith, informed by more than 740 sources and, in my case, by twenty-five years of watching how engineering really gets done. Now, 20 months later, it is fair to ask whether we were right.
This is not a study, either. A study would write its search rule in advance, search every database and read everything it found. This is an opinion with receipts. Every fact can be checked; the selection is mine, and I say below how I made it; what goes against us comes first; and if you know a source that does better, the score moves.
The score
Where each bet landed.
Fig. 1
Of our four bets, one went further than we said and three were partly right; none has been contradicted yet.
Where I place each of our four bets today: a judgment, not a measurement. The reasons and the sources are below, so you can put the marks somewhere else. How many sources stand behind a bet is not drawn, because it says where my essays looked, not how much evidence exists.
HypothesisPartly right
From compromise to coherence
The bet: AI would make trade-offs obsolete. In our words: “We’ve seen how Generative AI redefines optimization, making compromise obsolete.” The Engineer's Moment
Trying out ideas did become absurdly cheap: the price of an AI answer of a given quality fell some three hundred times in under two years. Cheap options, however, did not make trade-offs vanish. A robot laboratory looking for a better material had to balance two goals, how well the film conducts and how hot it must be made, and the best it could do was map the combinations where one goal cannot improve without the other getting worse. Someone still had to pick one. Physics, it turns out, had not read our white paper.
Partly right: the search got far better, and the choice stayed. Our own epilogue said so a few paragraphs later, “Generative AI is the map. Engineers must choose the path.”, which is the polite way of contradicting yourself in the same document.
Partly right
From rigidity to resilience
The bet: digital twins would see trouble coming. In our words: “Where once we validated our designs after deployment, we now preempt reality before it unfolds.” Trail 2: Smart Digital Twins
It works when the job is small and someone can take back control. At a chemical plant run by ENEOS Materials, AI ran one distillation column on its own for a five-week trial, then for almost a year, and in 2023 the plant formally adopted it. Google let AI control the cooling of its data centres, but every action was checked twice, and operators could switch back to the usual rules at any moment. Where the promise was bigger, reality pushed back. A robotic laboratory announced 41 new materials made in 17 days; after other chemists questioned the results, its corrected paper counts 36, new to its own database but “not necessarily new to science”. And AI agents that excelled at laboratory questions did poorly when they had to run a real microscope. Knowing the manual is not holding the tool.
Partly right: the twin does see trouble coming, inside a small box with an off switch. We wrote that engineering would become “prescient”. It has become careful, which is less poetic and more useful.
Went further
From tools to teammates
The bet: AI would be a colleague, not a tool. In our words: “The calculator has evolved into a collaborator.” Trail 3: AI as Your Problem-Solving Ally
We bet on a colleague. We got one, and then some. In a field experiment with 791 Procter & Gamble professionals, people working alone with AI did as well as teams working without it. Then the colleague started acting: at its researchers’ request, an AI emailed a company to have the proteins it had designed made for laboratory tests, in emails that said Claude had written them; another wrote olympiad mathematics to a gold-medal standard. The bill arrives when someone has to check the work. One company released 719 mathematical manuscripts at once, more than any mathematician could read in a lifetime, and in an experiment where AI agents bargained for employees, the people given the weaker agent got worse deals and never noticed.
Went further: the colleague no longer only suggests, it acts. We were early on this one, which is pleasant to write and less pleasant to think about, because what an acting colleague does to the people beside it is still to be settled.
Partly right
From execution to ethics
The bet: engineers would answer for what AI does. In our words: “As AI systems take on more design, decision, and operational authority, engineers step into a new role: ethical orchestrators, balancing innovation with impact, velocity with vigilance.” Generative AI in Engineering
The work is moving as we said: in some 400,000 sessions with one coding agent, people made about 70 percent of the planning decisions and only 20 percent of the execution decisions. Answering for the result is not moving with it. One AI assistant’s terms of service bind the user to whatever it “may deem” the user asked for. Data-centre electricity grew 17 percent in 2025, against 3 percent for electricity as a whole: the technology we hoped would make engineering sustainable has an electricity bill growing almost six times faster than everyone else’s. And students who practised with an ordinary chatbot did 17 percent worse on the exam they then took without it.
Partly right: people are moving from doing the work to deciding it, as we bet. Being able to answer for it does not come free with the promotion. It has to be designed, and so far it mostly is not.
How to read a source
Which way it points, and how solid it is.
Every record asks two separate questions of its source, and refuses to mix them. A company’s announcement can point straight at “right” and stand on sand; a careful study can point at “too early to tell” and stand on rock. Mixing the two is how headlines are made.
Which way does it point?
One of five answers, for the sentence of ours that the record tests. One rule: only a source that observed something can say right or wrong. A promise of how a product will behave, or an opinion however eminent, is too early to tell. Until someone measures it, a promise is kept and broken at the same time; physicists are comfortable with that, contract lawyers rather less.
- Contradicted
- The source points the other way. It happens to the best forecasters, and below, to us.
- Partly right
- Right in part; the sentence needs an editor.
- Right
- The source shows that it holds.
- Went further
- It came true, sooner or further than we said: wrong in the pleasant direction.
- Too early to tell
- The source cannot settle it yet: a promise, an opinion, or a result nobody has checked.
How solid is it?
Six kinds of source. Follow one new car engine through all six: each step tells you something the step before could not.
- Announcement
Someone reports their own result. The maker says its new engine does a hundred miles to the gallon. Makers always do.
What it cannot tell you: nobody else has checked it yet.
- Benchmark
A score on a standard test. It does, on the standard test bench every engine goes through. AI systems are compared the same way, on fixed sets of problems.
What it cannot tell you: a test bench is not a road.
- Peer-reviewed study
Checked by independent experts before publication. Independent engineers read how it was measured and find the method sound.
What it cannot tell you: nobody has yet obtained the result again.
- Replication
Found again by someone else. Another laboratory, with its own engine, gets the same number, to the maker’s visible relief.
What it cannot tell you: it may still not matter outside the laboratory.
- Deployment
In real use, with real stakes. People drive it to work every day.
What it cannot tell you: what it costs, or how it fails, over the years.
- Long-term effect
Observed over years. Ten years on, we know how long it lasted and what it did to the traffic.
What it cannot tell you: why it happened, unless something else could be ruled out.
How the hand was dealt
Why 28 sources, when my essays cite 279.
My five essays cite 279 sources. 113 of them are older than the bets: the history and research the arguments stand on, from Herbert Simon in 1956 to the Go players of 2023. They can explain, but they cannot tell us what happened after February 2025. The other 166 date from 2025 or 2026, and they mix studies and data with press coverage, product launches and funding news, which are not the same thing however confidently they are written.
The essays ask eleven questions. For each, I kept the two or three sources that test one of our sentences most directly: 28 in all, 3 of them older ones that an essay uses as a test, each marked as such. Every one was read at the source when its essay was written. It is not a sample; it is a hand of cards, and I am telling you that I dealt it.
So the number of sources behind a bet says where my essays looked, not how much evidence exists. None of them is about optimization yet, so the bet from compromise to coherence rests on 2, and nothing in them contradicts it outright: a hole in my looking, not a proof.
The receipts
28 sources, by the question they answer.
Under each question, what goes against us comes first.
For each of our four bets, from compromise to coherence, from rigidity to resilience, from tools to teammates and from execution to ethics, what has happened since we made it?
Contradicted
Data-centre electricity
We wrote: “It’s about more sustainable AI, not just faster AI.” Generative AI in Engineering
What the source shows: Electricity used by data centres grew by about 17 percent in 2025, against roughly 3 percent for electricity as a whole.
Why “Contradicted”: We asked for more sustainable AI; the electricity AI runs on is growing nearly six times as fast as the rest.
Source: Data centre electricity use surged in 2025, even with tightening bottlenecks driving a scramble for solutions, IEA, 16 April 2026. Long-term effect: observed over years.
What it cannot show: One year of a total; data centres serve more than AI, and the total says nothing about the energy each task needs.
Read on 24 October 2026, for The Second Act.
Right
Individuals with AI, teams without
We wrote: “No longer a solitary process of deduction, engineering becomes a dialogue between human intuition and machine suggestion.” Trail 3: AI as Your Problem-Solving Ally
What the source shows: In a preregistered field experiment with 791 Procter & Gamble professionals, individuals working with AI matched the performance of teams working without it, and the AI helped people from technical and commercial functions produce more balanced solutions.
Why “Right”: A person with AI did what a team did: the machine contributed like a colleague.
Source: The Cybernetic Teammate: A Field Experiment on Generative AI and Teamwork, Organization Science 37(4), 1217–1242, 2026. Peer-reviewed study: checked by independent experts before publication.
What it cannot show: The sessions lasted a day; they did not measure how the participants’ capabilities developed over years.
Read on 26 October 2026, for Essay 04.
Went further
The price of a thought
We wrote: “Creativity had to be rationed. With generative AI, the bottleneck explodes.” Trail 1: Engineering Optimization Unleashed
What the source shows: Matching GPT-4 on a set of PhD-level science questions cost $37.50 per million tokens when GPT-4 appeared in March 2023, and about twelve cents by December 2024: some three hundred times less. Across tasks, Epoch AI measures price declines of nine to nine hundred times a year.
Why “Went further”: We expected creativity to stop being rationed; the price of trying an idea fell faster than we imagined.
Source: LLM inference prices have fallen rapidly but unequally across tasks, Epoch AI (data updated 20 November 2025, CC BY), 12 March 2025. Long-term effect: observed over years.
What it cannot show: The price of reaching a test score with a language model, not the cost of an engineering study; it leaves out the capital and the electricity behind the price.
Read on 24 October 2026, for The Second Act.
In which fields of engineering do AI systems now act (run tools, change designs, order experiments) rather than only propose, and how is it documented?
Right
Miles with no one at the wheel
We wrote: “AI-driven fleets now gather billions of miles of data, refining decision-making continuously.” Trail 3: AI as Your Problem-Solving Ally
What the source shows: Through June 2026, Waymo’s cars had driven 271.3 million miles with no human driver. The company reports 95 percent fewer crashes with serious injury or worse, and 82 percent fewer injury-causing crashes, than human drivers would have had over the same distance in the same areas.
Why “Right”: The fleets did learn to drive: hundreds of millions of miles with no one at the wheel, and fewer injuries than people would have caused.
Source: Safety Impact (rider-only miles through June 2026), Waymo, read 25 October 2026. Deployment: in real use, with real stakes.
What it cannot show: Company figures, for the cities, streets and conditions studied; not a promise about every road.
Read on 25 October 2026, for Essay 02.
Went further
Protein designs ordered by email
We wrote: “The calculator has evolved into a collaborator.” Trail 3: AI as Your Problem-Solving Ally
What the source shows: A research version of Claude searched for proteins that might bind scorpion toxins and, at its researchers’ request, sent the sequences to the company that would make them, in emails saying they were written by Claude. Of 211 designs tested in the laboratory, 17 met the binding criteria.
Why “Went further”: A collaborator suggests; this one placed the order for the laboratory work itself.
Source: How Claude designed protein binders against scorpion neurotoxins, Anthropic Research, 20 October 2026. Announcement: someone reports their own result.
What it cannot show: The company’s own account. Binding a toxin is not neutralizing it, still less treating a patient.
Read on 24 October 2026, for The Second Act.
How much recent AI-assisted mathematics has been checked in a proof assistant such as Lean, and what exactly was checked: the whole argument, or selected steps?
Right
Formal proofs at olympiad level
We wrote: “This approach not only automates proof generation but also enhances the reliability of published results by providing a formal verification mechanism.” Mathematics and AI
What the source shows: AlphaProof combined learned guidance, reinforcement learning and formal search, producing proofs a program can check; with AlphaGeometry 2 it reached a silver-medal standard at the 2024 International Mathematical Olympiad. The work was later published in Nature.
Why “Right”: Machines now produce proofs a program can check, as we said; at olympiad level, not yet in published research.
Source: Olympiad-level formal mathematical reasoning with reinforcement learning, Nature 651, 607–613 (published online 12 November 2025), 2026. Peer-reviewed study: checked by independent experts before publication.
What it cannot show: Olympiad problems, not research; it does not measure the reliability of published results.
Read on 25 October 2026, for Essay 01.
Which AI-assisted mathematical results have been independently reviewed by mathematicians, and what did the reviewers judge: correctness, novelty, or significance?
Partly right
Research problems, refereed
We wrote: “AI can handle the brute force of calculations and explore large solution spaces, but human insight remains crucial for guiding these explorations.” Mathematics and AI
What the source shows: Four systems, ten problems drawn from active research, thirty-nine submissions refereed by experts. Seven problems received at least one solution judged acceptable, and the referees found some genuinely interesting routes alongside the errors.
Why “Partly right”: Machines solved research problems themselves; the human part that stayed crucial moved from guiding the search to judging the answers.
Source: First Proof: Second Batch, arXiv 2606.18119, 16 June 2026. Benchmark: a score on a standard test.
What it cannot show: A preprint: ten problems chosen by the organizers, four systems, and solutions judged acceptable, not yet important.
Read on 25 October 2026, for Essay 01.
Went further
Olympiad gold in plain language
We wrote: “These systems are capable of tackling advanced reasoning tasks, such as those found in the International Mathematical Olympiad (IMO), where they achieved a medal-level performance.” Mathematics and AI
What the source shows: In July 2025 an advanced Gemini model reached an officially assessed gold-medal standard at the International Mathematical Olympiad, writing its solutions in natural language.
Why “Went further”: We reported medal-level work by specialised systems; a year later a general model reached gold, writing in plain language.
Source: Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad, Google DeepMind, 21 July 2025. Benchmark: a score on a standard test.
What it cannot show: Olympiads are not research; a company’s announcement of an officially assessed result.
Read on 25 October 2026, for Essay 01.
How many machine-generated mathematical manuscripts are produced and submitted, and what becomes of them: accepted, corrected, retracted, or ignored?
Too early to tell
Twenty-five Fields Medalists
We wrote: “In this way, AI acts as an accelerant for mathematical progress, augmenting human capabilities rather than replacing them.” Mathematics and AI
What the source shows: Twenty-five Fields Medalists warned of “a severe misalignment” between AI development and mathematics, citing understanding and the formation of future researchers.
Why “Too early to tell”: An opinion, however eminent, cannot settle the question; it tells us that the people best placed to judge are worried.
Source: A Severe Misalignment of AI in Mathematics, What’s New (Terence Tao’s blog), 11 September 2026. Announcement: someone reports their own result.
What it cannot show: A statement of concern by leading mathematicians, not a measurement.
Read on 25 October 2026, for Essay 01.
Partly right
Seven hundred and nineteen manuscripts
We wrote: “In this way, AI acts as an accelerant for mathematical progress, augmenting human capabilities rather than replacing them.” Mathematics and AI
What the source shows: In October 2026 OpenAI released 719 mathematical manuscripts in 372 families, with about 42 percent of their top-line results formalized; the repository itself warns that unformalized results may contain errors.
Why “Partly right”: The output sped up beyond anyone’s ability to read it; progress still needs someone to say what is true and what is new.
Source: openai/math: mathematical manuscripts and supporting proof artifacts, GitHub repository, checked 25 October 2026, October 2026. Announcement: someone reports their own result.
What it cannot show: It counts output, not progress: which results are correct, new or important is not yet known.
Read on 25 October 2026, for Essay 01.
Where in engineering and operations do agents act without prior human approval, and which of those actions can be rolled back?
Contradicted
A weaker agent nobody noticed
We wrote: “As AI takes on more cognitive load, engineers must act as both interpreters and governors—ensuring solutions are safe, meaningful, and aligned with human goals.” Trail 3: AI as Your Problem-Solving Ally
What the source shows: Agents bought and sold goods for 69 employees: 186 real deals worth just over $4,000. In separate runs, people represented by the less capable model closed fewer deals at worse prices and did not notice; they rated the fairness of their deals 4.06 out of 5, against 4.05 for the better represented.
Why “Contradicted”: We wanted people to govern their AI; these people could not even tell that theirs had served them badly.
Source: Project Deal: our Claude-run marketplace experiment, Anthropic, 24 April 2026. Announcement: someone reports their own result.
What it cannot show: A company’s experiment with household goods; the unequal runs were not the binding market, so no one lost money.
Read on 25 October 2026, for Essay 02.
Too early to tell
Approval before it acts
We wrote: “Because when machines act on our behalf, the burden of conscience is ours.” Trail 4: Sustainable Tomorrow
What the source shows: Claude Duet prepares work on its own and waits for the user’s approval before it sends a message, makes a payment, signs a document or deletes anything; some things, such as moving money between accounts, it will not do even if the user says yes.
Why “Too early to tell”: A promise of how the product will behave; nobody has measured whether it does.
Source: Introducing Claude Duet, Claude, 21 October 2026. Announcement: someone reports their own result.
What it cannot show: An announcement of intended behaviour in a research preview, not measured reliability.
Read on 25 October 2026, for Essay 02.
Partly right
An agent appointed by the terms
We wrote: “Because when machines act on our behalf, the burden of conscience is ours.” Trail 4: Sustainable Tomorrow
What the source shows: Instinct’s terms, revised on 26 August 2026, appoint the assistant “as your agent to enter into agreements, commitments or transactions on your behalf”, binding on the user “as if entered into directly by you”, for whatever actions it “may deem” responsive to the request.
Why “Partly right”: The burden is ours, as we said, but by contract, and without the control that answering for something needs.
Source: Terms of Service, instinct.com, last revised 26 August 2026, read 26 October 2026. Deployment: in real use, with real stakes.
What it cannot show: Terms say what the company may do and who answers for it; they do not say how often the assistant acts without asking.
Read on 25 October 2026, for Essay 02.
How much human supervision sits behind systems described as autonomous, and how is it reported?
Partly right
The AI shopkeeper
We wrote: “The calculator has evolved into a collaborator.” Trail 3: AI as Your Problem-Solving Ally
What the source shows: Claudius, an AI running a small shop in Anthropic’s office, sold tungsten cubes for less than it had paid for them, was talked into discount after discount and asked to be paid into an account it had invented. People did the restocking, and its emails could not reach real suppliers.
Why “Partly right”: It ran a shop like a colleague would, but in a setting people had arranged for it, and it sold below cost when customers asked nicely.
Source: Project Vend: Can Claude run a small shop? (And why does that matter?), Anthropic, 27 June 2025. Announcement: someone reports their own result.
What it cannot show: A company’s experiment in an arranged setting; it shows what was tested, not what a shop would face.
Read on 25 October 2026, for Essay 02.
Right
Who plans, and who executes
We wrote: “As Generative AI scales, so too must engineering judgment, transparency, and responsibility.” The Engineer's Moment
What the source shows: In some 400,000 sessions of Anthropic’s coding agent, people made about 70 percent of the planning decisions but only 20 percent of the execution decisions. Sessions run by novices reached verified success about 15 percent of the time, those of people with at least intermediate expertise 28 to 33 percent.
Why “Right”: Judgment is where people’s work is going: they kept the planning, handed over the execution, and expertise made the difference.
Source: Agentic coding and persistent returns to expertise, Anthropic, 16 June 2026. Deployment: in real use, with real stakes.
What it cannot show: Preliminary findings from one company’s product, with expertise rated by a model.
Read on 25 October 2026, for Essay 02.
Right
More freedom, more steering
We wrote: “As AI takes on more cognitive load, engineers must act as both interpreters and governors—ensuring solutions are safe, meaningful, and aligned with human goals.” Trail 3: AI as Your Problem-Solving Ally
What the source shows: New users let Anthropic’s coding agent act without asking in about a fifth of their sessions, experienced users in more than two fifths; yet the experienced also stepped in more often, interrupting it in about nine turns out of a hundred against five.
Why “Right”: Experienced users gave the agent more freedom and stepped in more often: governing it, as we asked.
Source: Measuring AI agent autonomy in practice, Anthropic, 18 February 2026. Deployment: in real use, with real stakes.
What it cannot show: It does not show that experience caused better supervision or safer outcomes; one company’s product.
Read on 25 October 2026, for Essay 02.
Which self-driving laboratories choose and run their next experiment without a person in the loop, at what scale, with what failure rates, and how are their results verified?
Contradicted
Agents at the microscope
We wrote: “Virtual prototyping flips the innovation script, transforming imagination into action at the speed of thought, where bold ideas no longer wait for reality to catch up.” Trail 2: Smart Digital Twins
What the source shows: Language-model agents that excelled at materials-science questions performed poorly when driving an atomic force microscope, and sometimes drifted from their instructions, a habit the authors call sleepwalking.
Why “Contradicted”: Describing an experiment is not running one: the agents stumbled at the instrument.
Source: Evaluating large language model agents for automation of atomic force microscopy, Nature Communications 16, 9104, 14 October 2025. Peer-reviewed study: checked by independent experts before publication.
What it cannot show: One instrument and the models of 2025.
Read on 26 October 2026, for Essay 03.
Partly right
A laboratory that maps the frontier
We wrote: “We’ve seen how Generative AI redefines optimization, making compromise obsolete.” The Engineer's Moment
What the source shows: The Ada self-driving laboratory explored palladium films while balancing two goals at once, conductivity and processing temperature, and measured a Pareto frontier: the recipes where neither goal can improve without costing the other.
Why “Partly right”: The laboratory found the best compromises between two goals; it did not make the compromise disappear.
Source: A self-driving laboratory advances the Pareto front for material properties, Nature Communications 13, 995, 22 February 2022, before our bets. Peer-reviewed study: checked by independent experts before publication.
What it cannot show: Published in 2022, before the first study; one material and two goals. It maps the frontier; it does not make the choice along it.
Read on 26 October 2026, for Essay 03.
Partly right
An autonomous laboratory, corrected
We wrote: “Virtual prototyping flips the innovation script, transforming imagination into action at the speed of thought, where bold ideas no longer wait for reality to catch up.” Trail 2: Smart Digital Twins
What the source shows: The A-Lab reported 41 novel compounds from 58 targets in 17 days. After other researchers questioned how the products had been identified, a January 2026 correction withdrew one target, confirmed 36 of the 40 remaining successes, and changed “novel materials” to “inorganic materials” in the title: the targets were new to the prediction platform, “not necessarily new to science”.
Why “Partly right”: The laboratory did make compounds fast, but fewer than it announced, and not all new to science: reality caught up.
Source: Author Correction: An autonomous laboratory for the accelerated synthesis of inorganic materials, Nature 650, E1, 19 January 2026. Peer-reviewed study: checked by independent experts before publication.
What it cannot show: One laboratory’s account, corrected after independent scrutiny; four results remain inconclusive from diffraction alone.
Read on 26 October 2026, for Essay 03.
How do closed-loop systems in robotics and process industries assign the authority to stop, and how is that authority tested?
Right
One distillation column
We wrote: “Not replicas, but reflexes—systems that don’t just observe reality, but predict it, respond to it, even shape it in real time.” Trail 2: Smart Digital Twins
What the source shows: After a 35-day trial of autonomous control in a chemical plant in 2022 and almost a year of further operation, ENEOS Materials and Yokogawa announced in 2023 its formal adoption at the plant, for one distillation column.
Why “Right”: Autonomous control works: for one column, in one plant, inside conditions people already understood.
Source: In a World First, Yokogawa’s Autonomous Control AI Is Officially Adopted for Use at an ENEOS Materials Chemical Plant, Yokogawa press release, 30 March 2023, before our bets. Deployment: in real use, with real stakes.
What it cannot show: Company accounts of a bounded application: one column in one plant, not a factory that redesigns itself.
Read on 26 October 2026, for Essay 03.
Right
Cooling, with a way back
We wrote: “The more intelligent the system, the more responsible its engineer must be.” Trail 2: Smart Digital Twins
What the source shows: When DeepMind moved from recommending changes to directly controlling the cooling of Google data centres, each action was checked against safety constraints, then again by the local control system, and operators could take back control at any time and return to the established rules.
Why “Right”: The system could be trusted to act because its engineers kept the means to check it and stop it.
Source: Safety-first AI for autonomous data centre cooling and industrial control, DeepMind, 17 August 2018, before our bets. Deployment: in real use, with real stakes.
What it cannot show: Described in 2018, before the first study, by the company; a specialized controller, not a general agent.
Read on 26 October 2026, for Essay 03.
What does measured evidence say about how AI assistance changes what early-career engineers learn, as distinct from what they produce?
Contradicted
A new library, then a quiz
We wrote: “As Generative AI scales, so too must engineering judgment, transparency, and responsibility.” The Engineer's Moment
What the source shows: Fifty-two professional Python developers learned an unfamiliar library through a coding task, with or without an AI assistant, then took a quiz without it. Those who had learned with it scored 50 percent, against 67 percent for those who had learned without.
Why “Contradicted”: Those who learned with the assistant knew less once it was gone.
Source: How AI Impacts Skill Formation, arXiv 2601.20245, January 2026. Announcement: someone reports their own result.
What it cannot show: A randomized experiment posted as a preprint, not yet peer-reviewed; small groups and one library.
Read on 26 October 2026, for Essay 04.
Partly right
A chatbot and a tutor
We wrote: “As Generative AI scales, so too must engineering judgment, transparency, and responsibility.” The Engineer's Moment
What the source shows: In a Turkish high school, nearly a thousand students practised with nothing, with an assistant like ordinary ChatGPT, or with a tutor that gave hints rather than answers. The ordinary assistant raised practice grades by 48 percent, and its students then did 17 percent worse on the exam taken without it. The tutor raised practice grades by 127 percent, with no detectable difference on the exam.
Why “Partly right”: Judgment grew only when the help was built to teach; help that gave answers left students weaker on their own.
Source: Generative AI without guardrails can harm learning: Evidence from high school mathematics, PNAS 122(26), e2422633122 (affiliation corrected, PNAS 122(34)), 25 June 2025. Peer-reviewed study: checked by independent experts before publication.
What it cannot show: One school, four sessions, mathematics; it measures the exam that followed, not careers.
Read on 26 October 2026, for Essay 04.
Right
Go players who learned from the machine
We wrote: “As Generative AI scales, so too must engineering judgment, transparency, and responsibility.” The Engineer's Moment
What the source shows: In nearly six million decisions by professional Go players from 1950 to 2021, Shin and colleagues found that human play improved after superhuman programs arrived (PNAS, 2023). A separate study of professional players, with its own data and method, also found that their moves improved, most among young players.
Why “Right”: Players who learned from superhuman programs became better players, and two separate studies found it.
Source: How does artificial intelligence improve human decision-making? Evidence from the AI-powered Go program, Strategic Management Journal 46(6), 1523–1554, 2025. Replication: found again by someone else.
What it cannot show: A game with fixed rules and a superhuman referee; the diversity of moves narrowed as the new knowledge spread.
Read on 26 October 2026, for Essay 04.
What is measured, and what is merely assumed, about AI’s effect on junior engineering roles?
Contradicted
Danish earnings and hours
Not one of our bets, but a worry of mine in Essay 04: “The work that once exposed juniors to mistakes is being finished before they have had time to make one.”
What the source shows: Linking surveys of chatbot use to Danish administrative records, the authors found precise null effects on earnings and recorded hours two years after ChatGPT, with tasks changing all the same.
Why “Contradicted”: In Denmark, two years on, no measurable change in pay or hours; if junior work is disappearing, it does not show there yet.
Source: Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI (first circulated as “Large Language Models, Small Labor Market Effects”), NBER Working Paper 33777, revised March 2026. Long-term effect: observed over years.
What it cannot show: A working paper; one country and period, and earnings and hours, not who gets the first job.
Read on 26 October 2026, for Essay 04.
Too early to tell
Workers aged 22 to 25
Not one of our bets, but a worry of mine in Essay 04: “The work that once exposed juniors to mistakes is being finished before they have had time to make one.”
What the source shows: In American payroll data, employment of workers aged 22 to 25 in the most AI-exposed occupations stands 19 percent below where it would be had it kept pace with their less-exposed peers, mostly through less hiring rather than more firing.
Why “Too early to tell”: Fewer young people in the jobs AI touches most, but the authors cannot yet say that AI is the cause.
Source: Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence, Stanford Digital Economy Lab working paper, revised, 12 August 2026. Long-term effect: observed over years.
What it cannot show: A working paper; the authors call these early, descriptive signals, not causal estimates, and the pattern weakens when education is controlled for.
Read on 26 October 2026, for Essay 04.
Right
Junior hires at one AI company
Not one of our bets, but a worry of mine in Essay 04: “The work that once exposed juniors to mistakes is being finished before they have had time to make one.”
What the source shows: At Anthropic, the junior share of technical hires fell from 27 to 9 percent between spring 2025 and mid-2026, while it held in non-technical teams; hiring managers increasingly look for autonomy and judgment rather than execution and narrow expertise.
Why “Right”: At one AI company the junior share of technical hires fell from 27 to 9 percent: the first rung is thinning there.
Source: How employment is changing at Anthropic, Anthropic, 16 October 2026. Announcement: someone reports their own result.
What it cannot show: One fast-growing AI company, partly self-reported; it cannot establish a trend across the economy or isolate AI as the cause.
Read on 26 October 2026, for Essay 04.
The log
Every time the score moved.
27 October 2026
First score published: four bets, 28 sources from the five essays, two or three per question.
27 October 2026
Two sources moved to “too early to tell”. A reader asked why only one source was undecided. The reader was me, and the answer was not flattering: I had let a promise (Claude Duet waiting for approval before it acts) and an opinion (the Fields Medalists’ statement) count as evidence. Neither had observed anything.
27 October 2026
Epoch AI’s prices moved from benchmark to long-term effect: two years of observed prices, not a score on a test. No reading changed.
Mark it yourself
The exam stays open.
I marked my own exam and showed you the marking. You may disagree with a mark, and you probably should with at least one: that is what the receipts are for. If you know a source that does better than mine, above all one that proves us wrong, send it to me on LinkedIn. If it holds, the score moves, and the log says why.