On Working with Wizards

Ethan Mollick writes that AI interaction is shifting from what he called "co-intelligence," in which humans guide and correct a chatbot as an intern or co-worker, toward a model he calls "wizards." In this mode, he says, users get sophisticated output from vague requests but have no part in the process and cannot confirm how choices were made.

As one illustration, Mollick says he fed his book Co-Intelligence and roughly 140 One Useful Thing posts into NotebookLM and used its video overview option with a basic prompt about what has happened in AI. He writes that he checked the resulting video's facts and found no substantive errors; it got numbers right, including MMLU scores and neurosurgery exam results. His stated issue was that it omitted that he was one of several co-authors on a Boston Consulting Group study that also introduced the term "jagged frontier."

Mollick describes GPT-5 Pro, accessible only to paying users, as feeling most like a wizard. He says he asked it to critique the methods of his job market paper, figure out better methods, and apply them; nine minutes and forty seconds later he received a detailed critique. He writes that GPT-5 Pro apparently ran experiments using code, did Monte Carlo analysis, and re-interpreted fixed effects. It concluded his paper's "headline claim survives scrutiny," he writes, and found a small previously unnoticed error involving two linked sets of numbers in two tables. He checked the results and found them correct but says he does not know how the AI discovered the problem.

He also describes giving Claude 4.1 Opus, which recently gained file-working ability, a multi-tab Excel file from an entrepreneurship class exercise on a desk manufacturing business, asking it to update the exercise for a cheese shop. He says it read the lesson plan and old spreadsheets, including formulas, and produced a transformed spreadsheet; he spotted a few formula and business modeling choices he would have made differently but called them a difference of opinion rather than a substantive error. A follow-up request for a PowerPoint produced what he calls a solid start to a pitch deck without major errors, but not ready to go.

Mollick writes that these systems are essentially agents that plan and act autonomously, and that in agent systems powered by reinforcement learning, no one selects the steps. He argues users cannot fully know what a system did or verify accuracy without checking every fact, and that handing work to AI removes chances to develop expertise. He proposes learning when to use AI as co-intelligence versus a wizard, becoming connoisseurs of output rather than process, and embracing provisional trust. He notes that in the three tasks he gave AI he is an expert and saw no factual errors, though there were minor formatting issues and choices he would have made differently.