🔍 Read the full analysis: Consultants’ Data Science Gains With ChatGPT Didn’t Carry Over on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
In a randomized trial at Boston Consulting Group, consultants using GPT-4 scored higher than a control group on three data-science tasks outside their usual skill set. In a later technical test without ChatGPT, researchers found no evidence that the AI group performed better than the control group; the authors describe the task gains as temporary.
A randomized trial at Boston Consulting Group found that consultants using ChatGPT performed better on three data-science tasks than colleagues using conventional online resources, but researchers found no evidence that the AI group retained an advantage on technical questions completed without the chatbot. The study, published October 26 in Nature Human Behaviour, bears on whether workplace AI assistance builds lasting skills or mainly helps people perform while the tool is available.
The trial involved consultants assigned to one of two groups. The treatment group received 15 to 20 minutes of prompting instruction and used GPT-4; the control group received instruction in using Google and sites including Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks: writing Python code to clean and combine datasets, evaluating a statistical model, or building a model to predict international football match outcomes.
Across the tasks, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. The researchers also reported that the ChatGPT group finished the coding task about nine minutes sooner and the prediction task about 14 minutes sooner on average. Time spent on the statistics task did not differ significantly between groups.
The benchmark was the average performance of 44 BCG data scientists who completed the tasks without AI. The ChatGPT group still scored below that benchmark on average across all three tasks. After the work, participants answered five technical questions on related topics without ChatGPT. The authors report that they “find no evidence” the treatment group did better than the control group on that tool-free test.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Gains Did Not Show Skill Retention
The findings distinguish between doing a technical task with AI and being able to do related work independently. For employers, that difference affects how they interpret productivity gains: stronger work completed with a chatbot is not, by itself, evidence that employees have learned the underlying methods.
The study suggests AI can help workers take on tasks beyond their existing expertise while it is available. But the follow-up result offers no evidence that this short intervention produced a measurable advantage on the test without the tool. Companies evaluating AI training may need separate measures for assisted output and unaided capability, as well as quality checks for work produced with AI. The trial does not establish that all workplace AI use fails to teach skills; it tests a particular group, training approach and set of tasks.
Products Worth Considering
LG UltraGear 32-Inch QHD 180Hz Curved Gaming Monitor
Apple Mac mini Desktop Computer with M6 Chip (2026)
Apple MacBook Pro 14.2-Inch Laptop with M6 Chip, Space Black
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The researchers preregistered the trial on March 13, 2024, before collecting data, and ran it at BCG in late March and early April. Of 986 consultants allocated to the groups, 573 began the survey and 487 submitted work on both assigned tasks; that group formed the main analysis sample. Participants had at least a bachelor’s degree, and the authors said most reported no coding experience.
BCG data scientists and OpenAI’s Economic Impacts Research team developed the tasks so ChatGPT could provide useful guidance but could not correctly answer every question independently, according to the study. This design aimed to test whether participants could use the tool to solve problems that required their own judgment, rather than simply copy a complete answer.
The paper also reports a subgroup result: consultants identified as having prior coding experience scored about five percentage points below the data scientists on coding and statistics when using ChatGPT. The authors found no evidence of a difference from the experts on those two tasks for this subgroup, while its prediction score remained 20 points below the benchmark. These subgroup findings do not change the overall result that the ChatGPT group, on average, remained below the expert benchmark.
“AI lets nontechnical workers handle complex tasks outside their skill set, “but once the technology is removed, the enhanced capabilities disappear.””
— Wiles, Krayer and co-authors, describing the role of generative AI
Limits of the Retention Test
The study’s follow-up consisted of five technical questions completed without ChatGPT after the task session. The supplied study summary does not establish how long any learning might last, whether a longer or different training program would change retention, or whether the same results would apply to other occupations and AI systems.
The paper describes its finding about increased confidence in ChatGPT’s ability to solve problems it could not solve as exploratory. The available source material does not provide enough detail to quantify that effect or establish its consequences for workplace decisions. The trial also does not show that every use of AI assistance prevents learning; it tests one short training intervention and specified data-science tasks.
Testing Longer-Term Workplace Learning
The paper’s reported trial is complete, and the findings were published in Nature Human Behaviour on October 26, 2026. The next evidence needed would come from research that tests whether different training methods, repeated practice or longer periods of AI use lead to skills workers can demonstrate without assistance.
For employers applying the findings now, the study points to a practical distinction to track: how well employees perform with AI, and how well they handle comparable work when the tool is unavailable. The trial does not specify a required workplace policy or establish how organizations should structure training; those decisions remain outside what this experiment tested.
Key Questions
What did the BCG study find about ChatGPT and data-science tasks?
Consultants using GPT-4 scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group using conventional online resources. The ChatGPT group still scored below the data-scientist benchmark on average.
Did the consultants retain an advantage after ChatGPT was removed?
No advantage was demonstrated in the follow-up. The authors said they “find no evidence” that the ChatGPT group did better than the control group on five technical questions completed without the tool.
Who took part in the trial?
The trial was conducted with consultants at Boston Consulting Group. Of 986 people allocated to groups, 487 submitted work on both assigned tasks and made up the main analysis sample.
Does the study prove that ChatGPT cannot teach workplace skills?
No. It examined a specific short training session, tool and set of tasks. Its findings do not establish whether longer training, repeated practice or other workplace settings could produce different learning outcomes.
Source: ThorstenMeyerAI.com
