The chatgpt Classroom Experiment: How AI Impacts Student Learning in Engineering – And What Educators Are Doing About It
(Published April 22, 2024)
The rise of artificial intelligence, particularly large language models (LLMs) like ChatGPT, has sparked a critical debate in education. Are thes tools a threat to academic integrity, or a potential catalyst for innovative learning? Researchers at the University of Illinois Urbana-Champaign’s Grainger College of Engineering decided to find out, conducting a unique semester-long experiment to directly compare ChatGPT’s performance against human students in a rigorous undergraduate control systems course. The results, while nuanced, offer valuable insights for educators and students alike, and highlight the evolving landscape of higher education in the age of AI.
The Experiment: ChatGPT as a Virtual Student
Driven by the realistic assumption that students will utilize AI tools for coursework, the research team, led by Ph.D. student Gokul Puthumanaillam and advisor Melkior Ornik, treated the free version of ChatGPT as a fully enrolled student. ChatGPT received the same assignments, lectures (via provided course materials), and grading criteria as the human students. The goal wasn’t to catch AI use, but to understand its capabilities and limitations within a demanding engineering curriculum. The study meticulously mirrored a real-world classroom surroundings: assignments were completed synchronously, prompts were identical to those given to students, and there were no collaborative projects to introduce confounding variables.
The Verdict: A’s on Arithmetic, D’s on Deep Thinking
The findings were striking. ChatGPT excelled at routine,mathematically-focused homework,consistently achieving an “A” grade. However, its performance plummeted when faced with problems requiring critical thinking, analysis, and nuanced reasoning.
“ChatGPT technology can achieve top marks on structured, straightforward questions,” explains Puthumanaillam. “But on open-ended questions, it scored a 62, bringing its overall semester grade down to an 82 – a low B. In contrast, the human students averaged 84.85%, demonstrating their superior ability to tackle complex analytical challenges.”
This disparity reveals a crucial point: while AI can automate basic problem-solving, it currently struggles with the higher-order cognitive skills essential for true understanding and innovation. The study suggests a student relying solely on ChatGPT could potentially pass the course with a B, but at the cost of genuine learning. The grade might be achieved thru a combination of flawless execution on simple tasks and a demonstrably weak grasp of underlying principles.
Beyond Accuracy: The problem of “Hallucinations” and Inappropriate Technical Language
The researchers also observed concerning instances of chatgpt generating incorrect or misleading details – a phenomenon often referred to as “hallucination.” Despite being provided with all course materials,the AI occasionally introduced technical jargon not used in the lectures or readings,and even presented demonstrably false statements.”It used terms like ‘quasi periodic oscillations’ wich were never covered in the class,” Puthumanaillam notes. “This highlights the importance of critical evaluation, even when the answer appears confident and well-formatted.” This underscores a critical risk: students blindly accepting AI-generated responses without verifying their accuracy.
Implications for Educators: Adapting to the New Reality
The study isn’t a condemnation of AI, but a call to action for educators.Ornik, Puthumanaillam’s advisor, emphasizes the need to adapt pedagogical approaches.
“Like calculators before it, ChatGPT is a tool that’s here to stay,” says Ornik. “This research highlighted the need to redesign courses to incorporate more higher-level questions and project-based assignments.Students can leverage AI for simpler calculations, but we need to challenge them with open-ended problems that demand critical thinking and a deeper understanding of the material.”
This shift towards more complex assessments isn’t about making courses harder; it’s about ensuring students develop the skills necessary to thrive in a world increasingly shaped by AI. It’s about fostering learning, not just achieving a passing grade.
The Future of AI in Education: A Cautious Optimism
While the free version of ChatGPT showed limitations, the researchers acknowledge that the premium version may offer improved analytical capabilities and memory for tackling more complex problems. However,they deliberately focused on the free version to reflect the likely access of the average student.
Interestingly, the study also revealed ChatGPT’s capacity for limited learning. When corrected on multiple-choice questions, it demonstrated improvement on similar variations. though,this learning was incremental and didn’t translate to significant overall progress.
**Key Takeaways & What
