Chatbots in class?

Throughout my time at Exeter, history classes and essay writing have been inseparable.

My own public school history class assessments mostly consisted of multiple-choice exams, memorizing facts and circling the correct answer, occasionally compiling them in a string of sentences that passed as an essay. But at Exeter, we strive to teach history as a means for developing critical thinking skills, emphasizing use of evidence, engaging narration, and close reading of sources. And while there are many ways to hone these skills, writing has been a crucial one; it’s hard to imagine teaching history at Exeter without having a stack of essays to grade at some point each term. But last year, ChatGPT threatened to upend everything. 

An acronym for Chat Generative Pre-trained Transformer, ChatGPT is perhaps the most infamous version of a number of new artificial intelligence chatbots that can converse using “humanlike conversational dialogue.” According to one tech website, “the language model can respond to questions and compose various written content, including articles, social media posts, essays, code and emails.” Basically, ask ChatGPT a question, and it will give you an answer, sometimes a lengthy one. Shortly after it was released last November, I decided to try it out by asking it to write the essay my students were currently working on. I typed in the essay question, and in a matter of seconds, the interface produced what seemed to be, at first, a well-organized and cogent three-page essay. 

I’ll be honest that I had a moment of alarm. Was it possible that every student in my class could simply spend two minutes using ChatGPT and come away with a submittable essay? But, as I read the piece more carefully, I became less impressed. The analysis was general, vague, and lacked specific examples. When I prompted the interface to provide more specifics, problems arose. Scenarios and page numbers were inaccurate, and the writing did not clearly link examples to analysis. We now know these problems to be widespread; one lawyer submitted a ChatGPT-written brief that cited imaginary court cases, and librarians have received bibliographies of sources that do not exist. 

Feeling confident that the essay assignment was secure, I decided to go bold: I walked into class and projected ChatGPT for the students to see. “I know you’ve heard about this, but I just want to show you why it’s no good,” I explained as I entered the essay prompt. As the screen started writing, though, the reaction of my students was not what I wanted. 

“This is amazing!” several students said. “I’m never going to write an essay again,” one student remarked. (Perhaps my boldness rubbed off on them.) Even as I tried to point out the vagueness, generalizations and analytical flaws, the students remained captivated. The best I could do was to remind them of our new history department policy that using AI was simply not allowed, but I left the class wondering if they had gotten the message.

Despite my chagrin in that moment, I still have hope for the longevity of human-written essays, and I plan to continue assigning them. After all, I have high standards for my students: to distinguish evidence from generalizations; to support their analysis with strong, specific evidence; and to inspect the texts they read for these details, too. I haven’t yet seen evidence that AI can do this type of analysis well; AI-generated text seems to specialize in writing the generalizations that make me comment in the margins, “What evidence supports this claim?”

Most importantly, I want my students to write about history with accuracy and nuance. “Accurate” is the first category on the rubric I give my students, and I weight it twice as much as any other category when I grade essays. And despite the promises that technologies will improve over time, the “bugs” of inaccuracy I have described seem to be “features” of AI-generated text that works through prediction. But even if AI-generated text is accurate, I want my students to be able to explain how we know it’s accurate, and that requires attention to sources, inspecting them for trustworthiness and limitations. Arguably, in an era where “information” (accurate or not) is just at our fingertips, evidence-based essay writing using credible sources is more important than ever. Until we can trust the accuracy of AI, its uses for history essays will remain limited.