Harnessing AI to Assess Team-Based Performance

• At a recent webinar, two educators shared how they now view AI not as a potential tool for cheating, but as a powerful way to enrich student learning and foster deeper engagement in course concepts. 
• Both have integrated AI tools into their classrooms to gain insights into student group dynamics, evaluate individual student contributions, and deliver individualized feedback.
• The presenters agreed that the technical and logistical challenges of using AI are balanced by its potential to deliver comprehensive, consistent, and tailored feedback to students.

 
Years from now, professors will likely divide the timeline of higher education into two distinct phases: “Before ChatGPT” and “After ChatGPT.” What today’s professors do in these early years of the “after” era will shape future best practices for deploying artificial intelligence (AI) in the classroom.

This early exploration into AI’s potential in higher education inspired a webinar delivered in January by AACSB’s Online Learning Affinity Group. The webinar, titled “AI in Assessment of Group Work,” featured presentations by two professors from the University of Iowa’s Tippie College of Business in Iowa City. Both educators shared how the results of their first experiments using AI-powered tools to gauge their students’ communication, collaboration, and leadership skills in group assignments.

First, Erin Nelson, associate professor of practice of management and entrepreneurship, described how she has created a custom GPT in ChatGPT to analyze transcripts of student discussions and generate feedback that students can integrate into their reflection essays. Then, Jon Garfinkel, Henry B. Tippie Research Professor in Finance, discussed his use of an AI-supported assessment platform to track student learning and team dynamics in a large introductory finance course.

Nelson and Garfinkel noted how quickly they have transitioned from fearing that AI would be “a tool for cheating” to appreciating its transformative power to assist their teaching. Both highlighted how AI has reshaped the way they assess team-based learning experiences—and the way their students engage with course assignments.

Creating a Custom GPT

Nelson began her presentation by noting how little personalized feedback she could provide to student teams in her course. As a single instructor, she could enter a few online breakout rooms to hear snippets of conversations and offer quick suggestions, but she had no way to analyze every team meeting or deliver comprehensive constructive feedback.

She realized that ChatGPT could offer her a solution. In 2023, she began designing an AI-powered team coach for MBA students in her Maximizing Team Effectiveness course. She first sought guidance from a colleague who had created a custom GPT for his classroom. Next, she worked with the school’s Office of Teaching, Learning, and Technology (OTLT) to create a back-end prompt for her AI-powered team coach.

She provided the prompt with information about who her students were and what the course was designed to achieve. She also included rubrics based on the “seven drivers of team effectiveness” highlighted in the course’s textbook. These drivers include communication, cognition, coordination, capability, conditions, coaching, and cooperation.

Once Nelson had created her custom GPT, she tested it with colleagues from the OTLT. She also tried it out with the members of a faculty committee that she led, who agreed to have the chatbot analyze their meetings’ transcripts and offer feedback on their interactions. “It’s humbling when the chatbot tells you that you’ve been too dominant in the meeting!” Nelson said. “It’s a fun exercise.”

Nelson tested her AI team coach with faculty colleagues who agreed to have the chatbot analyze their meetings’ transcripts and offer feedback on their interactions.

After testing the AI team coach, Nelson decided to pilot its use during two simulations that student teams complete as part of Maximizing Team Performance. Students complete the simulations in online breakout rooms; after each simulation, the teams write reflection papers that are the only graded portion of the activity.

Today, student teams can integrate the AI coach’s feedback into their reflections. However, its use remains optional, Nelson explained, because she still is refining the tool and its output. Before the simulations begin, Nelson asks teams to inform her if they want to use the tool. She then provides those teams with instructions and screenshots that walk them through the process.

To use the AI coach, teams first must enable the breakout room platform to create transcripts of their meetings. They save those transcripts as Notepad files, with an option to change their names in the file if they would like to anonymize who is speaking. They then email those files to Nelson, who uploads the text to the AI platform. After analyzing the transcripts, the AI coach generates detailed feedback, which Nelson emails back to the teams.

This back-and-forth process might seem “clunky,” Nelson admitted. However, she views it as a necessary concession, since many of her students still use the free version of ChatGPT, which limits the size of the files they can upload. Nelson can use her paid subscription to process the large files generated by each 90-minute meeting.

Appreciating AI’s ‘Neutral’ Criticism

Teams can choose what, if any, of AI-generated feedback to include in their reflection papers, along with how they plan to adjust their performance in response. However, students don’t always think that the AI coach’s conclusions are accurate. “I’ve had times when they pushed back and said, ‘Hey, it gave us this for feedback, but we don’t really agree with that.’” In these cases, the students often show Nelson evidence that the AI coach’s feedback missed the mark.

Nelson is pleased with how well the AI coach has performed so far, and she hopes to find more ways to use the AI coach throughout the course. For example, she would like to adapt the prompt to analyze student interactions on discussion forums, as well as to answer students’ specific questions about how to improve their teams. She also is exploring whether she might use AI to analyze videos of team meetings to glean insights from each student’s tone of voice.

The number of teams choosing to use the tool has gradually increased, from 50 percent in the spring 2023 semester to nearly 100 percent in the fall 2024 semester. The more students hear positive feedback from their peers about the tool’s usefulness, Nelson said, the more they want to use it themselves.

In fact, some students have told Nelson that they are more open to feedback from the AI coach than they would be to feedback from a human instructor. They appreciate that the AI is “a neutral tool,” she emphasized, which makes it easier for them to accept constructive criticism that they otherwise might not want to hear.

Using AI as a ‘Fly on the Wall’

In his presentation, Garfinkel said that, like Nelson, he originally viewed ChatGPT as a threat to learning. But he soon realized that he could use AI to assess students’ group work, as well as their understanding of course concepts. His primary challenge, he said, was that he teaches a large introductory finance course that enrolls more than 250 students; students often break into more than 50 groups to complete many of the course assignments.

In a single 75-minute session, Garfinkel said, “there’s no way for me as a professor to walk around the room and hear what’s going on … for more than a handful of groups.” And while he enlists the help of teaching assistants to provide teams with feedback, that feedback can be inconsistent.

“What I really needed,” said Garfinkel, “was a kind of ‘fly on the wall’ that watches the entire meeting.” He wanted to use AI to track how much students contributed to their peers’ understanding of the material; assess how well they understood the course concepts; and provide tailored, consistent feedback at the individual, group, and class levels.

Pairing AI With Bloom’s Taxonomy

Unlike Nelson, who built her own chatbot, Garfinkel turned to Breakout Learning, an edtech startup that provides educators with AI-powered assessment. Breakout Learning worked with Garfinkel to train the AI platform to apply his chosen rubrics. Garfinkel used Bloom’s taxonomy to guide his metric, asking the AI to determine which of the model’s six levels of learning students demonstrate during each activity: remembering, understanding, applying, analyzing, evaluating, or creating.

Garfinkel clarified that because of the introductory nature of his course, students are unlikely to reach the creating stage. Therefore, he focuses on the first five levels. 

In Garfinkel’s course, student groups first must choose a date and time to meet for each assigned question and activity. Next, they reserve online breakout rooms using Breakout Learning’s platform, where the technology will monitor the content of their interactions. The platform also generates screenshots of any documents or spreadsheets that students create to support their discussions.

AI links students’ own words with the rubric to analyze where their understanding falls on the Bloom’s taxonomy scale.

The platform “will just hang out for the entirety of the meeting for each of the 50 groups and listen to everybody,” Garfinkel said. “I can’t pull that off.” And unlike human teaching assistants, the AI can apply the course rubric consistently across all groups, using the same standards and terminology.

For one assignment, Garfinkel asks students to work through a mini-case study. In the case, a soda company’s executives must decide whether to expand their product line into the coffee business. As students work through the problem, they must differentiate between sunk costs and incremental costs, as well as determine projected revenues and costs over the next seven years, while factoring in inflation.

AI listens to students’ discussions of the case and then analyzes where their understanding falls on the Bloom’s taxonomy scale, from Level 1 to Level 5. The AI “links the student’s own words with my rubric to decide on Bloom’s taxonomy-level comprehension,” Garfinkel said in his webinar presentation.

Interestingly, it also indicates when students do not participate in discussions of concepts at all. In such a large lecture environment, Garfinkel noted, some students can allow “a few vocal outliers” to dominate the conversations. He wants to “figure out a way to encourage more discussion.” For instance, he predicts that as students become more accustomed to receiving and responding to AI’s feedback, they might contribute more to group conversations.

Viewing AI as an Asset, Not a Threat

Garfinkel acknowledged that many AI platforms are still working out some technical difficulties. That said, his experience using AI to assess group work has been positive, and he plans to continue to incorporate the technology in his course. “Assessment of group work by AI listening and watching in the background is feasible,” he emphasized.

Both Nelson and Garfinkel agreed that the “After ChatGPT” era holds great promise for higher education. Rather than being a professor’s worst nightmare, they suggested, AI is likely to turn out to be one of a professor’s greatest assets, filling in where traditional teaching methods fall short.