An AI Best Practice for B-Schools

• AI-supported evaluation of small group discussions offers real-time tailored feedback, enhancing student engagement and improving student learning.
• Using AI tools, instructors of large courses can gain insights into each group’s conversation, which they can use to focus classroom time on areas where students are struggling most.
• New classroom tools can give quieter individuals more confidence to engage in discussions and expose students to AI-based evaluation methods that will become common in modern workplaces.

 
Instructors have long assigned short write-ups of cases in their courses, both to ensure that students came to class prepared for discussions and to assess their understanding of the central concepts in each case. But for those of us who teach classes based on cases, generative artificial intelligence (GenAI) has created a problem: The latest generation of large language models (LLMs) has ruined the usefulness of write-ups as a pedagogical practice.

Students today do not need to read the case, because an LLM can summarize it for them. AI may not produce A-level answers, but it often generates B+-worthy responses. With this technology available, these pre-class assignments no longer ensure either that students have adequately prepared or that instructors can accurately assess their work.

Fortunately, GenAI also provides a solution to these problem, because it can facilitate the evaluation of small group discussions.

We tested this solution last year in graduate and undergraduate business courses at New York University, the University of Arizona, and the University of California, Los Angeles. We used AI-supported technology available through Breakout Learning, a platform for implementing evaluated small group discussions. Using this technology, we found, led to even more engaged classroom interactions than our traditional methods.

Tailoring the Classroom Experience

In our classrooms, we begin by asking students to complete pre-work assignments, whether reviewing a traditional case, reading a magazine article, or watching a video. They then log into the platform to discuss the material in small groups.

During these discussions, AI sits in the background and evaluates the quality of student commentary against a grading rubric. AI can identify each speaker as long as students are in the same virtual space. The AI platform is especially beneficial for asynchronous online courses, where there are fewer opportunities for students to engage in small group discussions.

After students discuss the material, professors receive reports from the AI platform that summarize each student’s comments and gauge comprehension at the group and individual levels.

The software we use is built to accommodate groups of three to six students each. This limitation isn’t related to the AI technology, but rather reflects what we consider to be the ideal size for rich interactions. If the groups were smaller, the dynamic might lack the diversity of perspectives needed for a robust discussion. If the groups were larger, more confident students would tend to listen to each other less as they vie for personal airtime, while shyer students would find it easier to hide, resulting in an imbalance in participation.

After these interactions, professors receive reports from the AI platform that summarize each student’s comments and gauge comprehension at the group and individual levels. The platform evaluates the content of student discussions in comparison to predetermined rubrics.

For customized experiences, professors can create these rubrics in collaboration with Breakout Learning’s team. Professors also can opt to use case studies based on existing rubrics developed by academic experts who are members of Breakout Learning’s academic advisory board and faculty recruited to author cases and simulations.

Insights Into Every Discussion

Numerous studies have demonstrated the effectiveness of learning through small group discussions. Think of what happened during the pandemic. Though many of us struggled with the challenges of teaching and learning via Zoom, we eventually discovered that the technology did prove useful in facilitating small group discussions. We found then, as now, that peer-to-peer discussions eliminate the possibility of students hiding behind more vocal classmates or free-riding. Social accountability naturally drives them to perform well in front of their peers.

The problem, however, has been that such unmoderated Zoom sessions often end up being inefficient. Having each group share what they discussed with the full class consumes valuable time. Moreover, many of these report-backs end up being repetitive because groups have often discussed similar sets of issues.

If we could sit in on every one of the small group discussions, we could manage the process much better. But as human instructors, we can be in only one place at a time. By combining small groups with GenAI, we can gain insight into all of these interactions as we manage, monitor, and evaluate them at scale.

This approach has several advantages. The fact that students understand that they are being monitored eliminates the free-rider problem sometimes seen in unmonitored group discussions. Furthermore, when facultly assign synchronous, in-class small group discussions, they do not have the same opportunity to use the assistance of an LLM to evaluate student participation and performance.

Supercharging the Case-Based Method

We can use the platform’s feedback in a few ways. For example, the AI assessment allows faculty to include these small group discussions as components of the course grade.

In addition, AI reporting makes classroom debriefs and discussions more effective and efficient. We find it valuable to review the AI summaries prior to class, so that we can spend more class time on concepts that the students have had a harder time understanding. When the reports show that students have grasped the concepts, we can fast forward to more subtle aspects of the case. And after reviewing the AI reports and curated commentary, we can call on students who we know have unusual, interesting, or important perspectives.

AI-evaluated small group discussions also have had a benefit that we did not anticipate. After each of us adopted this approach, we noticed that students who had never said anything in class before began to speak up in the full group as well.

This outcome has been reflected in student feedback. “I do not have that much experience with public speaking,” one student noted, but the small group discussions “allowed me to gain more practice with presenting out loud or debating with another member on what we agree or disagree on. At first, I was a little nervous because I did not know how it was going to go and I feared saying the wrong thing. But after the first case, I felt more comfortable knowing that the material we were discussing had no right or wrong answer.”

The AI feedback that students receive helps them understand how well they have synthesized the lesson material, as well as how effective they are as listeners and communicators.

That feeling of comfort and safety, which encourages students to test ideas, carries over to the classroom. More hands are in the air; more students are eager to speak. Discussions have more depth, include a broader diversity of voices, and foster more welcoming classroom environments.

These small group discussions double down on two key advantages of case-based teaching. First, they require students to apply theoretical ideas to real-world contexts. Second, they require students to practice thinking critically on the spot. The AI feedback that students receive helps them understand how well they have synthesized the lesson material, as well as how effective they are as listeners and communicators. Students must learn to listen and react to each other, express themselves clearly, and build on one another’s ideas.

Both aspects prepare them for future workplace settings, whether in-person or online. Furthermore, given that AI is a reality of the workplace, we believe that it is critical that students gain experience with AI evaluation.

Refining AI’s Application in Education

One open question with this approach remains: How much can AI assist with student evaluation? Using Breakout Learning’s platform, we can assess students in the small group discussions using case-specific rubrics. The AI also uses Bloom’s Taxonomy to assess the depth of their contributions—do students simply remember relevant facts, or do they understand the material so well that they can apply it outside the context of the case?

Further, the platform provides individualized feedback to each student on factors such as talk time and individual discussion contributions. This kind of feedback is simply not feasible when instructors use traditional methods to evaluate students’ participation in classroom discussions.

When using the platform as a grading tool, we spot-check the AI-generated scores, much as we would with the scores provided by any teaching assistant (TA). We review some of the transcripts ourselves. Drawing from our combined experiences, we have found that the AI assessments have been quite accurate, as good as or even better than assessments conducted by the average TA.

As faculty and students become more comfortable with this technology, and as efficacy studies validate its impact on engagement and comprehension, we expect AI to play a more significant role in grading. The software also can track each student’s performance and progress across all their experiences throughout the semester.

We will continue to explore and refine the role of AI in education. As we do, these small group discussions, powered by AI integration, could well become a cornerstone of modern business education.