Oh my god, this sounds awful. After the first few paragraphs, I was ready to be impressed, but then they started dropping all these insane details:
---
> Only 13% preferred the AI oral format. 57% wanted traditional written exams. 83% found it more stressful.
> Here is an email from a student: "Just got done with my oral exam. [...] I honestly didn't feel comfortable with it at all. The voice you picked was so condescending that it actually dropped my confidence. [...] I don't know why but the agent was shouting at me."
> Student: "Can you repeat the question?" Agent: paraphrases the question in a subtly different way.
> Students would pause to think, and the agent would jump in with follow-up probes or worse: interpret the silence as confusion and move on.
---
Based on these highlights, you'd think the experiment was a wash. The author disagrees!
> But here's the thing: 70% agreed it tested their actual understanding: the highest-rated item.
Man, you could shoot me with a gun, then make me write an essay, & I'd be forced to agree that you had tested my "actual understanding." That doesn't mean my performance wouldn't suffer. Also, 70% is not very high. That's barely two thirds.
Even the grading was done by LLMs (rather than having a TA grade a transcript, and the results were lower. The author defends this by saying, "Students will be evaluated outside the university, and the world is not known for grade inflation," but the world isn't "known for grade inflation" because it doesn't grade you at all. That's not even an excuse, it's just nonsense. It'll toughen you up, or whatever. Was this post written by an LLM too?
> Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression.
"Regression"? I mostly wrote pen & paper exams, and I only graduated a few years ago. If students want more flexibility, team up with other courses to supervise multiple exam sessions. Leaked questions aren't going to be any more of a problem than it was for take-home exams, especially since they can't take the booklets with them when they go.
It sounds like these students had a terrible time, and for what? Written exams work fine. These guys just wanted to play with LLMs.