Latest / Embodied AI 101 / SoCRATES: Evaluating LLM Mediators in Conflict Scenarios
Transcript
- SoCRATES: Evaluating LLM Mediators in Conflict Scenarios. Large language models (LLMs) are being tested in increasingly complex interactive roles — not just as static question-answerers, but as agents in multi-person dialogues, negotiation games, and even simulated decision-makers. One intriguing role is mediator: an impartial third party who helps two (or more) disputants reach agreement. But how can we reliably test whether an LLM is good at mediation? SoCRATES (Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations) is a new benchmark designed to answer precisely that question. Introduced in mid-2026 by researchers at KAIST and Chungnam National University, SoCRATES claims to be the first comprehensive framework for evaluating proactive LLM mediators in realistic, emotionally rich dialogues. In this episode we’ll break down what SoCRATES is, why it’s needed, and what its initial results tell us about LLMs as mediators. Imagine two people arguing over how to split resources — say, two managers deciding on a budget split, or neighbors debating a community issue. In a mediated dialogue, a neutral third party listens to both sides and interjects with proposals or guiding questions. This is a dynamic, social process: the mediator must track multiple topics (budget items, common concerns), understand the participants’ intentions and emotions, and adapt its style and suggestions accordingly. SoCRATES recognizes that mediation is inherently social and evolving: as the authors note, mediation “unfolds as a real-time trajectory shaped by disputants’ shifting emotions, intentions, and context”. In other words, unlike one-off tasks, a good mediator must handle a moving target — what one party says may change the emotional tone or the goals of the discussion. Existing LLM evaluation benchmarks have not fully captured this complexity. Many current testbeds for conversational AI use expert-authored domains and simple scoring schemes. For example, past dialogue benchmarks may script a handful of topics and ask an LLM to play one role, scoring each turn against all possible goals. As SoCRATES’s authors point out, these methods are limited: they “rely on a few expert-authored domains” and tend to vary only strategic posture (e.g. asking the model to be more or less agreeable). Worse, some evaluations simply score every turn against every topic, which means if the conversation veers slightly off one issue, the scoring gets noisy and uninformative. The paper bluntly notes that these approaches “score every turn against every topic, introducing off-topic noise”. In practice, that means an utterance about budget might be (wrongly) graded on progress in discussing scheduling, for example. SoCRATES was built to overcome these shortcomings. It provides a multi-domain testbed of realistic conflicts and a refined evaluation method that focuses on what actually matters. Rather than a handful of canned dialogues, SoCRATES constructs scenarios “from real conflicts through an agentic pipeline across eight domains”. These domains are not fully listed in the abstract, but one can imagine they span the kinds of disputes found in everyday life: workplace disagreements, family or community issues, maybe hypothetical social dilemmas. Crucially, SoCRATES doesn’t just script a fixed pattern. They vary five socio-cognitive adaptation axes to make each scenario different and to stress-test the mediator under diverse conditions. The five axes are: strategic posture, party composition, history length, emotional reactivity, and cultural identity. In simpler terms, these axes cover aspects like:. Strategic posture: Are the disputants cooperative or adversarial? (For example, do they start off willing to compromise, or firmly take opposite sides?). Party composition: Who is participating? (Two individuals? A group of three? Are they peers or an expert/novice mix?). History length: How much prior disagreement is there? (Is the conflict freshly described, or do the parties have a long record of arguments already?). Emotional reactivity: How emotionally charged is the dispute? (Do the parties quickly get upset or stay relatively calm?). Cultural identity: Are the parties from different backgrounds? (Differences in language, norms or national culture can vastly change how messages are interpreted.). These axes capture facets of real-world mediation. For instance, a culturally mixed dispute might require the mediator to translate not just languages but values. High emotional volatility might require tone adjustments or empathy. By systematically varying these factors, SoCRATES explores if an LLM mediator can adapt along social dimensions as well as content ones. The benchmark itself pairs these axes with the multi-topic nature of mediation: a scenario may have several issues at play, and the mediator must advance each one. To handle this multi-topic nature, SoCRATES introduces a topic-localized evaluation. Instead of scoring a mediator’s utterance on all issues, each response is only judged on the specific topic it addresses. This eliminates the “off-topic noise” problem. For example, if one topic is “pricing” and another is “delivery schedule,” and the LLM suggests a compromise on pricing, the evaluator only scores the mediator’s impact on the pricing issue. This fine-grained approach better reflects whether each suggestion is on-point for the issue under discussion. Impressively, their automatic evaluator shows a very strong agreement with human judges — about 0.82 alignment — roughly doubling the accuracy of a naive per-turn baseline. In other words, machine scoring in SoCRATES almost matches what a human would say about the mediator’s effectiveness on each topic, at a much finer scale than previous benchmarks. Building the Scenarios. How are these dialogues constructed? SoCRATES uses an “agentic pipeline” to generate scenarios from real conflict data. Though the paper abstract doesn’t detail the pipeline, this typically means a combination of AI agents or tools is used to simulate realistic conversations. For example, they might start with news reports or role-play prompts about a real dispute, then use LLMs (or rule-based agents) to play the roles of the disputants and mediator in a conversational loop. By automating generation this way, SoCRATES can cover many variations: different initial positions for the parties, different sequences of turns, and so on, all while grounding the dialogue in plausible human conflict. The result is a rich set of multi-turn conversations, each annotated so that evaluation can identify which turns advance which topics, what the consensus status is, and (implicitly) how well the mediator helped. Evaluation Metric – The Consensus Gap. One key metric in SoCRATES is the consensus gap, describing how far apart the disputants remain after (or without) mediation. Imagine a dispute where the ideal agreement would be 100% overlap between what each side wants. Before any mediation, the “consensus gap” is large — parties start from different viewpoints. A perfect mediator would close this gap to zero (full agreement); a poor mediator might do little. Concretely, the authors report how much of this gap remains after the LLM mediator has intervened. Across their benchmarks, even the best mediating LLM leaves about two-thirds of the gap unresolved. In their words, “even the strongest mediator closes only about a third of the unmediated consensus gap”. Put differently, on these realistic, varied testbed dialogues, state-of-the-art LLMs reduce only about 30% of the disagreement that would otherwise remain. Key Piloting Insights. SoCRATES’ initial experiments shed light on current LLM strengths and weaknesses in social dialogue. They benchmark eight “frontier” LLMs (the exact models aren’t listed in the abstract, but think GPT-4, GPT-4o, Claude 3, Llama 3, Gemini, etc.). All of them manage some improvement, but none achieve close-to-complete mediation. Moreover, performance is highly uneven. The same model might do reasonably well if the dispute is simple or low-emotion, but collapse when the environment changes. The paper notes performance “varies sharply by socio-cognitive axis”. For instance, an LLM might handle two rational, like-minded managers negotiating fine, but struggle when one party becomes very emotional or when parties come from different cultures. This suggests that social adaptation is the bottleneck: LLMs may know language and common negotiation tactics, but fail to tune their style to emotional cues or cultural differences. In short, the SoCRATES results highlight that current LLM mediators lack real-world subtlety: they make some progress, but still leave most disagreements unsolved, especially under challenging socio-cognitive conditions. To put SoCRATES in context, it helps to compare with other recent benchmarks for LLM conversation roles. Multi-agent negotiation has been assessed in other ways. For example, Sahar Abdelnabi et al. (ICLR 2024 workshop) introduced the LLM-Deliberation benchmark: a suite of multi-issue, multi-agent text games where LLMs negotiate deals in a back-and-forth conversation. In their experiments, GPT-4 was significantly better at reaching deals than earlier models, demonstrating how requiring arithmetic, planning, and exploration stretches an LLM’s capabilities. Similarly, the PACT testbed (Pairwise Auction Conversation Testbed) frames a 20-round bargaining game between buyer and seller. Here, an LLM sits in as the buyer or seller with private valuations, sending short messages and price quotes each round; success is measured by how much profit the model nets overall. Such benchmarks test language-based negotiation skills — probing reasoning over value and trust dynamics — but they are still direct contests between two parties. By contrast, SoCRATES uniquely puts an LLM in the middle, mediating others instead of being one of the disputants. Another related line is work on conflict analysis: Özgecan Koçak et al. (Dec 2024) tested GPT-3.5 and GPT-4 on diagnosing the root of disagreements (whether they stem from factual versus moral differences). They found that GPT-4 had a human-like sense of this distinction — it could usually tell when parties disagreed about facts versus values — but it still had systematic biases (tending to label conflicts as “factual” too often). SoCRATES builds on those insights: if an LLM can’t even accurately interpret what’s at stake in a disagreement, it’s unlikely to resolve it. Taken together, SoCRATES and these related studies paint a picture: LLMs are getting better at handling parts of negotiation or conflict tasks, especially in controlled conditions (like pure bargaining or simple moral judgments). But real mediation remains hard. SoCRATES’ main contribution is a methodology to measure how hard, by simulating rich, evolving dialogues and evaluating improvement more precisely. Why This Matters. One might wonder: why do we need such a complicated testbed? In practical terms, if LLMs are ever used in tools for conflict resolution — whether consumer chatbots for interpersonal disputes, or advisory systems for diplomatic talks — we must know their limits. An automated mediator that misunderstands an emotional cue or cultural norm could do more harm than good. Indeed, a recent Stanford policy study used wargame simulations to test LLMs in crisis decision scenarios and found that all of five tested LLMs (including GPTs and others) sometimes made dangerously escalatory moves unless properly fine-tuned. Their conclusion was that unaligned LLM agents can unpredictably amplify conflict. SoCRATES is aligned with this cautionary view: it serves as a stress-test to see where and how LLMs falter in conflict-like settings. By quantifying that even “strong” models only solve a fraction of the problem, SoCRATES tells us we are far from hazard-free LLM mediators. Interpreting the Results. The finding that current LLMs close only ~1/3 of the consensus gap is sobering. It means that if two people are 100 points apart in disagreement, an LLM mediator might only bring them 30 points closer on average. In practice, that could mean leaving them still in serious conflict. Why do LLMs fall short? The SoCRATES paper suggests some reasons: the performance “varying sharply by socio-cognitive axis” implies the models had specific weaknesses. For example, it’s likely that in scenarios with high emotional reactivity, the LLM responses became formulaic or tone-deaf, undermining trust. In cross-cultural contexts, the model might unknowingly default to one culture’s norms and confuse the other party. Also, if the history of the dispute is long and convoluted, an LLM’s limited memory might cause it to repeat itself or ignore prior points. In contrast, a skilled human mediator would tailor questions, show understanding of each party’s perspective, and revisit earlier agreements systematically. Current LLMs generally lack that level of nuanced tracking and empathy. The strong alignment score (0.82) of SoCRATES’s evaluator with humans also raises an interesting point: the evaluation itself is credible. It means we can trust the numbers. Benchmarks often struggle with who “judges” performance in such complex tasks — SoCRATES’ use of a calibrated automated scorer (akin to a trained human judge) doubles our confidence in the result. And the drop-off compared to a simple baseline underscores that those naive scores were indeed misleading. Using SoCRATES. The authors of SoCRATES have promised to release their benchmark (as hinted by a project page link on their abstract). This means other researchers can now run new or improved models through the same testbed. If a lab develops an LLM that is expressly fine-tuned for negotiation or empathy, SoCRATES will reveal whether it truly behaves better across all the social dimensions. It also provides a dataset of scenarios and outcomes that can be mined for error patterns. For example, if an LLM consistently fails to advance a negotiation in the “emotional” variant but not in the “logical” variant of the same conflict domain, that is a strong clue that modeling emotion is the sticking point. Caveats and Next Steps. As with any benchmark, SoCRATES has limitations. It simulates conflicts using an LLM-driven pipeline — the dialogues are generated, not human transcripts — so they may not capture every nuance of real human speech. But this is a common trade-off: realistic enough to challenge models, yet controllable for systematic testing. Also, the five axes cover many social factors, but not all (for instance, they don’t explicitly vary gender or age as far as we can tell). Future work might expand these axes or add new ones (like negotiation power or incentives). On the LLM side, the results suggest clear directions for improvement: integrating models of emotion (perhaps by fine-tuning on counseling dialogues), enforcing cross-cultural sensitivity (fine-tuning on multilingual or multicultural corpora), and enhancing memory for long conversations. Compared to prior evaluation efforts, SoCRATES is unique in directly assessing the mediator role. Previous benchmarks like PACT or LLM-deliberation mostly treat the LLM as a negotiator on one side. Koçak et al.’s work looked at an LLM’s understanding of disagreements, but not its conversational strategy. SoCRATES provides a new lens: instead of just the content of a single utterance or outcome of one negotiation, it looks at the trajectory of a dialogue under a guiding hand. In doing so, it reveals that language understanding alone is not enough — the social and affective layers are where LLMs still lag. Related Research Highlights. It’s worth highlighting a few representative findings from related studies. As mentioned, Abdelnabi et al. demonstrated that GPT-4 can negotiate complex deals in structured games, but older models stumble. The PACT paper showed how to rigorously measure negotiation skill via profits in buyer-seller auctions. Koçak et al. gave evidence that GPT-4 mostly “knows” the difference between factual vs. moral disputes, but still exhibited human-like biases (overemphasizing factual differences over moral ones). A Stanford policy brief found that, without careful alignment, even powerful LLMs tend to amplify international conflicts rather than quell them. So the emerging picture is consistent: LLMs are linguistically capable, but without careful conditioning, they can mishandle sensitive, multi-turn social situations. SoCRATES adds to this by quantifying how much of a conflict an LLM can resolve, turning these qualitative observations into measurable scores. Concluding Thoughts. The SoCRATES paper is a valuable early step toward understanding complex LLM capabilities in conversational roles. As one of the first benchmarks explicitly built for LLM “mediators”, it provides a systematic way to gauge progress. Its sobering conclusion — that even top models only shrink disputes by about 30% — is a call to action. It tells us that if we want LLMs to play mediator (or teacher, coach, or any similarly nuanced social role), we have more work to do in imbuing them with real socio-cognitive savvy. For now, SoCRATES gives us the yardstick. Future research can use it to measure improvement: whether through better training data, enhanced model architectures, or specialized fine-tuning for empathy and cultural understanding. In summary, SoCRATES brings much-needed rigor to automated mediation evaluation. It blends realistic dialogues (the “eight domains”), social diversity (five adaptation axes), and fine-grained scoring (topic-localized evaluators) into a unified framework. For practitioners and researchers building AI for cooperative dialogue, it highlights that success requires more than knowledge — it demands empathy, adaptability, and context-awareness. As LLMs continue improving, benchmarks like SoCRATES will ensure we test those human-like qualities, not just surface fluency. Sources: All details above are drawn from the SoCRATES preprint and related literature. In particular, the SoCRATES abstract and summary provide the core descriptions. For broader context, we referred to studies on LLM negotiation (PACT, LLM-deliberation) and LLM conflict analysis (Koçak et al. 2024) as well as a Stanford report on LLMs in conflict scenarios. These help frame why SoCRATES’s focus on socio-cognitive factors and realistic dialogues is novel and important.