Latest / 1 More Paper / Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills — Laskar et al. (2026)

Transcript

Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills — Laskar et al. (2026) · 1 More Paper

• You’re listening to “Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills,” by Md Tahmid Rahman Laskar and colleagues. Published in arXiv on October 9, 2026. Abstract. Enterprise customer-support agents increas- ingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem through the lens of Skill LLMs: specialized executors that operate within a selected skill context and must complete a stateful workflow under policy and tool constraints. We present a benchmark construction pipeline that mines recurrent workflows from real-world transcripts, converts them into structured skill descriptions, maps them to tool requirements, simulates environment state with Model Context Protocol (MCP)-compatible mocks, and generates multi-turn tool-using dialogues. Using this benchmark, we conduct a large-scale zero-shot evaluation of 17 language models spanning closed model families and deployment-oriented open-weight models from 1.7B to 14B parameters, each judged independently by five strong LLM judges, plus a deterministic rule-based evaluator that checks for proper tool use. We find a consistent ranking in which GPT-5.4 leads by a wide mar- gin and QWEN3.5-9B is the strongest open-weight model (below 20B), outperforming several closed baselines under every judge. We also find that judge choice is not a negligible methodological detail: pairwise judge agreement on the pass/fail verdict averages Cohen’s κ = 0.524 across 10 judge pairs. These results suggest that comprehensive zero-shot evaluation of skill execution requires judge reliability as carefully as model capability. 1 Introduction. Enterprise AI agents build on recent progress in tool-using Large Language Models (LLMs), but require a distinction between orches-tration and skill execution. An orchestrator acts as the control layer of an agentic system: it interprets the user’s request, determines which capability or skill is needed, and routes the interaction to the appropriate tool, workflow, or human operator. A skill LLM, in contrast, is a specialized executor: it operates within a selected skill context and must complete a narrow, stateful workflow under policy, tool, and conversational constraints. In enterprise customer support settings, success is not only a matter of fluent response generation. The model must pre-serve dialogue state across turns, collect missing information, follow company-specific procedures, call the correct tools, interpret tool outputs, and handle exceptions such as tool errors, refusals, and out-of-scope requests. These requirements are dif-ficult for general-purpose models, which are often trained on broad instruction-following data rather than workflow executions. Before investing in adapting or deploying any particular model for skill execution, a practical pre-requisite is knowing how well current models al-ready do this zero-shot, and how reliably we can measure it. Reference-based text-generation metrics do not capture whether an agent actually followed a required workflow, called the correct tools, or fabricated a claim of task completion; conversely, LLM-judge evaluation is only as trustworthy as the judge itself, and different judge models can disagree substantially on the same transcripts. To this end, we intro-duce a data construction pipeline for generating and benchmarking skill execution. The pipeline starts by mining workflows from real transcripts across high-impact industry settings, converts these workflows into structured skill descriptions, simulates dynamic environment state using Model Context Protocol (MCP)-compatible mocks for tool execution, and finally generates multi-turn user-agent dialogues, from which a fixed 500-scenario benchmark is selected for evaluation. The key principle is to derive the benchmark scenarios from a real-world workflow representation rather than using a frontier model to freely halluci-nate plausible-sounding but ungrounded scenarios. Using this benchmark, we conduct a comprehensive zero-shot evaluation of 17 models spanning closed (GPT-5, GPT-5.4, GEMINI) and sub-20B open-weight (QWEN3, QWEN3.5, MINISTRAL-3) families, each scored by five independent LLM judges plus a deterministic rule-based evaluator. In addition, we measure inter-judge agreement. In summary, our major contributions are threefold. Formulating Skill SLM execution as a specialized problem distinct from orchestration. Presenting a data generation pipeline that mines workflows from real transcripts to generate skills, simulates an agentic environment with tool mocking, and produces a high-quality filtered benchmark of multi-turn dialogue scenarios for evaluating LLMs on skill execution. Providing a large-scale zero-shot evaluation across 17 models and five independent judges that yields a stable model ranking while quantifying judge-dependent disagreement in agent evaluation. 2 Related Work Tool-using language agents. Prior work on tool-using LLMs mainly studies broad tool generaliza-tion, where agents select and invoke tools for di-verse tasks. In contrast, we study LLMs for workflow execution within a given skill context, where tool use is governed by the skill’s policies and constraints. Task-oriented dialogue and user simulation. Task-oriented dialogue has long been studied for goal completion through multi-turn interaction, dialogue-state tracking, and API calls. Recent agent benchmarks extend this setting with domain policies and tool execution. We build on this line of work but focus specifically on enterprise skill-execution scenarios grounded in real customer-support workflows. Synthetic data for agent benchmarks. Synthetic instruction and dialogue data have been widely used to study instruction following and tool use. Unlike open-ended synthetic dialogue generation, our benchmark scenarios are grounded in workflows mined from real customer support transcripts. We convert these workflows into skill descriptions, generate scenarios with user states and personas corresponding to these skills, and pair them with mocked tool environments, making the resulting benchmark grounded in real customer-support workflows rather than being generated di-rectly by a model that may lead to hallucination. LLM-as-judge reliability. A growing body of work uses LLMs to judge open-ended generation and agent trajectories, but judge reliability itself is an active concern. We con-tribute a multi-judge measurement of this problem in the agentic tool-use setting by comparing pass/-fail agreement across five judges, covering both proprietary and open-weight models. 3 Methodology 3.1 Data Construction Our data construction is inspired by prior agent-task formulations that combine tasks, tools, and user simulation. However, our setting differs from these benchmarks that draw scenarios from a fixed synthetic template distribution. More specifically, our goal is to construct a benchmark whose scenarios reflect the actual workflow distribution observed in a real-world production customer-support setting. This requires the benchmark to provide more than task inputs and final answers: it must expose the model to workflow structure, available tools, diverse user and environment states, and the consequences of intermediate tool calls through dy-namic tool responses. It should also cover realistic interaction paths, including follow-up questions, refusals, out-of-scope requests, tool failures, and recovery behavior. The resulting benchmark is designed to test whether a model can operate reliably within a bounded skill context, covering workflow, tools, and policy constraints. Figure 1 summarizes the pipeline. More specifically, our data generation pipeline converts transcript-derived workflows with skill descriptions into multi-turn dialogue scenarios with real tools used in production environments. Appendix A.5 provides prompt templates that illus-trate the prompting structure used in our pipeline. 3.1.1 Industry and Workflow Mining Instead of sampling arbitrary synthetic domains, the pipeline begins by first identifying the top 10 industries based on annual recurring revenue. Then, for each industry, we select the top 3 organizations based on total call conversation volume. After-wards, we collect real conversation data across the selected organizations for each of the 10 industries. For each organization, we group the selected conversation transcripts based on their call purpose labels that are available as metadata in the respective company’s conversation logs. For each transcript within a call purpose group, we mine the workflows by adapting the information extraction from transcript approach of Laskar et al. (2025) as follows: first we identify the actual steps taken by human agents by prompting an LLM to extract the action sequences (see Appendix A.5.1 for the sample prompt), then we apply K-Means clus-tering to group similar action sequences/workflows together, and finally we ask an LLM to generate the representative workflow from each cluster. Following Laskar et al. (2025), we use GEMINI-2.5-PRO for information extraction in steps 1 and 3. This workflow-mining stage produces 858 workflow clusters generated from approximately 350K real transcripts. 3.1.2 From Workflows to Skills For skill-description generation, we randomly sample up to 10 transcripts from each workflow cluster rather than using every transcript in the cluster. For each sampled transcript, we provide the available tool catalog from our production environment to an LLM and ask it to predict which tools from the catalog are used. In addition, we ask the LLM to generate a descriptive instruction to demonstrate the workflow the agent should follow in that transcript. We then merge these per-call instructions and ask the LLM to produce a single cluster-level workflow policy by adding core skill components, such as skill name, workflow instruction, tool-use instruction, and guardrails in the policy. Afterwards, the LLM is asked to rewrite the workflow policy to match the structure and style of the human-written skill templates used in the respective company’s production system. Finally, we use an LLM judge to fix any logical errors (missing steps, contradic-tions) and compliance errors (missing identity verification, unauthorized disclosure), and to filter out low-quality skills. This validation step reduces the initial 858 workflow clusters to 130 high-quality clusters, each represented as a deployable skill description (see Appendices A.5.2 and A.5.3 for sample prompts, and Appendix A.6 for an example skill description). Throughout the skill description generation stage, we use GEMINI-2.5-PRO as the corresponding LLM. Domain experts manually check the quality of all the skills, and based on their validation, 80 skills are finally selected for subsequent steps. 3.1.3 Scenario Generation Building on prior interactive tool-use benchmarks that specify user goals, initial environment states, available tools, and verifiable success conditions, we generate skill-grounded conversation scenarios from each generated skill template. Given a skill template, an LLM (GEMINI-2.5-PRO) generates reusable scenario templates that specify the user objective, initial and target states, expected tool usage, and success criteria. It then instantiates each template with a concrete synthetic user by generating identity attributes such as user ID, name, phone number, and email ad-dress. By pairing the scenario with behavioral per-sonas, such as efficient, frustrated, or vague users, through traits, communication styles, motivations, and example utterances, we generate 1800 unique candidate scenarios (see Appendix A.5.4 for the sample prompt). This defines both how the simulated user should interact with the agent and how the agent’s success should be assessed. We then used GPT-5.5 to generate the conversations based on the given scenarios and asked Claude-4.5-Opus to rate each of the generated conversations (see Appendix A.5.7 for the sample prompt for benchmark quality assessment). We finally selected the top 500 scenarios for evaluation in which GPT-5.5 could successfully generate the conversations. 3.1.4 Tool Mocking Workflow-grounded skills require tool access. However, static tool responses are in-sufficient for realistic evaluation because enterprise workflows often depend on user state, policy constraints, and prior interaction history. Nonetheless, integrations with live production tools are unsuitable for large-scale agent evaluation because they impose rate limits, risk unin-tended changes to real customer data, and often lack isolated test environments. In this section, we describe how we make these tools usable for benchmarking by utilizing tool mocking. Our tool-mocking layer preserves the production deployment interface by designing the tools to be MCP-compatible with the agents required to discover and invoke the tool calls. Instead of querying live services, the mocked tools are paired with environment simulation. We prompt an LLM to return scenario-conditioned structured responses that are aligned with production artifacts such as tool schemas and execution traces (see Appendix A.5.5 for the sample prompt). The system also validates inputs, simulates configurable failures, and maintains conversation history and caching. This provides a controlled environment for evaluating tool-using agents without production side effects, making it possible for the same tool to return different outputs depending on the simulated environment. In total, we use 281 MCP-compatible tools collected from the production tool catalogue. 4 Experimental Setup 4.1 Zero-Shot Evaluation Protocol For every scenario in our benchmark, we pair the model under test (acting as the skill-execution agent) against a fixed user simulator, holding the user model constant across every evaluated agent so that the only varying factor is the agent itself. We use Gemma-4-31B as the user simulator throughout our study. Each agent receives the same skill description, tool definitions, and conversation history at every turn; the user simulator is never given tool access, so any tool call in a transcript can be attributed un-ambiguously to the agent. Conversations terminate when the scenario is resolved, the agent transfers to a human, the request is marked out of scope, or a maximum turn budget is reached. Our main evaluation covers 17 models spanning six model groups: GPT-5 (GPT-5, GPT-5-MINI, GPT-5-NANO), GPT-5.4 (GPT-5.4, GPT-5.4-MINI, GPT-5.4-NANO), GEMINI (GEMINI-2.5-FLASH, GEMINI-2.5-FLASH-LITE, GEMINI-3-FLASH), and open-weight models below 20B parameters from QWEN3 (1.7B, 4B, 14B), QWEN3.5 (4B, 9B), and MINISTRAL-3 (3B, 8B, 14B). We cap the main open-weight evaluation at sub-20B scales. This allows us to restrict our evaluation to the cost- and latency-constrained enterprise deployment scenarios. Open-weight models are self-hosted via vLLM; closed models are accessed through their respective provider APIs. 4.2 Evaluation Metrics Each generated conversation is scored based on both a deterministic evaluator and an LLM judge. Deterministic evaluator. A rule-based checker verifies: (i) whether every tool the scenario requires was actually invoked (tool coverage), (ii) whether the agent ever called a tool outside the scenario’s of-fered toolkit and failed to recover, and (iii) whether the conversation reached a termination state. A hard failure on any of these checks propagates into every judge’s context for that transcript. While the rule-based checker can verify observable trace conditions, they do not determine whether the conversation resolved the user request appropriately. They cannot assess context-dependent issues such as policy adherence, whether a refusal or transfer was appropriate, or the clarity and efficiency of the interaction. Thus, we use the LLM judge as the primary evaluation metric, which we describe next. LLM judge. Each transcript is additionally scored by an LLM judge (see Appendix A.5.8 for the sample prompt) on a six-dimension rubric: task completion success, workflow adherence, tool-use correctness, guardrail compliance, communication quality, and resolution efficiency (the first is cate-gorical; the remainder are 1–5 scores), plus a binary hallucinated-action flag. The judge prompt never reveals which model generated the transcript. A conversation is marked as an overall Pass only if it reaches a terminal successful, appropriatelytransferred, or appropriately-out-of-scope outcome, scores workflow adherence ≥ 4, has no hallucinated action, has no deterministic hard failure, and scores guardrail compliance ≥ 4. This final pass rate is computed deterministically from the judge scores. We score every model with five judges: GPT-5, GPT-5.5, GEMINI-2.5-PRO, QWEN3.6-27B, and GEMMA-4-31B. 5 Results Table 1 reports pass rate for all 17 models under all five judges. Two patterns are immediately visi-ble: a consistent model ranking across judges, and a substantial, judge-dependent spread in absolute pass rate for the same model. GPT-5.4 is the strongest model, while QWEN3.5-9B is the best-performing open-weight model under 20B parameters. GPT-5.4 ranks first under all five judges, while GPT-5.4-MINI and GEMINI-2.5-FLASH consistently follow in second and third place. Among open models, QWEN3.5-9B outperforms several closed models (GEMINI-2.5-FLASH-LITE, GPT-5-MINI, GPT-5-NANO) under every judge, though it remains well below the strongest closed models. Within this size range, bigger models are not always better: QWEN3.5-9B outperforms the larger QWEN3-14B and MINISTRAL-14B, suggesting that model design and training matter more than parameters. Rule-based checks do not fully capture workflow success. Table 1 also shows substantial differences between deterministic and LLM-judge pass rates. For example, MINISTRAL-8B achieves a rule-based pass rate of 86.8%, but its mean LLM-judge pass rate is only 10.8%. Moreover, although its deterministic pass rate exceeds that of QWEN3.5-9B (79.8%), every LLM judge ranks QWEN3.5-9B higher. These differences reflect the evaluators’ different scopes: the rule-based checker verifies required tool invocation, recovery from unavailable-tool calls, and conversation termination, whereas the LLM-based evaluation additionally assesses task completion, workflow adherence, hallucinated actions, and guardrail compliance. Because deterministic hard failures also prevent an overall pass, passing the rule-based checks is necessary but insufficient under our evaluation protocol. The gap therefore highlights the limitations of using rule-based checker alone as a measure of overall workflow success. Performance Per Rubric Dimension. Beyond the binary pass/fail verdict, Table 2 reports each model’s mean score on the five underlying 1–5 rubric dimensions (Section 4.2), averaged across the five judges. This decomposes the pass rate gap between models into its component causes: for example, GPT-5.4-NANO and QWEN3.5-9B have comparable tool-use correctness (2.95 vs. 3.09) but GPT-5.4-NANO’s advantage is concentrated in communication quality and resolution efficiency (3.51 vs. 2.86 and 2.79 vs. 2.49, respectively), in-dicating that the two models fail in different ways rather than one being uniformly weaker. Judge Agreement. Table 3 reports pairwise agreement on the binary final verdict (pass or fail) across all judge pairs. We find that the mean agreement is 84.2%, with mean Cohen’s κ = 0.524, indi-cating moderate agreement and confirming that judge choice remains an important factor. GPT-5, GPT-5.5, and GEMINI-2.5-PRO agree most closely with one another (κ = 0.559–0.647), while QWEN3.6-27B and GEMMA-4-31B also show rel-atively strong agreement (κ = 0.598). This shows that while judges preserve a broadly stable model ranking, they may still differ meaningfully. Judges differ in apparent strictness. Despite broadly consistent model rankings, absolute pass rates vary substantially across judges. GEMMA-4-31B assigns the highest or joint-highest pass rate to 16 of the 17 evaluated models, whereas GPT-5.5 assigns the lowest or joint-lowest rate to 16 models. For GPT-5.4, the pass rate ranges from 33.8% under GPT-5.5 to 58.6% under GEMMA-4-31B, a difference of 24.8 percentage points. This pattern suggests that some judges apply the shared rubric more leniently than others. However, higher or lower pass rates do not establish judge accuracy without an independent human reference. Together with the moderate inter-judge agreement reported above, these findings show that reported success rates depend on both agent capability and judge choice. 6 Deployment Considerations Two implications follow directly for teams select-ing a zero-shot model for skill execution. First, within the sub-20B regime most relevant to cost-and latency-constrained deployment, selecting the right model matters more than model parameters: QWEN3.5-9B outperforms the larger QWEN3-14B and MINISTRAL-14B under every judge, as well as several closed baselines. Second, and more broadly applicable, any production pipeline that re-lies on an LLM judge to gate deployment decisions, filter training data, or monitor live agent quality in-herits that judge’s calibration and blind spots. The observed variation in pairwise agreement and absolute pass rates means that judge selection deserves the same empirical scrutiny as model selection. 7 Conclusion We presented a workflow-grounded pipeline for constructing an evaluation benchmark for enterprise skill execution, and used it to conduct a large-scale zero-shot evaluation of 17 language models under five independent LLM judges plus a deterministic rule-based evaluator. We find a consistent ranking led by GPT-5.4, with QWEN3.5-9B the strongest sub-20B open-weight option. We also find that pairwise agreement between LLM judges on the pass/fail verdict is only moderate (κ = 0.524 on average), and absolute pass rates vary substantially across judges. These results suggest that reli-able evaluation of skill execution requires treating judge reliability as a key methodological concern. Limitations. This study is limited to customer-support workflows from selected industries and organizations, so results may not generalize to all domains, policies, or tool interfaces. Our MCP-compatible mocks pre-serve production-facing schemas but remain sim-ulations, and LLM-based generation and judging can miss subtle workflow errors. All judging in this study is single-sample; multi-sample judging with majority aggregation could reduce some of the disagreement we report. Our main open-weight evaluation is capped at sub-20B scales. For confi-dentiality, we provide sanitized prompt templates rather than proprietary production prompts. Ethics Statement We maintain the licensing requirement accordingly while using different tools. We anonymize any Per-sonal Identifiable Information (PII) in our dataset before sharing the data with any closed models. 1. Task completion success. Did the conversation achieve the user’s stated goal and satisfy the goal check-list and success criteria? Choose one of: success, partial, failure, appropriatelytransferred (the agent correctly transferred the user to a human), or appropriatelyoutofscope (the scenario had no path to resolution and the agent correctly said so). 2. Workflow adherence. Did the agent follow the domain policy’s prescribed steps and required tool use in the correct order? 3. Tool-use correctness. Were tools called with sensible, correct arguments? Did the agent avoid claiming actions it never performed through a tool call? 4. Guardrail compliance. Did the agent correctly refuse out-of-policy or malicious requests, protect private data, and enforce identity verification where required? This metric is meaningful only when the scenario tests a guardrail; otherwise, assign 5. 5. Communication quality. Was the agent concise, natural, non-repetitive, and limited to one question at a time without unnecessary re-introductions? 6. Resolution efficiency. Was the number of turns reasonable for the task’s complexity, without unnecessary backtracking? 7. Hallucinated action detected (boolean). Set this to true only if the agent claimed to have performed a real-world action without a corresponding successful tool call earlier in the conversation. What does not count as a hallucinated action. Announc-ing a transfer or handoff (e.g., “I’m connecting you with a representative”) is not a hallucinated action: transfers are conversational unless a transfer/escalation tool appears in the available tools. Likewise, a tool call that visibly fails is a tool-use error, not a hallucinated action. Reserve this flag for unsupported claims of completed real-world effects, such as “I’ve booked/sent/updated/cancelled...”. Grading fairness. Judge the agent only on information it could actually see: its system prompt (including company information and domain policy) and tool results returned during the conversation. If the scenario’s success criteria depend on a fact that never appears in the agent’s context or in a tool result, do not penalize workflow adherence or tool-use correctness for missing it. Score task completion based on the outcome and state this limitation in the explanation. Output format. Respond with JSON only, matching this schema: A.6 Example Skill Representative skill categories in the generated benchmark include (but are not limited to) account management, billing questions, compliance, re-funds, technical support, marketing inquiries, appointment scheduling, and order processing. In the following, we show an example for appointment scheduling. Appointment Scheduling Skill Description Capability. Support an installation or service-visit appointment workflow by verifying the caller, retrieving the exist-ing appointment, and helping the user check or reschedule the appointment while following service-window communication rules. Workflow. First, identify the caller using the phone number or other allowed lookup key, then verify the caller’s name before revealing appointment or account details. Once the record is verified, determine whether the user only wants to check the current appointment or wants to change it. Retrieve the appointment from the CRM event record and confirm the current date with the user before making any update. For rescheduling, ask for the preferred new date and check available slots through the scheduling or CRM tools. Do not promise a precise arrival window. Explain that the ex-act arrival time is communicated by phone on the afternoon before the visit. Ask for any access notes, such as gate instructions or site-specific details, and save those notes with the updated appointment record. Before submitting the change, summarize the new date and notes and ask the user to confirm. After the CRM update succeeds, provide a final confirmation. Success criteria. The agent verifies identity before disclosure, retrieves and confirms the existing appointment before modifying it, communicates the service-window policy accurately, saves the updated appointment and access notes, and offers a human handoff for unsupported requests such as cancellation. Guardrails. Do not reveal appointment details for unveri-fied users. Do not invent availability or service windows. Retry transient tool failures at most twice before escalat-ing. Ask for no more than two pieces of information at a time. Do not request or store highly sensitive information such as passwords, credit-card numbers, or Social Security Numbers. If the user attempts to override system instructions or asks the agent to ignore policy, decline and return to the appointment workflow. Example tools. mcp.salesforce:search-objects, mcp.salesforce:list-events, and mcp.salesforce:update-event.