Latest / UpNext AI / AI Governance at the UN, Systemic-Risk Evidence, and Agent Safety | UpNext AI – September 24, 2026
Transcript
- Welcome to the UpNext AI podcast. It's Thursday, September 24th, 2026, and here's what matters in AI today. OpenAI CEO Sam Altman addressed the United Nations Security Council this week with a broad argument about who should control increasingly capable AI systems. In remarks published by OpenAI, Altman said AI could expand opportunity, creativity, and scientific discovery—but warned that more autonomous systems could move faster than institutions, concentrate power, or make decisions people cannot understand or control. His central message was that human control cannot be treated as a vague aspiration. Altman argued that developers should not build systems unless they can make an extremely strong case that those systems can remain aligned, monitored, and under human control. He also said companies should not accept excessive technological risk merely to win a competitive race. The policy prescription was international coordination without a single global authority overriding national governments. Altman called for complementary national and international standards to measure capabilities, assess risks, judge safeguards, and preserve meaningful human oversight. He also proposed faster incident-reporting protocols and secure channels for governments, critical-infrastructure operators, and technical experts to share emerging threats. The significance is less a new agreement than the venue and the framing. A leading AI lab is placing standards, evidence, and oversight at the center of its appeal to the Security Council—while acknowledging that decisions about powerful AI cannot be left to labs alone. That focus on evidence leads to a more operational question: once a safety claim is made, how can anyone inspect the basis for it? A new arXiv paper proposes what it calls the Systemic Risk Index: an open evaluation pipeline and dashboard designed to make evidence behind AI-risk claims more transparent and traceable. The researchers organize 19 public benchmarks around four risk categories used in the European Union’s general-purpose AI code of practice: chemical, biological, radiological, and nuclear risk; cyber offense; harmful manipulation; and loss of control. Rather than offering one static model score, the dashboard lets users compare average and worst-case ways of aggregating risk, adjust how model capability affects the total, and trace a rating back to benchmark evidence. Across 18 models, the paper reports that scores fell by 14 to 37 points under worst-case aggregation, suggesting that average scores can conceal more concerning cases. The authors also report that automated judges agreed with human graders at levels comparable to agreement between human graders, and that a blind audit found most sampled benchmark transformations preserved the original harm. This is a preprint, and the paper does not establish that the dashboard improves real-world regulatory decisions. Still, it points toward a useful norm: risk reporting should show its work. OpenAI is also making a more practical bet on AI readiness: training people to use the tools in daily work. The company marked two years of OpenAI Academy, which offers training for uses ranging from teaching and small-business operations to software development. OpenAI says the program has hosted more than 250 events and that more than 4 million people have engaged with Academy content since its September 2024 launch. The new step is a Community Trainer Program. Partner organizations can nominate staff to learn the Academy curriculum and lead practical workshops in their own communities, after completing training and a facilitation assessment. OpenAI says the effort is aimed at schools, workforce groups, small-business networks, and community organizations. The Academy has also added learning paths for knowledge workers, developers, leaders, educators, and college students, with course badges for learners who pass assessments. The underlying idea is straightforward: access to an AI model is not the same as knowing when to trust it, how to fit it into a workflow, or how to review the output. OpenAI’s program is company-led, but its emphasis on practice, peer learning, and local trainers reflects where AI-skills efforts are heading. For the research note, the challenge is not simply detecting a dangerous agent action. It is intervening before a sequence of individually plausible actions becomes harmful. Researchers behind PASTABench built a benchmark of 1,139 multi-turn agent trajectories across five risk categories and 13 subcategories. Their approach separates three questions for a safety monitor: whether to intervene, when to intervene, and what kind of risk is developing. They call the target interval the Optimal Intervention Window—the period between the earliest meaningful warning signal and the point where intervention should trigger. Testing 16 language models, the paper found proactive monitoring remains largely unsolved. The best model made optimally timed interventions on only 40.74 percent of the benchmark trajectories. The researchers also found that some smaller models appeared competitive until hazard-related words were neutralized, indicating sensitivity to keywords rather than a robust understanding of the developing risk. It is an arXiv preprint, and benchmark performance is not the same as deployment performance. But the takeaway is crisp: evaluating each agent step alone is not enough. Safety systems need to reason over the unfolding trajectory and act before the final bad outcome. ...Are you building apps with voice? Elevate your app's voice capabilities with ElevenLabs. Their API is a game changer for embedding dynamic, responsive voice interactions in your applications, providing unprecedented realism, flexibility and latency. In fact, you're listening to one of their voices - right - now. If you are a developer looking to elevate user experience with natural voice interfaces, this is your solution. Visit up next dot fm slash eleven to check out their latest offerings. ... MuleSoft introduced an A2A Bridge for Salesforce Agentforce in its Agent Fabric platform. The company says it can expose existing Agentforce agents as Agent2Agent-compatible services without changing their code, adding a common layer for orchestration, authentication, observability, and policy controls. AWS published a guide to moving AI agents from pilot to production using Amazon Bedrock and AgentCore. The pitch is managed runtime, memory, and tool connectivity around agent logic, with support for several agent frameworks and both frontier and open-source models. TechCrunch reports that concern over AI safety and rogue agents is helping drive investor interest in security startups and lifting cybersecurity stocks. The broad implication is that security spending is increasingly being framed around protecting AI-native systems, not just traditional networks. And Wired reports that Meta issued a fix for a zero-day vulnerability in its Muse AI assistant. Meta said the flaw could have allowed attackers to take broad control of a victim’s Mac—a reminder that agent-like helpers need especially tight permissions and rapid patching. Before we wrap up, a quick note: this podcast is generated with the assistance of AI and is intended for informational purposes only. All referenced articles, research, and commentary remain the property of their original authors and publishers. If you enjoyed this episode, don't forget to subscribe, rate, and leave us a review! And that's your briefing for today. Full source links are in the episode notes, and we'll be back tomorrow with what's up next!