Peacekeeping leadership and military intelligence units rely heavily on automated systems to generate situation reports and community engagement frameworks. This widespread integration poses direct risks because deployed systems rarely receive oversight from understaffed, under-resourced gender advisory teams. To counter these systemic omissions, Our Secure Future and the Collective Intelligence Project introduced the first public benchmark to measure model adherence to international security commitments. Raising baseline model performance would mitigate cognitive offloading and enable human advisors to direct their attention toward high-level political analyses.
Key points
Peacekeeping personnel and defense agencies are increasingly adopting automated systems t…: Peacekeeping personnel and defense agencies are increasingly adopting automated systems to prepare conflict reviews and operational strategies.
Commercial models default to gender-blind perspectives that exclude more than half the gl…: Commercial models default to gender-blind perspectives that exclude more than half the global population from security analyses.
Enhancing baseline model standards allows under-resourced gender advisors to focus on ess…: Enhancing baseline model standards allows under-resourced gender advisors to focus on essential political analysis and local community engagement.
Sources checked by our analysis pipeline: 1
Collective Intelligence Project (CIP) has been hard at work on bringing collective intelligence into AI evaluation. Over the past few months, we have fleshed out our participatory methodology and the Weval platform to elicit expertise from domain experts that we can translate into criteria for higher-quality evals. Through partnerships with coalitions and civil society, we are focused on publishing social impact evals that measure the performance of AI in fields with large-scale implications for society.
Below, Moira Whelan and Jesper Frant of Our Secure Future (OSF) detail how they used Weval to develop a benchmark on the ability of LLMs and AI tools to follow Women, Peace and Security (WPS) principles in global peacekeeping.
Our research has found that LLMs don’t reflect reality – systematically failing to consider the life experiences of more than 50% of the global population – the majority of whom are in the Global South. We call this the WPS Competence Gap, and it is why OSF built the WPS AI Leaderboard around the first public AI benchmark of its kind. The latest findings validate and…
Block has developed Buzz, an open-source collaboration platform that allows employees and AI agents to work together in a shared workspace. The system lets users create specialized agents, select their underlying models or coding harnesses, and invite them directly into one-on-one or group discussions alongside colleagues. Instead of toggling between isolated AI tools, teams use Buzz for everyday tasks such as coding and analytics, supported by an environment that can be self-hosted privately on an organization's…
Enterprise teams can now interact directly with Databricks data intelligence without leaving their primary chat channels. The integration brings source-cited responses, automated actions, and scheduled data analyses straight into collaborative conversations to speed up operational decisions. Strong permission safeguards ensure that sensitive assets remain restricted based on the querying user's established access profile.
Enterprise artificial intelligence teams are adopting structured frameworks to reliably deploy complex systems that orchestrate tools, internal knowledge, and external APIs. Conventional continuous integration and unit testing fall short because of non-deterministic behavior and independently shifting external dependencies. To guarantee consistent performance, organizations are instituting rigorous evaluation cycles anchored by business-approved gold standards.