SWE-bench Multimodal is a benchmark extending the original SWE-bench with 517 issues containing visual elements like screenshots, mockups, and diagrams to evaluate AI systems' ability to interpret and act on multimodal information. Version 2 refines the benchmark to 480 reproducible tasks with improved testing infrastructure, removing flaky tests and addressing dependency issues.
Two research papers investigate the stability and reliability of LLM APIs. Chauvin et al. present methods for tracking log probability changes in LLM APIs at ICLR 2026, while a related team develops token-efficient techniques for detecting changes in LLM API behavior at ICML 2026.