Abstract for: BEAMS: Benchmarking and Evaluating AI for Modeling and Simulation

AI tools to support real-world decision making must be able to build simulation models that inform their recommendations and render them interpretable. AI-enabled tools that can automate aspects of modeling practice must complement human expertise, not replace it. The BEAMS Initiative aims to guide the development of AI-enabled tools for modeling and simulation toward forms that are responsible and ethical by establishing benchmarks for human‑centered modeling and simulation practices. The BEAMS Initiative is based on open digital and organizational infrastructure to collaboratively evaluate AI tools for modeling and simulation. The open source sd-ai project hosted by the BEAMS Initiative establishes transparency and enables contributions from collaborators to be shared broadly. The BEAMS steering group focuses on prioritizing potential benchmarks, while the BEAMS technical group focuses on implementing the benchmarks in the form of automated tests for the sd-ai project. Tests for several distinct categories of evaluation have been implemented and applied to AI-enabled tools that support qualitative model building, quantitative model building, and model discussion. These include tests for causal translation, model iteration, causal reasoning, conformance, model behavior explanation, suggested model building steps, and suggested model fixes. When engines from the sd-ai project are coupled with different LLMs, their performance on these evaluations reveals variability across different AI-enabled tools. The evaluations implemented by the BEAMS Initiative demonstrate that AI-enabled modeling tools perform better at discussion and basic qualitative tasks than with causal reasoning and quantitative error fixing. No single LLM dominates across engine types, highlighting the importance of specific tasks and tradeoffs between speed and accuracy. Ongoing efforts of the BEAMS Initiative aim to incorporate benchmarks that address concerns about bias by considering alternative perspectives and stakeholder‑centered use cases. Subject of study and writing aid for initial drafting