Abstract for: How do we know it’s working? Developing measures to gauge systems thinking and system dynamics skills in children and adolescents

The field of systems thinking and system dynamics (ST/SD) has grown with efforts to scale learning earlier in schools. Proponents tout the plasticity of the developing brain and the positive impact of ST/SD education. However, few tools have been published to measure SD skill attainment, which is a barrier to curriculum integration. We proposed a comprehensive, developmentally appropriate survey to gauge SD skills of 5th to 12th-grade students. We developed a hierarchical scale that links technical ST/SD skills to mental models. Based on this framework, we designed both an extended survey and a streamlined three-question version, which can be used as pre- and post-tests to measure shifts before and after learning system dynamics concepts. Initial survey items were developed using AI models and refined through expert review. The three-question measure was taken by six 5th-grade students participating in a system dynamics-based service-learning project at an elementary school in St. Louis, Missouri, USA. Grading by a fine-tuned AI model shows early-stage systems thinking typical for a pretest. The data show the feasibility of quantifying system dynamics skills with survey items. A subsequent post-test will be administered at the end of this school year. Future steps include administering the extended survey to students in higher grade levels, testing each question for construct appropriateness with a Cronbach’s alpha, and making modifications accordingly. The final product may be applied to various experimental designs. Overall, the process demonstrates how to use AI-assisted large language models to develop age-appropriate, comprehensive surveys for measuring students' ST/SD skills. Potential limitations are survey length and literacy disparities. AI is a component of the methodology. Initial survey measures were developed with AI models.