Abstract

Abstract for: How do we know it’s working? Developing measures to gauge systems thinking and system dynamics skills in children and adolescents

Introduction The field of systems thinking and system dynamics (ST/SD) has grown with efforts to scale learning earlier in schools. Proponents tout the plasticity of the developing brain and the positive impact of ST/SD education. However, few tools have been published to measure SD skill attainment, which is a barrier to curriculum integration. We proposed a comprehensive, developmentally appropriate survey to gauge SD skills of 5th to 12th-grade students.
Approach We developed a hierarchical scale that links technical ST/SD skills to mental models. Based on this framework, we designed both an extended survey and a streamlined three-question version, which can be used as pre- and post-tests to measure shifts before and after learning system dynamics concepts. Initial survey items were developed using AI models and refined through expert review.
Results The three-question measure was taken by six 5th-grade students participating in a system dynamics-based service-learning project at an elementary school in St. Louis, Missouri, USA. Grading by a fine-tuned AI model shows early-stage systems thinking typical for a pretest. The data show the feasibility of quantifying system dynamics skills with survey items. A subsequent post-test will be administered at the end of this school year.
Discussion Future steps include administering the extended survey to students in higher grade levels, testing each question for construct appropriateness with a Cronbach’s alpha, and making modifications accordingly. The final product may be applied to various experimental designs. Overall, the process demonstrates how to use AI-assisted large language models to develop age-appropriate, comprehensive surveys for measuring students' ST/SD skills. Potential limitations are survey length and literacy disparities.

Introduction	The field of systems thinking and system dynamics (ST/SD) has grown with efforts to scale learning earlier in schools. Proponents tout the plasticity of the developing brain and the positive impact of ST/SD education. However, few tools have been published to measure SD skill attainment, which is a barrier to curriculum integration. We proposed a comprehensive, developmentally appropriate survey to gauge SD skills of 5th to 12th-grade students.
Approach	We developed a hierarchical scale that links technical ST/SD skills to mental models. Based on this framework, we designed both an extended survey and a streamlined three-question version, which can be used as pre- and post-tests to measure shifts before and after learning system dynamics concepts. Initial survey items were developed using AI models and refined through expert review.
Results	The three-question measure was taken by six 5th-grade students participating in a system dynamics-based service-learning project at an elementary school in St. Louis, Missouri, USA. Grading by a fine-tuned AI model shows early-stage systems thinking typical for a pretest. The data show the feasibility of quantifying system dynamics skills with survey items. A subsequent post-test will be administered at the end of this school year.
Discussion	Future steps include administering the extended survey to students in higher grade levels, testing each question for construct appropriateness with a Cronbach’s alpha, and making modifications accordingly. The final product may be applied to various experimental designs. Overall, the process demonstrates how to use AI-assisted large language models to develop age-appropriate, comprehensive surveys for measuring students' ST/SD skills. Potential limitations are survey length and literacy disparities.