Gaming is a highly effective tool for management training and education. Management games work as catalysts in the processes of group decision making. They counteract narrow specialization, lead to improved communication between different corporate functions, and encourage the identification and the pursuit of shared values and overall objectives. Management practitioners and students confirm that these learning environments increase the motivation to try new ways of behavior and to improve the understanding of complex decision problems. However, especially with respect to the effectiveness of management simulators, objective data about their effectiveness are missing. In a series of investigations it was tried to get a more reliable picture, how performance and the use of management simulators correlate with each other. The research design varies between a simple questionnaire and an elaborate investigation, how the participants react under several sets of circumstances.
To test the hypothesis of a positive relation between the availability of a management simulator and the achieved performance it was first investigated, how knowledge about systems in general and System Dynamics in particular play a role with respect to the intensity of use of the management simulator LEARN! and the quality of the insights gained from it.
Besides LEARN!-a typical management flight simulator (see e.g. Graham/Morecroft/ Senge/Sterman 1992 for a definition)-a multi-person Corporate Planning Game called LOBSTER (an acronym from Learning Organization By Simulating The Economic Reality) is used in academic education and corporate management training since many years at Mannheim University. At the beginning of the game, all corporations have a product with the same level of technological sophistication and with the same market share.
The game deals with the processes of Research and Development, the time-to-market, and the time-to-volume for new products. It focuses on the substitution between different generations of innovative products. It requires decision inputs for all classical fields of corporate management, i.e. budgeting and resource allocation for R&D and advertisement, investment in production capacity, and the way to finance it, personnel recruitment, etc. The timing of market introduction, investment and production planning, cost management, pricing policies, product quality, and delivery delays are key control variables in these processes (Milling 1996).
During the game, each group of players is confronted with interrelated decision making requirements. It is difficult to understand intuitively, how the decisions interact with each other, how the competitors and the whole system will react. To improve their market performance, the group must identify and collect relevant information. The team members must derive alternative courses of action and evaluate their expected consequences. A feeling for complex system behavior should be gained (Vennix 1990) and our experience supports this expectation.
The simulator LEARN! (for Learning Environment in an Artificial Reality Network) is used to investigate the particular market dynamics, especially to support decision making in the artificial reality of our corporate gaming environment. In a man-machine dialog it allows to test the market response of different courses of actions. Similar to LOBSTER, LEARN! deals with the processes of Research and Development, the time-to-market, and the time-to-volume for new products. It requires decision inputs for the same classical fields of corporate management as LOBSTER. The difference to the LOBSTER-Game-which in the context of the flight simulator is interpreted as the „reality„-are the missing direct actions and reactions of the competitors. They are included as endogenous model variables.
Two groups (out of four) can use the simulator to investigate the expected consequences of their actions. The other two rely for their analyses only on conventional tools like spreadsheets. Different behavior modes and different performance of the groups with and without simulator should be expected. This hypothesis, however, is only partly confirmed. Significantly different modes of behavior or profit performance could not be confirmed (see Maier and Strohhecker 1996). In addition to this research design the students had to answer in each gaming period several questionnaires because we wanted detailed information about their aspirations, motivation, contentment, and intensity in using LEARN! as a decision support tool.
Although LEARN! can be seen as an ideal decision support system for the virtual reality of LOBSTER, it was not used very intensively during the sessions or in preparation for the sessions. In the contrary, on an average the students used LEARN! only for 1 hour, and during this time they did only one single simulation run. As the main reason for such a rare use, it was frequently mentioned, that the reality, i.e. LOBSTER, is complex and requires most of the time for analyzing the results and discussing future decisions. No time was left, to apply other means of analysis or support.
It could be assumed, that the players did obviously not recognize the effectiveness of LEARN! as a transitional object for argument and debate. They might have failed to appreciate the power of such a tool, to improve the understanding and to analyze the consequences of their decisions. However, the analysis of the filled questionnaires showed a somewhat different picture.
In the first round of the game, the students were asked for their expectations about the potential of LEARN! and in the last round they had to evaluate its effectiveness on the basis of the experience they gained. As shown in Figure 1, most of the students expected, that the use of LEARN! would lead to a better understanding of LOBSTER, enhance insight in the interrelations of this complex system, allow the investigation of decision consequences. All this would cause improved decisions.
| The students had to give their view of the following statements:
Please indicate to what extent you agree or disagree: (1 = I fully disagree; 6 = I fully agree) | ||||
| Statistics: Statements: | Mode | Mean | Std. Dev. | N |
| I expect better understanding | 5 | 5.25 | .50 | 4 |
| I did get a better understanding | 5 | 4.56 | .70 | 18 |
| I expect enhanced abilities to analyze consequences | 6 | 5.75 | .50 | 4 |
| LEARN! did enhance my abilities to analyze consequences | 2 | 3.22 | 1.17 | 18 |
Students strongly agreed, that they got an improved understanding by using LEARN! But they did not see it as a tool for the analysis of decision consequences. They expected the simulations of LEARN! to start with the actual initial conditions of the management game, i.e. to act as a decision tester, based on the current data of their virtual reality, which clearly is not the intention of the simulator. The use of LEARN! should lead to a more „general„ learning in and about complex systems, and therefore to improved decision effectiveness.
To investigate whether a better understanding of the complexity and the dynamics of the systems was actually achieved, they were asked each period to estimate the development of some of the model variables like next quarter's sales volume, profits, and market share. These estimations were compared with the actual results. The goodness of fit between expected and achieved outcome serves as an indicator whether the groups with access to LEARN! had an improved understanding of system reaction.
Due to the small sample size, the results, the validity and the reliability of the statistical results estimations have only preliminary character. They seem, however, to indicate a trend. Figure 2 shows some of the values: the mean error of the groups estimations, the standard deviation and the standard error for the variables sales volume and market share together with the t-statistics. The results show a slight tendency that company 1, which did use the management simulator LEARN! for more than 14 hours, has a better understanding of the consequences of their decisions. Their estimation errors are significantly smaller than those of company 2 and 3.
Compared with company 2-which was the best performing company and did not have access to LEARN!-the first company's estimations are only insignificantly better. However, since the second company had access to detailed data from the management game of the year before, this group can not serve as a reference group. Company 2 used the information and just applied the strategy which has been proven successful in an earlier round of the game. In some sense, this group learned from failures of their ancestors, but it is doubtful, that they managed to utilize their data for a better understanding of the game's dynamics. The goodness of fit between estimated and realized sales volume and market share supports this hypothesis although the measures are not significant.
Due to the small sample, the results are not statistically valid and reliable, and the ongoing research efforts try to get a broader sample size and improve the acceptance of LEARN! Management flight simulators could improve decision effectiveness, and tools that allow insight into model structure-like Vensim's causal tracing facility-might lead to even better understanding.
Graham, A. K.; J.D.W. Morecroft; P. M. Senge and J. Sterman 1992: Model-supported Case Studies for Management Simulators. In: European Journal of Operational Research, Vol. 59, pp. 151-166.
Maier, F. H. and J. Strohhecker. 1996. Do Management Flight Simulators Really Enhance Decision Effectiveness? System Dynamics ´96. eds. George P. Richardson and John D. Sterman. Volume 2, Cambridge, MA. pp. 341-344. Also available through: http://iswww.bwl.uni-mannheim.de/pr/sd96/index_ab.htm.
Milling, P. M. 1996. A Management Simulator to Support Group Decision Making in a Corporate Gaming Environment. System Dynamics ´96. eds. George P. Richardson and John D. Sterman. Volume 2, Cambridge, MA. pp. 369-372.
Vennix, J. A. C. 1990. Mental Models and Computer Models. Design and Evaluation of a Computer-Based Learning Environment for Policy-Making. Diss. University of Nijmegen.
ISDC '97 CD Sponsor 