Use and misuse of case studies and experiments in Game Software Engineering

Context: The growing complexity of the video game industry has led to the emergence of Game Software Engineering (GSE), a sub-discipline concerned with the development and maintenance of video game software. Unlike traditional Software Engineering, GSE is inherently interdisciplinary and places strong emphasis on delivering enjoyable user experiences. While Empirical Software Engineering has achieved increasing methodological rigor, empirical research practices within GSE remain relatively underdeveloped.

Objective: This study analyzes the use and methodological accuracy of the terms case study and experiment in GSE literature, assessing whether published studies correctly apply established empirical research definitions and criteria.

Methods: A systematic literature review identified 48 primary studies in the GSE domain. Each study was examined to determine whether its classification as a case study or experiment complied with widely accepted methodological standards. The evaluation focused on adherence to defining characteristics and essential criteria of each research strategy.

Results: The findings reveal substantial misclassification of empirical research strategies in GSE publications. Only 1 out of 37 (3%) of the studies labeled as case studies met the established criteria, whereas 36 out of 37 (97%) were incorrectly classified. Similarly, only 2 out of 11 (18%) of the studies identified as experiments satisfied all defined experimental requirements. These proportions are considerably lower than those typically observed in traditional Software Engineering research, indicating weaker methodological alignment within GSE.

Conclusion: The results highlight a significant need to strengthen methodological rigor in GSE empirical research. Beyond identifying misapplications, this study provides illustrative examples and practical recommendations to support more accurate use of empirical research strategies. Improving methodological clarity is essential.