Examining the efficacy of generative artificial intelligence in item generation: comparative analysis of human-developed and AI-generated reading tests

초록

This study examined the utility of questions created with artificial intelligence (AI) compared to those created by human item writers for English reading tests. Despite the rapid adoption of generative AI, such as ChatGPT, in educational settings, there is limited amount of empirical evidence comparing AI-generated and human-generated items based on actual test administration. Given the practical need for empirical validation of AI-generated test items, the present study analysed item difficulty and perceived quality using test data from high school students in South Korea. The findings revealed that AI-generated items were slightly more difficult due to occasional inappropriate language choices, although the difference was not statistically significant. Test-takers generally perceived AI-generated items as comparable to human-developed ones, indicating cautious optimism regarding the integration of generative AI into language test development and preparation.

키워드