Supplementary Materials and Data for a Domain-Based Evaluation of GenAI Chatbots in Language Education

Full description

This record provides the study materials and scoring data for a domain-based evaluation of 13 general-purpose GenAI chatbots across reading, writing, listening and speaking in German.

The record contains two files:

1. Supplementary Materials (Appendices A–C): the study prompt set and follow-up instructions, analytic rubrics with indicator-level anchors, and documentation of task inputs and source materials. Shared inputs and their application across chatbot–skill–domain combinations are identified in Appendix C and Table C1.
2. GenAI_Chatbot_Evaluation_Data.xls: dimension scores for each of the three runs in the Run_scores worksheet, paired independent ratings used to calculate the intraclass correlation coefficients (ICCs) in the ICC_matrix worksheet, and scoring and calculation notes in the Data Documentation worksheet.

Third-party listening recordings, reading passages and accompanying questions are not reproduced in this deposit for copyright reasons. Source information and official access links are provided in Appendix C. The availability of other media inputs is documented separately in the Supplementary Materials.

The deposited scores support recalculation of the reported summaries and provide a descriptive baseline for future comparisons under comparable testing conditions.