Initial and final keyword lists

Full description

The final set of publications was then scanned for mentions of technologies in order to establish a baseline collection of technology keywords. Again, the corpus was too large to complete examine manually, so the Mistral LLM was used to search for mentions of technologies. The LLM was given the title and abstract of each publication in the corpus in turn with instructions to “extract any technologies mentioned in those papers” (full system prompt in section A.1). For the purposes of this analysis, I considered a technology to be any of: programming languages, software tools/libraries, hardware or hardware platforms, and named conceptual or algorithmic frameworks the authors applied as part of their method. I took a broad stance for the definition of technology with the dual intention of a) not eliminating anything that could be useful for further analysis and b) being able to manually review and cull the list generated by the LLM. The 17,476 results returned by the LLM did feature a large number of duplicates, either as explicit duplicates, or as ontological duplicates or subsumption redundancies; as well as spurious entries arising as either a misidentification of a technology by the LLM or from publications that passed the previous filter step but were not dealing with automated assessment of training and education. I used the Python’s rapidfuzz with a conservative similarity setting of 90% and regular expressions libraries to condense duplicates and near duplicates (plurals, acronyms, and highly similar words) into single entries, resulting in 11,560 technology keywords. I then culled spurious entries by removing all items with less than 20 mentions across the corpus, reducing 11,560 keywords to 1,907. With the keyword list now tractable for human review, I then manually removed any keywords unrelated to automated assessment or too broad to be useful, resulting in a final list of 455 technology keywords relevant to automated assessment.

The final step was to flatten subsumption redundancies and create a taxonomy that mapped the keywords to a useful classification system for analysis. I used a set of mutually exclusive hierarchical categories with a most-specific-level coding rule (assigning each item to the most specific category it qualifies for, falling back to a broader/residual category if no finer one applies) with NEC (Not Elsewhere Classified) residual categories at each level of generality, to code the 458 technology keywords into 13 categories deemed useful for our analysis: LLM, Natural Language Processing, Deep Learning, Machine Learning, Artificial Intelligence, Rules-Based Systems, Statistical Assessment Tools, Educational Data Mining, Blockchain, Intelligent Tutoring Systems, Massive Open Online Course, Multiple Choice Questions, and General Automated Assessment Concepts.