Hand coded publication classifications and model classifications

Full description

This query returned a large number of results using ‘automated assessment’ in an alternative sense, such as in medical imaging, psychometry, and structural engineering analyses. This volume of results was far too large to parse by hand, so I used the Mistral LLM [16] to identify from context how the term automated assessment is being used in each work. In order to ensure the LLM was capable of identifying publications dealing with automated assessment of training and education, I first hand-coded 330 publications based on title and abstract as either ‘on-topic’ (i.e. dealing with ‘automated assessment’ in an education or training context), ‘off-topic’ (i.e. dealing with ‘automated assessment’ in an alternative context), or ‘unknown’ (i.e. classification cannot be determined from title and abstract alone) as an evaluation set. Hand-coding resulted in an evaluation set of 153 ‘on-topic’ publications, 153 ‘off-topic’, and 24 ‘unknown’. This set was then submitted to the mistral-large-latest model. The model classified 239/330 (72.4%) correctly. This may seem an unusable accuracy, but a further breakdown shows the model classified 146/153 (95.4%) ‘on-topic’ publications correctly – the inaccuracy overwhelmingly stems from ‘false positive’ classification of ‘off-topic’ (88/153 classified correctly) and ‘unknown’ (0/24 classified correctly) publications into the ‘on_topic’ category. This means that the LLM could be used as first-pass filter with confidence that it would not artificially exclude publications that should be in the analysis.