Full description
A collection of 357 annotations from 13 climate experts for Climate Adaptation NLP. It contains evaluations of Large Language Model (LLM) capability on climate questions as well as the raw responses from LLMs.This collection is useful for benchmarking models on their generation performance to expert climate adaptation questions and also measuring their evaluation performance in comparison to experts.
Lineage: 3 Language Models (LLMs) were used to generate responses to 50 questions posed by climate experts. LLMs generated responses using climate data/scientific literature, and without, amounting to 300 unique responses.
13 Experts were asked to annotate responses from the different with a set of 7 criteria (annotation guidelines included), which were created by experts. 57 responses were annotated twice by a different expert to measure inter-annotator agreement.
Evaluation criteria can be found in the user testing guide PDF. Each sub-criterion is labelled in lexicographic order (i.e., structure_a refers to the first sub-criterion in the user testing guide for the structure criteria). A score of 1 indicates that an expert believes the response meets that sub-criteria, while a score of 0 indicates the response failed to meet that sub-criteria.
Available: 2025-05-20
Data time period: 2024-12-01 to 2025-01-31
Subjects
Artificial Intelligence |
Climate Change Impacts and Adaptation |
Climate Change Impacts and Adaptation Not Elsewhere Classified |
Environmental Sciences |
Information and Computing Sciences |
Natural Language Processing |
climate |
climate nlp |
expert evaluation |
generation |
large language model |
llm |
llm as judge |
nlp |
question answering |
User Contributed Tags
Login to tag this record with meaningful keywords to make it easier to discover
Identifiers
- DOI : 10.25919/X5WQ-N705
- Handle : 102.100.100/706136
- URL : data.csiro.au/collection/csiro:65515
