Project - eReefs phase 4, AIMS eReefs Visualisation Platform - 2018 - 2020

Brief description

This record describes the AIMS component of the eReefs phase 4 research program that ran from 2018 - 2020. The primary focus of eReefs phase 4 was to transition from research focused projects to operational services. The eReefs work was extended in the eReefs 2020-2024 project. eReefs is a collaboration between the Great Barrier Reef Foundation, CSIRO, the Australian Institute of Marine Science, Bureau of Meteorology, and Queensland Government. It aims to develop a platform that provides a picture of current and historical environmental conditions on the Great Barrier Reef. eReefs has many components developed and maintained by each of the organisations in the collaboration, including catchment modelling (Queensland Government), remote sensing (BOM and CSIRO), hydrodynamic modelling (BOM and CSIRO) and biogeochemical modelling (CSIRO). AIMS's contribution was providing data agregation and visualisation services and well as analysis of the biogeochemical model for inclusion in the Reef Plan report card. A major goal of the AIMS eReefs phase 4 project was to re-engineer the AIMS eReefs platform to use distributed cloud service components to make a platform that can scale its computing resources based on demand, to allow more visualisation and aggregation products to be supported in the future. The eReefs model data is very large (>15 TB) resulting high computation and storage costs associated with its processing. The new re-engineered architecture improve the computational cost efficiency by ~10x and the storage cost efficiency by 1.5x allowing a more complete set of visualisation and aggregation products to be made available. Key goals of this project were: Generating updated water quality scores based on eReefs BGC data for the Reef Plan report card based on methods developed in NESP TWQ 3.2.5. Re-engineering the AIMS eReefs platform for improved scaling. Developing a data extraction tool to allow easy access to time series data from eReefs data. The data for AIMS eReefs platform is stored on the Amazon cloud in S3 storage, managed by the AIMS Knowledge Systems team. All the data is derived from the original eReefs model data. The software was deployed on the Amazon Cloud, using AWS Batch to dynamically adjust the number of executing servers performing aggregation and animation tasks based on load. Coordination tasks are performed using AWS Lambda functions, which communicate using SNS messaging. The system coordinates the activities of all the executing tasks using a central MongoDB database. The setup of the AWS infrastructure is deployed using CloudFormation, which is a language for describing cloud computing infrastructure as code. The configuration of all the generated products (aggregation and animation) along with the infrastructure code is stored in the ereefs-definitions Git code repository. This repository has restricted access for security reasons. All other code will be made publicly available under an open source license by June 2021 as part of the eReefs 2020-2024 project. All the software for the platform is contained in 37 Git code repositories stored on GitHub. These repositories are linked to from https://github.com/open-AIMS/ereefs-documentation. This platform produces aggregation and animations of the CSIRO Hydrodynamic (GBR4 and GBR1) and BioGeoChemical eReefs models. The platform downloads model data from the National Computing Infrastructure (NCI) THREDDS data service. This data is then preprocessed to remove unused variables and half the depths, retaining only depth data from < 145m. This subsetting reduces the total storage needed by the system. This subsetted mirror is then produced to produce a range of temporal aggregations (daily, monthly, annual) and exposure products. The derived products are regridded from the original curvilinear grid used in the modelling to a regular rectangular grid. This regridding is performed to allow the data to be compatible with GIS software such as ArcMap and QGIS. All model data and derived model data is stored in NetCDF file format.

Lineage

Maintenance and Update Frequency: daily

Statement: In the phase 3 of eReefs (2015 - 2018) the AIMS Knowledge System team developed an aggregation and visualisation platform that provides quick access to the eReefs model outputs. This was implemented using two new custom software tools (ncAggregate and ncAnimate) developed in Java. These tools were run on a single large server on the Amazon cloud. This system needed to process and visualise the backlog of 5+ years of model data, but also support daily updates in near-real time. Reprocessing the backlog took months, during which the near-real-time updates needed to be halted. The key problem was the difficulty in balancing the high peak computing demands during the processing of the historic backlog data with near-realtime updates. These issues resulted in a significant cap in the number of aggregation and visualisation products that could be produced for a reasonable computing budget. These limitations resulted in a high operating cost for the services leading to the need to re-engineering the platform in the phase 4 project.

Notes

Credit
Lawrey, E., Smith, A., Hammerton, M., Lafond, G, Logan, M.

Credit
Funding body: eReefs phase 4, Great Barrier Reef Foundation, Australia