Semi-automated Shallow Marine Mask for Northern Australia and GBR Derived from Sentinel-2 Imagery (NESP MaC 3.17, AIMS)

Full description

This dataset represents a comprehensive mapping of shallow marine areas across Northern Australia and the Great Barrier Reef, encompassing intertidal zones, shallow subtidal habitats (down to approximately 5 meters depth in turbid waters), and offshore reef features visible at depths of up to 40 meters in clear waters. Derived from Sentinel-2 composite imagery, it identifies benthic areas where seafloor features are visible in the satellite imagery, providing a mask for mapping of marine habitats from Sentinel 2 imagery. Covering a broad geographic extent from Western Australia to the east coast of Queensland, including remote territories such as Christmas Island, Cocos (Keeling) Islands, Lord Howe Island, and Norfolk Island, this dataset serves as a critical input for reef boundary mapping and shallow water habitat modelling. This dataset provides an essential intermediate step for applications in reef boundary mapping and shallow habitat classification. For reef boundary mapping it is intended to be combined with separate manual mapping of the reef boundaries to estimate shallow soft sediment areas (shallow areas that are no reefs). For habitat mapping the intended application is to use this dataset to removing areas where the visual benthic clarity is insufficient for effective habitat mapping. The dataset provides four levels of mask sensitivity, to allow researchers to choose the mask that best matches their research objectives. These levels of sensitivity represent the change in the strength of the visual signal relative to the surrounding water. While some manual data cleaning has been applied to this dataset (via the Cleanup-remove-mask) these focused on correcting false positives due to cloud anomalies in the satellite imagery. No manual corrections have been applied to the semi-automated boundaries. This dataset was initially developed to identify and map reef boundaries and as a result the detectors were optimised for mapping reefs across a wide range of water types and reef depths. We found that the automated mapping was useful for detecting reefs that we had missed in manual mapping, but found the mapped boundaries were not as good as manual mapping and so we ended manually mapping the reef boundaries and using the workflow in this dataset as a cross checking process to ensure reefs were not missed. For this cross checking process we used a small smoothing filter of 9 pixels to ensure small reefs were detected. The downside of this small filter size is significant noise in the coastal features. In the published version of this dataset we shifted the focus to coastal mapping, with the goal being that the dataset should map enough of the subtidal region to connect fringing reefs to their associated island. To ensure the nearshore subtidal zone was mapped cleanly a larger smoothing filter of 21 pixels was used (noise_reduction_median variable in https://github.com/eatlas/AU_NESP-MaC-3-17_AIMS_Shallow-mask/blob/main/05-create-shallow-and-reef-area-masks.py). This results in sharp features having a rounding of approximately 100 m and features smaller than this not being detected. Dataset production: This dataset was created by comparing the satellite composite imagery against an its surrounding water. The water estimate was created using an initial manual rough mapping of the shallow areas and reefs. These mask areas were in-filled with interpolation from surrounding water, using a large gaussian blur, followed by a full image blur to smooth out any unmasked shallow areas. The shallow features were then mapped from multiple detectors tailored for inshore, midshelf and offshore mapping. These detectors are adjusted to manage the varying water conditions from very turbid to clear water. The detectors were based on the magnitude of the difference between original satellite imagery and the water estimate for a select number of the visual bands. For example in offshore the blue channel from the all tide composite imagery provided the strongest signal, whilst for inshore turbid areas the red and green bands of the low tide image composite imagery provided the cleanest signal. Automated masking derived from the water estimate was used to spatially mask the different detectors to regions where they performed best, significantly reducing then level of false positives. Four variants of the dataset were produced by varying the sensitivity of the detectors. Each variant is produced from a combination of two levels of sensitivity. One to determine which features should be included in the map and one to determine the boundary of the polygons. This split approach helps to ensure the boundaries correspond to slightly deeper, more complete, feature boundaries, without introducing additional false positive features that are just at the edge of detection. In the naming of the variants the first part of the name is the sensitivity level used for the boundaries and the second the sensitivity used to determine what features are included. For example High-Medium indicates that the high sensitivities were used to map the feature boundaries and the medium sensitivity level was used to determine the features to include in the mapping. The sensitivity levels were spaced with a geometric increase in detector sensitivity of 1.5 times between levels (Very Low, Low, Medium, High, Very High). In addition to the semi-automated shallow masks this dataset also provides the python code used to perform the processing, and the input data created to setup and reproduce the final datasets: - Project Source Code: Python code broken down into each of the processing steps to fully reproduce the dataset. This includes scripts for downloading all dependent input data, processing each stage of the dataset synthesis. - Rough-reef-shallow-mask: This is a manual mapping of the shallow marine areas to a scale of 1:500 k. This was used to improve the quality of the automated mapping by masking out shallow areas that would affect the accuracy of the water colour estimation. While this mask is relatively rough representation of the inshore shallow boundaries, significant effort was made to ensure that all offshore reefs where represented, regardless of size. This was to allow this dataset to be used as an indication of the presence of platform reefs. The rough reef mapping was very quickly mapped in 18 hours of manual digitisation, this was progressively improved to reduced the error in the mapping and to progressively map smaller reef features. A snapshot of the mapping was taken after 39, 57, and 87 hours of mapping. These snapshots are made available in this dataset. The intention is that these progressive improvements to the masks can be used to study then level of detail needed in the mapping for the semi-automated mapping to produce a good result. - Cleanup-remove-mask: This is used to mask out false positive detections from the automated mapping. This cleanup mask was created manually after verifying that no reefs exist in the exclusion area. The goal of this masking was to remove the bulk of the false positives caused by sun glint, turbid eddies, and cloud anomalies. This mask is specific to artefacts generated from composite imagery used. If this process was repeated with different composite imagery then this clean up mask would be of relatively little value. What does this dataset map?: This dataset aims to create a mask that is useful for clipping habitat mapping based on satellite imagery. It aims to map the areas where the benthos is sufficiently visible that the habitat mapping can be performed accurately. This dataset also aims to be useful for creating an estimate of shallow subtidal areas where there might be significant seagrass, sand banks and reef boundaries. For this reason the exact boundary is only loosely defined. For inshore areas where the water is highly turbid the boundary must, as a minimum, include the intertidal zone, i.e. the area that is exposed at low tide. Where the subtidal area is visible then it should be mapped to where there is a sharp drop off, or -5 m MSL. Where there are large shallow areas where the bottom is visible then the focus is on identifying the boundaries of the raised areas or seagrass. For features separated from the mainland or large islands in deeper waters the feature boundary corresponds to the deepest extent where the feature is raised above the seafloor that is shallower than -40 m. For offshore reefs this means the deepest visible extent. For sand banks in shallow water, where the bottom is visible, the boundary corresponds to where it is raised from the surrounding seafloor. For ancient coastline rocky reefs the boundary should include all areas that have evidence of exposed hard rocky substrate, regardless of depth or vertical relief. Source satellite imagery: This dataset was based on two Sentinel 2 composite image collections optimised for the marine environment. For mid-shelf and offshore waters we used an image composite that combined up to 200 images per Sentinel 2 tile from 2018 - 2023 (Hammerton and Lawrey, 2024a), from all tidal levels (referred to as the All-tide imagery). This imagery provides the clearest view of reef shoals and reefs (20 - 40 m) in clear offshore waters, a good view of mid-shelf features and an OK view of inshore turbid areas. Because this composite is made up from so many images, turbid eddies are largely averaged out resulting in relatively smooth water clarity gradients. To improve the mapping in highly turbid inshore areas a low tide image composite was used. This was created from the 10 lowest tide images per Sentinel 2 tile from 2018 - 2023 (Hammerton and Lawrey, 2024b). This low tide composite provides a much clearer view of shallow reefs and sediment in turbid inshore areas. In offshore areas the low tide imagery is not as clear as the All-tide composite due to the limited number of images in the composite. Water Color Estimation: The mapping process is based on first estimating the satellite imagery without islands, reefs, or shallow areas, just water. This water estimate is localised and follows the water colour gradients across the image. Identifying reef features, even when they are barely visible, can be achieved by comparing the original satellite imagery with the water estimate. The reefs are tonally darker, or lighter than their surrounding water. To estimate the water colour over reef features we use a three-step approach: 1. Creating a Rough Mask of Shallow Areas and Reefs: We manually created a rough mask of all shallow areas and reefs, at a scale of approximately 1:500 k scale. This mask is used to remove areas that are significantly brighter or darker than the local water colour, which would skew the estimate. This mask was manually digitised from the All tide and Low tide satellite composite imagery. In the GBR the existing reef boundary mapping (Lawrey and Stewart, 2016) was simplified and used as the starting mask. Additional masking was added for shallow inshore soft sediment areas and for deep reef areas not already covered by the GBR reef boundaries. This manual mapping took 52.9 hours to map northern Australia and 4.3 hours to mask of areas of the GBR. It should be noted that the quality of the inshore masking on the GBR is lower than that of northern Australia as it was not the project primary study area. 2. Infilling Masked Areas with Surrounding Water Colour: On a copy of the satellite imagery the rough masked areas are replaced with the colour of the surrounding water, estimated by using a very large blurring of the image (gaussian, sigma 160 pixels). This blur infills the masked areas with a mixture of the colour of the surrounding waters. With this sized blur masked isolated areas as large as 12 km across, or coastal areas 6 kms across can be infilled with an estimated water colour. 3. Final Blurring to Suppress Small Features: To cover off shallow areas and reefs that were not infilled by the initial masking (due to masking errors) a blur is applied to the image from stage 2. This blurs out the signal from unmasked out platform reefs and fringing shallow areas that are significantly smaller than the blurring radius. In this stage we used a gaussian blurring with a sigma of 40 pixels and so features smaller than 200 m will have their signal strength significantly reduced, making it possible to detect them as a difference to the original satellite imagery. In stages 2 and 3 we mask out the land area, plus a small buffer of 5 pixels, to ensure that pixel values from the land are not mixed into the surrounding water estimates. Shallow Feature Detection: To calculate the shallow marine areas mask we calculate the colour difference between the satellite image and the water estimate. Areas that deviate significantly from the water estimate indicate the presence of shallow habitats such as reefs or sediment. To address varying water conditions, we use three detectors optimized for specific environments: - Offshore Detector: Focuses on detecting deep reefs (20-40 m) in clear water. At the depths of these features they are only visible in the blue channel. This detector only uses the blue channel as it is the only satellite band that penetrates to the depth of these features. Including other bands would add noise into the process. This detector is limited to offshore areas by creating a offshore mask from the green channel and only focuses on detecting features that are brighter than the surrounding waters. - Mid-shelf Detector: This feature detector focuses on detecting features down to ~10 m in moderately turbid mid-shelf areas by combining blue and green channels. This detector is not spatially masked, but doesn't contribute much to offshore areas because its sensitivity is lower than the offshore detector. This feature only picks up features brighter than the surrounding waters. - Inshore Detector: This detector targets shallow areas in highly turbid environments, where features are close to the surface and a strong signal is visible across all channels. It aims to detect features that are both brighter and darker than the water estimate. Detecting features darker than the water allows seagrass and shallow reefs covered in macroalgae to be detected. Each detector applies specific detection thresholds, noise reduction techniques, and spatial masks to isolate features within its target environment. Detector processing: The core calculation for each detector involves the vector distance between the pixel values of the satellite imagery and the water estimate. For a detector using n bands, the calculation is: water_difference = (b1-bw1)^2 + (b2-bw2)^2 + ... + (bn-bwn)^2 difference_gain = 1/(bw1 + bw2 + ... + bwn + denominator_offset) where bn is the nth band of the satellite imagery and bwn is the nth band of the water estimate. The difference_gain normalises the strength of the difference across various water conditions. In shallow areas the strength of the water_difference signal is high, where as in offshore waters the contrast difference between the reef features and the surrounding water is low. The difference_gain boosts the strength of the signal in offshore waters where the water estimate is darker. The denominator_offset is a constant used to adjust how much the difference_gain should vary across difference water conditions. spatial_mask = clip((bw2 - min_out) / (max_in - min_out), 0, 1) The spatial extent of where a detector is used is limited to the inshore or offshore regions using a mask calculated from the green channel. A soft mask was used to separate the regions into areas that should be included with no modification (where the mask value is 1), a transition region where the strength of the detector is weighted by the mask (mask value >0 and < 1) and an exclusion area where the detector should create no signal (mask value of 0). This was achieved by scaling the All tide composite water estimate green channel from min_out -> 0, through to max_in -> 1. For the offshore detector the goal was to exclude inshore areas from the detector (min_out = 100, max_in = 70). For the mid-shelf detector no spatial masking was used and for the inshore detector the goal was to exclude the offshore areas (min_out = 70, max_in = 100) . We found the green channel was most useful for creating the spatial_mask, however it was not reliable in areas where there is high colour dissolved organic matter (CDOM) in the water. k_detector = clip(k_water_difference * k_difference_gain * k_spatial_mask, k_detection_threshold / 1.3, k_detection_threshold * 1.3) Where k corresponds to inshore, midshelf or offshore detectors. Each of these detectors will have a different combination of input bands and denominator_offset. For the inshore detector the red (B04), green (B03) and blue (B02) bands were used. For the midshelf detector the green (B03) and blue (B02) bands were used and for the offshore detector only the blue (B02) band was used. The detector output corresponds to the water difference, scaled by the difference gain, limited by the spatial mask. A soft threshold is applied to the scale difference by normalising the result from 0 to 1 from 30% below and above the detector threshold. This soft threshold allows for better integration of the signal strength from the multiple detectors prior to the final hard threshold needed in converting to a polygon. combined_detectors = (inshore_detector + midshelf_detector + offshore_detector) / 3 A combined detector was created by averaging the three individual detectors. Noise reduction was then applied using a median filter (9 - 25 pixels depending on the detector), followed by a dilation and erosion (5 pixels) to cluster close features and infill small holes in features. The raster was then clipped by the land and by the clean up removal mask. It was then upscaled to twice the resolution using bilinear interpolation and converted to a polygon. The interpolation was used to improve the smoothness of the polygon conversion. The polygon was then simplified using the Douglas-Peucker algorithm. All the polygon masks from the individual Sentinel 2 tiles were then merged, dissolved and split into single part polygons. The full details of the processing are described in greater detail in the code for this dataset https://github.com/eatlas/AU_NESP-MaC-3-17_AIMS_Shallow-mask. This code allows full reproduction of all steps in the creation of this dataset, from downloading the source input data, through to the generation of the final data. Limitations: - Using this mask against different imagery: This shallow marine mask is based on composite Sentinel 2 imagery that was optimised for marine conditions (Hammerton and Lawrey, 2024a, 2024b). If different imagery is used for subsequent habitat mapping then the mask may under or over estimate the area that can be reliably observed from that imagery. In many of the turbid environments across Northern Australia the features identified in this feature mask are only visible if the imagery is formed from a large composite of images (to average out the turbidity effects) or the image is selected from the very clearest days. As a result this mask covers many features that are not visible from most single daily satellite images. - Image anomalies lead to false features: The composite imagery used suffers from considerable cloud anomalies in Torres Strait and north eastern Cape York, as a result these areas contain many false positive features. A moderate level of effort was put into cleaning up the bulk of these anomalies using the cleanup-remove-mask. However, hundreds of false features are likely to remain in the dataset. - Less QAQC on GBR: This dataset was primarily developed for mapping the marine features of Northern Australia. Torres Strait and the Great Barrier Reef are included as test areas to assess the robustness of the approach developed. Very few optimisations on the imagery or approach were made for the Great Barrier Reef and so the quality in this region is lower. - The mapping is a mixture of substrates: This dataset maps the boundaries of fringing reefs, offshore reefs and shallow soft sediment. It does not distinguish between soft sediment areas and hard substrate areas (reefs). In many inshore areas the masked areas correspond to the union of reef areas and the surrounding shallow soft sediment areas. To separate these into different features would require an additional mapping process. - The shallow mapping is poor in rivers: The technique used here to identify shallow areas works on the assumption that the shallow features are a small fraction of the area. This allows the large blur to keep the ocean water and remove the small reefs. When this is not the case the water estimate may be close to the the colour of the shallow areas, resulting in deep areas being falsely interpreted as 'shallow' features to be mapped. The deeper features will be interpreted as different in colour from the water estimate and thus a reef. This effectively results in an inversion of the mapping. This is particularly the case for the inshore detector that assumes that features can be both lighter and darker than the water estimate are features. This dark detection is necessary to detect inshore reefs, which are darker than the surrounding water. This means that the technique is prone to failing to correctly map shallow portions of inshore river systems, except when they are very wide and mostly deep. As a result a lot of manual clean up was required on the river systems. Another problem in narrow rivers is caused by the water estimation process. As part of the water estimation step the land is masked out of the satellite imagery prior to the blurring used to create the water estimate. A small buffer (50 m) is applied to the land mask to ensure that small errors in the land mapping don't result in leakage of land pixels into the water estimate. This small buffer has the side effect of masking off small river areas. As a result the actual colour of the water in these rivers is not used in the water estimation. Instead the water estimate in these areas is determined from the infilled interpolation that is applied over masked areas. This can result in the water estimate in small rivers being significantly different to the actual water colour. Since the actual water colour, in the satellite imagery, is a significant deviation from the water estimate then it falsely is mapped as a shallow area. - NOT FOR NAVIGATION: This dataset should not be used for navigation as it does not reliably detect small reef features that can be a maritime hazard. - Inshore shallow areas affected by bias in Rough-reef-shallow-mask: The Rough-reef-shallow-mask is used to prime the automated shallow mapping by removing areas that would heavily influence the water estimation stage of the processing. For small isolated reefs the quality of the Rough-reef-shallow-mask doesn't really matter, however for shallow areas with a strong brightness gradient from the land to deeper areas the position of the Rough-reef-shallow-mask heavily influences the final mapped feature boundary. This means that any bias in the Rough-reef-shallow-mask will propagate through to the final automated mapping. Accuracy of the masking: This dataset includes four sets of masks: manual rough-shallow-reef-mask, and semi-automated shallow -reef-masking at low, medium, high and very high levels of sensitivity. Each of these products is intended to be an input to other phases of mapping. The goal of the rough-shallow-reef-mask is to ensure that all significant shallow areas are masked to improve the quality of the semi-automated reef-masking. This was done at a scale of 1:500k with a 90th percentile positional error of ~150 - 200 m. The manual rough-shallow-reef-mask included additional effort (~20%) in mapping all shallow reefs, regardless of size, to act as a reference for the position and rough size of all reefs, even those as small as 20 - 30 m across. The accuracy of the semi-automated mapping is difficult to determine given its slight loose definition of mapping shallow features that are visible in the imagery. The boundaries mapped have the following known failure cases. The algorithm is not optimised for handling rivers and so rivers narrower than 100 m are often incorrectly marked as shallow. This is due to several of the processing stages that are associated with land masking, feature clustering and the interpolation process. The water estimation process can interpolate over a maximum distance of approximately 6 km, this can result in large masked areas failing to be marked as shallow. There is one known example in Ningaloo where this has occurred in V1-1 of the dataset. Large deep seagrass areas in Shark bay are poorly represented by the shallow mapping. Many of the large seagrass areas are incorrectly interpreted by the processing as deep water. In deep large rocky reefs seen in southern portions of Western Australia we find that the detector primarily detects the paler sandy areas, rather than the darker rocky features. Format: Rough Shallow Reef Mapping: - Purpose: Masking for automated water estimation, recording the position and size of all reef features - Path: in-data/AU_Rough-reef-shallow-mask/AU_AIMS_NESP-MaC-3-17_Rough-reef-shallow-mask_Base.shp - Format: Shapefile - Size: 3.5 MB - Features: 7733 features - Digitisation time: 77.2 hours (northern GBR) + 9.9 hrs for GBR Cleanup Remove Mask: - Purpose: Input to the last stage of the semi-automated shallow mapping. This mask is used to remove spurious features detected, primarily from clouds, prior to converting the results into polygons. - Path: in-data/AU_Cleanup-remove-mask/AU_AIMS_NESP-MaC-3-17_Cleanup-remove-mask.shp - Format: Shapefile - Size: 163 kB - Features: 475 features - Digitisation time: ~4 hours Semi-automated Shallow Marine Mask: - Purpose: Clipping of automated habitat mapping, mapping shallow soft sediment areas at three levels of detection sensitivities. - Format: Shapefiles - Low: data/out/low/AU_NESP-MaC-3-17_AIMS_Shallow-mask_Low-VLow_V1-1.shp, - Medium: data/out/med/AU_NESP-MaC-3-17_AIMS_Shallow-mask_Medium-Low_V1-1.shp, - High: data/out/high/AU_NESP-MaC-3-17_AIMS_Shallow-mask_High-Medium_V1-1.shp, - Very High: data/out/vhigh/AU_NESP-MaC-3-17_AIMS_Shallow-mask_VHigh-High_V1-1.shp, - Size: Low (36.6 MB), Medium (38.9 MB), High (40.6 MB), Very High (43.3 MB) References: Beaman, R.J. 2017. AusBathyTopo (Great Barrier Reef) 30m 2017 - A High-resolution Depth Model (20170025C). Geoscience Australia, Canberra. http://dx.doi.org/10.4225/25/5a207b36022d2 Beaman, R. 2023. AusBathyTopo (Torres Strait) 30m 2023 - A High-resolution Depth Model (20230006C). Geoscience Australia, Canberra. https://dx.doi.org/10.26186/144348 Bishop-Taylor, R., Sagar, S., Lymburner, L., Beaman, R.L., 2019. Between the tides: modelling the elevation of Australia's exposed intertidal zone at continental scale. Estuarine, Coastal and Shelf Science. https://doi.org/10.1016/j.ecss.2019.03.006 Bishop-Taylor, R., Phillips, C., Newey, V., Sagar, S.(2024). Digital Earth Australia Intertidal. Geoscience Australia, Canberra. https://dx.doi.org/10.26186/149403 [Accessed 4 December 2024] Hammerton, M., & Lawrey, E. 2024a. North Australia Sentinel 2 Satellite Composite Imagery - 15th percentile true colour (NESP MaC 3.17, AIMS) (2nd Ed.) [Data set]. eAtlas. https://doi.org/10.26274/HD2Z-KM55 Hammerton, M., & Lawrey, E. 2024b. Tropical Australia Sentinel 2 Satellite Composite Imagery - Low Tide - 30th percentile true colour and near infrared false colour (NESP MaC 3.17, AIMS) (1st Ed.) [Data set]. eAtlas. https://doi.org/10.26274/2bfv-e921 Lawrey, E. P., Stewart M. 2016. Complete Great Barrier Reef (GBR) Reef and Island Feature boundaries including Torres Strait (NESP TWQ 3.13, AIMS, TSRA, GBRMPA) [Dataset]. Australian Institute of Marine Science (AIMS), Torres Strait Regional Authority (TSRA), Great Barrier Reef Marine Park Authority [producer]. eAtlas Repository [distributor]. https://eatlas.org.au/data/uuid/d2396b2c-68d4-4f4b-aab0-52f7bc4a81f5 Lebrec, U., Paumard, V., O'Leary, M. J., and Lang, S. C. 2021. Towards a regional high-resolution bathymetry of the North West Shelf of Australia based on Sentinel-2 satellite images, 3D seismic surveys and historical datasets. Earth Syst. Sci. Data Discuss. https://doi.org/10.5194/essd-13-5191-2021 Lebrec, U. 2021. A High-resolution depth model for the North West Shelf and Outer Browse Basin (20210025C). Geoscience Australia, Canberra. https://doi.org/10.26186/144600 Twiggs, E. 2023. Kimberley Region and WA Reefs Satellite-Derived Bathymetry Acquisition (20210024S). Geoscience Australia, Canberra. https://dx.doi.org/10.26186/148669 Change log: As updates to this dataset are published the changes will be recorded here. 2025-04-01 V1-1: Initial release of the dataset, optimised for shallow coastal mapping.

Lineage

Maintenance and Update Frequency: asNeeded

Statement: The Shallow Marine Mask is based on Sentinel 2 composite imagery from 2015 - 2024.

Notes

Credit
This work was funded by the National Environmental Science Program, Marine and Coastal (NESP MaC) hub matched by an equivalent amount of in-kind support and co-investment from the Australian Institute of Marine Science.

Purpose
The intended purpose of this dataset is to act as a clipping mask for benthic habitat mapping developed from Sentinel 2 imagery. This mask delineates regions where benthic habitats (reefs, sediments, seagrass) are potentially visible from satellite imagery, helping to remove areas where there are no features visible in the imagery. This can be used to improve the accuracy of automated habitat mapping techniques by removing areas where the results are likely to spurious or have a very low accuracy. An additional purpose of this dataset is to assist in the mapping of shallow soft sediment areas. The shallow soft sediment areas can be determined by subtracting the reef areas (coral and rocky), mapped separately, from this dataset.