From Terabytes to Insights: NEON's Journey to Earth Engine with Google Cloud

Google Earth Engine is a cloud-based platform providing access to satellite imagery and geospatial data, used across the scientific community to address environmental and societal challenges. Users can analyze, visualize, and share data within the platform, furthering collaboration and ground breaking research.

The National Science Foundation’s National Ecological Observatory Network (NEON), operated by Battelle, is one of the largest ecological observation facilities and sensor networks in the world and is working with Google Public Sector to share their data globally on Google Cloud, including through Earth Engine, with researchers. NEON collects high-resolution hyperspectral imagery and lidar-derived terrain data from their Airborne Observatory Platform (AOP). Airborne remote sensing surveys are conducted each year during peak greeness over many of NEON’s 81 field sites. The data collected provide quantitative information on land cover and changes to ecological structure, including effects of wildfires, invasive species, land management, and more.

These AOP data are of high value for ecological studies; however, the large data volume can make it difficult for researchers to work with the data. NEON is ingesting a portion of their airborne remote sensing data into the Earth Engine Data Catalog as a Publisher Catalog, so that the scientific community can leverage Earth Engine’s powerful compute, algorithms, and existing satellite data to perform continental scale environmental research. Earth Engine enables researchers to take advantage of the high volume AOP data that NEON is ingesting into the platform to share with the scientific community.

Using Google Cloud to move data into Earth Engine

Many scientific datasets use dedicated formats such as HDF5 or NetCDF, and this includes the NEON AOP data. Earth Engine supports standard geospatial formats, like GeoTIFF, which allows for seamless integration into the platform. This led NEON to use Google Cloud to preprocess their data for ingestion into Earth Engine.

The data was uploaded to Google Cloud Storage in the original format and transformed via a two-step process (listing and processing files) managed by Google's serverless processing and orchestration services that handled processing large amounts of data in parallel across a multi-step workflow. This enables quick, cost-effective scaling of NEON data conversion into GeoTIFF format.

Once the data was in GeoTIFF format, the data was ingested into Earth Engine. NEON and the Earth Engine team published these datasets in the Earth Engine Data Catalog, making them publicly accessible and searchable for users.

"The tools in Google Cloud, including Cloud Shell and Cloud Run functions, provide a seamless way to convert our data into GeoTIFF format and ingest it into Earth Engine. Instead of having to download the data locally, convert the format, upload back to the Cloud Storage, and ingest the data into Earth Engine, we are able to do this all in one place in just a couple steps, which makes for a much smoother and efficient processing pipeline.” - Bridget Hass, Remote Sensing Data Scientist, NEON.

Earth Engine Publisher Data Catalogs are now available for eligible organizations with commercial usage of Earth Engine to share their data to the broader Earth Engine community, empowering global collaboration and discovery, as well as driving research and solutions to global challenges. Discover more on how Google Cloud's infrastructure and Earth Engine's capabilities are empowering geospatial innovation with Public Sector agencies like US Forest Service and the State of Hawaii.

For more information about Battelle and the NEON program, please visit https://www.neonscience.org.

To learn how to use the NEON datasets on Earth Engine, please visit: Intro to AOP Data in Google Earth Engine Tutorial Series

To learn more about Earth Engine, please visit https://earthengine.google.com

To learn more about Google Cloud and the geospatial offerings, please visit https://cloud.google.com/solutions/geospatial