Cloud GeoTiff Backed Earth Engine Assets

Colab logo Run in Google Colab GitHub logo View source on GitHub

Note: The REST API contains new and advanced features that may not be suitable for all users. If you are new to Earth Engine, please get started with the JavaScript guide.

Earth Engine can load images from Cloud Optimized GeoTiffs (COGs) in Google Cloud Storage (learn more). This notebook demonstrates how to create Earth Engine assets backed by COGs. An advantage of COG-backed assets is that the spatial and metadata fields of the image will be indexed at asset creation time, making the image more performant in collections. (In contrast, an image created through ee.Image.loadGeoTIFF and put into a collection will require a read of the GeoTiff for filtering operations on the collection.) The performance of COG-backed assets is comparable to that of ingested assets in typical use cases.

To create a COG-backed asset, make a POST request to the Earth Engine ImportExternalImage endpoint. As shown in the following, this request must be authorized to create an asset in your user folder.

Start an authorized session

To be able to make an Earth Engine asset in your user folder, you need to be able to authenticate as yourself when you make the request. You can use credentials from the Earth Engine authenticator to start an AuthorizedSession. You can then use the AuthorizedSession to send requests to Earth Engine.

import ee
import json
from pprint import pprint
from google.auth.transport.requests import AuthorizedSession

ee.Authenticate()  #  or !earthengine authenticate --auth_mode=gcloud

# Specify the cloud project you want associated with Earth Engine requests.
ee_project = 'your-project'

session = AuthorizedSession(
    ee.data.get_persistent_credentials().with_quota_project(ee_project)
)

Request body

The request body is an instance of an ImageManifest. This is where the path to the COG is specified, along with other useful properties.

See this guide for details on how to configure an ImageManifest. It is possible to define one or more Tileset with each backing one or more bands. For ImportExternalImage, at most one ImageSource is supported per Tileset.

See this doc for details on exporting COGs.

One Tileset

The simplest ImageManifest is one with one Tileset. If no bands are specified, the resulting asset will contain all the bands of the GeoTIFF with the band names encoded in the GeoTIFF (in this case, "vis-red", "vis-green", and "vis-blue").

request = {
  'imageManifest': {
    'name': f'projects/{ee_project}/assets/cogdemo1',
    'tilesets': [
      { 'id': '0', 'sources': [ { 'uris': ['gs://ee-docs-demos/COG_demo.tif'] } ] }
    ],
    'properties': {
      'source': 'https://code.earthengine.google.com/d541cf8b268b2f9d8f834c255698201d'
    },
    'startTime': '2016-01-01T00:00:00.000000000Z',
    'endTime': '2016-12-31T15:01:23.000000000Z',
  },
}

pprint(request)

More than one Tileset

It is possible to specify an ImageManifest with more than one Tileset where each band of the resulting asset is backed by one of the bands of a Tileset using the tilesetId and tilesetBandIndex fields. This is useful in the case when different bands have different resolutions or data types. Bands can be listed in any order from any available Tileset. In the example below: * "b4b3b2.tif" has a 10m scale, while "b5b6b7" has a 20m scale. * The band order of the resulting asset is mixed from the input COGs (e.g. output band 0 is from Tileset 0, while output band 1 is from Tileset 1).

request = {
  'imageManifest': {
    'name': f'projects/{ee_project}/assets/cogdemo2',
    'uriPrefix': 'gs://ee-docs-demos/external_image_demo/',
    'tilesets': [
      { 'id': '0', 'sources': [ { 'uris': ['b4b3b2.tif'] } ] },
      { 'id': '1', 'sources': [ { 'uris': ['b5b6b7.tif'] } ] },
    ],
    'bands': [
      { 'id': 'red', 'tilesetId': '0', 'tilesetBandIndex': 0 },
      { 'id': 'rededge3', 'tilesetId': '1', 'tilesetBandIndex': 2 },
      { 'id': 'rededge2', 'tilesetId': '1', 'tilesetBandIndex': 1 },
      { 'id': 'green', 'tilesetId': '0', 'tilesetBandIndex': 1 },
      { 'id': 'blue', 'tilesetId': '1', 'tilesetBandIndex': 0 },
      { 'id': 'rededge1', 'tilesetId': '0', 'tilesetBandIndex': 2 },
    ],
  },
}

pprint(request)

Send the request

Make the POST request to the Earth Engine projects.images.importExternal endpoint.

url = f'https://earthengine.googleapis.com/v1alpha/projects/{ee_project}/image:importExternal'

response = session.post(
  url = url,
  data = json.dumps(request)
)

pprint(json.loads(response.content))

Details on COG-backed assets

Location

The Cloud Storage bucket location must be one of:

  • The US multi-region
  • Any US dual-region that includes US-CENTRAL1
  • The region US-CENTRAL1

The metadata of the bucket must be accessible so that Earth Engine can determine its location. The calling user must have the permission storage.buckets.get which is provided by the role "Storage Legacy Bucket Reader" (among others), see https://cloud.google.com/storage/docs/access-control/iam-roles

Storage class

The storage class of the bucket must be "Standard storage".

Permissions

The ACLs of COG-backed Earth Engine assets and the underlying data are managed separately. If a COG-backed asset is shared in Earth Engine, it is the owner's responsibility to ensure that the data in GCS is shared with the same parties. If the data is not visible, Earth Engine will return an error of the form "Failed to load the GeoTIFF at gs://my-bucket/my-object#123456" (123456 is the generation of the object).

Generations

When a COG-backed asset is created, Earth Engine reads the metadata of TIFFs specified in the manifest and creates an asset store entry. Each URI associated with that entry can have a generation. See the object versioning docs for details on generations. If a generation is specified, for example gs://foo/bar#123, Earth Engine will store that URI verbatim. If a generation is not specified, Earth Engine will store that URI with the generation of the TIFF at the time ImportExternalImage was called.

That means that if any TIFF comprising an external asset in GCS is updated (therefore changing its generation), Earth Engine will return a "Failed to load the GeoTIFF at gs://my-bucket/my-object#123456" error because the expected object no longer exists (unless the bucket enables multiple object versions). This policy is designed to keep metadata of the asset in sync with the metadata of the object.

Configuration

In terms of how a COG should be configured, the TIFF MUST be:

  • Tiled, where the tile dimensions are either:

    • 256x256
    • 512x512
    • 1024x1024
    • 2048x2048

  • Arranged so that all IFDs are at the beginning.

For best performance:

  • Use tile dimensions of 512x512 or higher.
  • Include power of 2 overviews.

See this page for more details on an optimized configuration.