Getting Predictions from Hosted Models

Earth Engine provides ee.Model as a connector to models hosted on Vertex AI. Specifically, Earth Engine will send image or table data as online prediction requests to a trained model deployed on a Vertex AI endpoint. The model outputs are then available as Earth Engine images or tables.

TensorFlow is an open source ML platform that supports advanced ML methods such as deep learning. The Earth Engine API provides methods for importing/exporting imagery, training and testing data in TFRecord format. See the ML examples page for demonstrations that use TensorFlow with data from Earth Engine. See the TFRecord page for details about how Earth Engine writes data to TFRecord files.

ee.Model

The ee.Model package handles interaction with hosted machine learning models.

Connecting to models hosted on Vertex AI

A new ee.Model instance can be created with ee.Model.fromVertexAi. This is an ee.Model object that packages Earth Engine data into tensors, forwards them as predict requests to Vertex AI then automatically reassembles the responses into Earth Engine data types.

Earth Engine supports TensorFlow (e.g. SavedModel format), PyTorch and AutoML models. To prepare a model for hosting, save it, import it, then deploy the model to an endpoint.

To interact with Earth Engine, a hosted model's inputs/outputs need to be compatible with a supported interchange format. For hosted TensorFlow models, this is the TensorProto interchange format, specifically serialized TensorProtos in base64 (reference). This can be done programmatically, as shown on the ML examples page, after training and before saving, or by reloading saved models. Other supported format include JSON (RAW_JSON mode) and multi-dimensional arrays (ND_ARRAYS). See the ee.Model.fromVertexAi() reference docs for details.

To use a model with ee.Model.fromVertexAi(), you must have sufficient permissions to use the model. Specifically, you (or anyone who uses the model) needs at least the Vertex AI user role. You control permissions for your Cloud Project using Identify and Access Management (IAM) controls.

Regions

When deploying your model and endpoint, you will need to specify which region to deploy to. The us-central1 region is recommended since it will likely perform best due to proximity to Earth Engine servers, but almost any region will work. See the Vertex AI docs for details about Vertex AI regions and what features each one supports.

If you are migrating from AI Platform, note that Vertex AI does not have a global endpoint, and ee.Model.fromVertexAi() does not have a region parameter.

Costs

Image Predictions

Use model.predictImage() to make predictions on an ee.Image using a hosted model. The return type of predictImage() is an ee.Image which can be added to the map, used in other computations, exported, etc. The image is forwarded to the hosted model in tiles. You control how the image is tiled using the inputTileSize, inputOverlapSize, and outputTileSize parameters. For example, a fully convolutional model may segment 256x256xChannels shaped inputs (scale is unchanged), but the edges of the tiles can be discarded by the specified overlap. Alternatively, an image classification model may accept 256x256xChannels shaped inputs, but return a 1x1x1 output of probability (reduced resolution) of some tile characteristic. Note that Earth Engine will always forward each band in the input Image as a 3D - specifically (inputTileSize[0], inputTileSize[1], 1) shaped - float32 tensor, even if the band's values are scalar.

Nearly all convolutional models will expect input of a fixed projection (that of the data on which the model was trained). In this case, set the fixInputProj parameter to true in your call to ee.Model.fromVertexAi() and specify the data's projection in the proj parameter. When visualizing predictions, use caution when zooming out on a model that has a fixed input projection. As explained in the EE reprojecting guide, zooming to a large spatial scope can result in requests for too much data, too many concurrent requests, or too much billable compute.

The input Image's projection and the model's fixed projection will affect the output values. See the EE Reprojection guide.

Table Predictions

Use model.predictProperties() to make predictions on an ee.FeatureCollection. Inputs are stored as properties of the table. The inputs and outputs can be scalar string values, scalar boolean values, or numeric values of any shape, from scalars to multidimensional arrays. The outputs of the model are stored as new properties in the output table.