Google is committed to advancing racial equity for Black communities. See how.

Depth API developer guide for Android

Learn how to use the Depth API in your own apps.

Depth API-supported devices

Only devices that are depth-supported should be able to discover depth-required apps in the Google Play Store. Discovery should be limited to depth-supported devices when:

  • A core part of the experience relies on depth
  • There is no graceful fallback for the parts of the app that use depth

To limit distribution of your app in the Google Play Store to devices that support the Depth API, add the following line to your AndroidManifest.xml, in addition to the AndroidManifest.xml changes described in the Enable ARCore guide:

<uses-feature android:name="com.google.ar.core.depth" />

Check if Depth API is supported

In a new ARCore session, check whether a user's device supports the Depth API.

Java

Config config = session.getConfig();

// Check whether the user's device supports the Depth API.
boolean isDepthSupported = session.isDepthModeSupported(Config.DepthMode.AUTOMATIC);
if (isDepthSupported) {
  config.setDepthMode(Config.DepthMode.AUTOMATIC);
}
session.configure(config);

Kotlin

val config = session.config

// Check whether the user's device supports the Depth API.
val isDepthSupported = session.isDepthModeSupported(Config.DepthMode.AUTOMATIC)
if (isDepthSupported) {
  config.depthMode = Config.DepthMode.AUTOMATIC
}
session.configure(config)

Retrieve depth maps

Call acquireDepthImage() to get the depth map for the current frame.

Java

// Retrieve the depth map for the current frame, if available.
try {
  Image depthImage = frame.acquireDepthImage();
  // Use the depth map here.
} catch (NotYetAvailableException e) {
  // This means that depth data is not available yet.
  // Depth data will not be available if there are no tracked
  // feature points. This can happen when there is no motion, or when the
  // camera loses its ability to track objects in the surrounding
  // environment.
}

Kotlin

// Retrieve the depth map for the current frame, if available.
try {
  val depthImage = frame.acquireDepthImage();
  // Use the depth map here.
} catch (e: NotYetAvailableException) {
  // This means that depth data is not available yet.
  // Depth data will not be available if there are no tracked
  // feature points. This can happen when there is no motion, or when the
  // camera loses its ability to track objects in the surrounding
  // environment.
}

Visualize depth data and occlude virtual objects

The image provided through acquireDepthImage() provides the raw image buffer, which can be passed to a fragment shader for usage on the GPU for each rendered object to be occluded.

This image is oriented in OPENGL_NORMALIZED_DEVICE_COORDINATES, and can be changed to TEXTURE_NORMALIZED by calling Frame.transformCoordinates2d().

Once the depth map is accessible within an object shader, these depth measurements can be accessed directly for occlusion handling.

Parse the depth information for the current frame

As shown in the following code, the helper functions DepthGetMillimeters() and DepthGetVisibility() can be used in a fragment shader to access the depth information for the current screen position. This can then be used to selectively occlude parts of the rendered object.

// Use DepthGetMillimeters() and DepthGetVisibility() to parse the depth map
// for a given pixel, and compare against the depth of the object to render.
float DepthGetMillimeters(in sampler2D depth_texture, in vec2 depth_uv) {
  // Depth is packed into the red and green components of its texture.
  // The texture is a normalized format, storing millimeters.
  vec3 packedDepthAndVisibility = texture2D(depth_texture, depth_uv).xyz;
  return dot(packedDepthAndVisibility.xy, vec2(255.0, 256.0 * 255.0));
}

// Return a value representing how visible or occluded a pixel is relative
// to the depth map. The range is 0.0 (not visible) to 1.0 (completely
// visible).
float DepthGetVisibility(in sampler2D depth_texture, in vec2 depth_uv,
                         in float asset_depth_mm) {
  float depth_mm = DepthGetMillimeters(depth_texture, depth_uv);

  // Instead of a hard Z-buffer test, allow the asset to fade into the
  // background along a 2 * kDepthTolerancePerMm * asset_depth_mm
  // range centered on the background depth.
  const float kDepthTolerancePerMm = 0.015f;
  float visibility_occlusion = clamp(0.5 * (depth_mm - asset_depth_mm) /
    (kDepthTolerancePerMm * asset_depth_mm) + 0.5, 0.0, 1.0);

 // Use visibility_depth_near to set the minimum depth value. If using
 // this value for occlusion, avoid setting it too close to zero. A depth value
 // of zero signifies that there is no depth data to be found.
  float visibility_depth_near = 1.0 - InverseLerp(
      depth_mm, /*min_depth_mm=*/150.0, /*max_depth_mm=*/200.0);

  // Use visibility_depth_far to set the maximum depth value. If the depth
  // value is too high (outside the range specified by visibility_depth_far),
  // the virtual object may get inaccurately occluded at further distances
  // due to too much noise.
  float visibility_depth_far = InverseLerp(
      depth_mm, /*min_depth_mm=*/7500.0, /*max_depth_mm=*/8000.0);

  const float kOcclusionAlpha = 0.0f;
  float visibility =
      max(max(visibility_occlusion, kOcclusionAlpha),
          max(visibility_depth_near, visibility_depth_far));

  return visibility;
}

Simulate occlusion

Simulate occlusion in the body of the fragment shader.

Use the following code to update the object's alpha channel based on the depth, which simulates occlusion:

const float kMetersToMillimeters = 1000.0;

float asset_depth_mm = v_ViewPosition.z * kMetersToMillimeters * -1.;

// Compute the texture coordinates to sample from the depth map.
vec2 depth_uvs = (u_DepthUvTransform * vec3(v_ScreenSpacePosition.xy, 1)).xy;

gl_FragColor.a *= DepthGetVisibility(u_DepthTexture, depth_uvs, asset_depth_mm);

Conventional versus alternative implementations of occlusion rendering

The hello_ar_java sample app uses a two-pass rendering configuration. The first (render) pass renders all of the virtual content into an intermediary buffer. The second pass blends the virtual scene onto the background based on the difference between the real-world depth with the virtual scene depth.

An alternative way to render occlusion is to use per-object, forward-pass rendering. This method determines the occlusion of each pixel of the object in its material shader. If the pixels are not visible, they would be clipped, typically via alpha blending, simulating occlusion on the user’s device.

The efficiency of each approach depends on the complexity of the scene and other app-specific considerations. The two-pass approach requires no additional object-specific shader work and generally produces more uniform-looking results than the forward-pass method.

For more detail and best practices for applying occlusion in shaders, check out the hello_ar_java sample app.

Alternative uses of the Depth API

The hello_ar_java app uses the Depth API to create depth maps and simulate occlusion. Other uses for the Depth API include:

  • Collisions: virtual objects bouncing off walls after a user throws them
  • Distance measurement
  • Re-lighting a scene
  • Re-texturing existing objects: turning a floor into lava
  • Depth-of-field effects: blurring out the background or foreground
  • Environmental effects: fog, rain, and snow

Extract distance from a depth map

Extract information from the depth map to start using the Depth API in ways that do not involve occlusion or depth data visualization. The following code uses the x and y coordinates of a certain depth map to return the millimeter distance between a given pixel and the user's device.

Java

public int getMillimetersDepth(Image depthImage, int x, int y) {
  // The depth image has a single plane, which stores depth for each
  // pixel as 16-bit unsigned integers.
  Image.Plane plane = depthImage.getPlanes()[0];
  int byteIndex = x * plane.getPixelStride() + y * plane.getRowStride();
  ByteBuffer buffer = plane.getBuffer().order(ByteOrder.nativeOrder());
  short depthSample = buffer.getShort(byteIndex);
  return depthSample;
}

Kotlin

fun getMillimetersDepth(depthImage: Image, x: Int, y: Int): Int {
  // The depth image has a single plane, which stores depth for each
  // pixel as 16-bit unsigned integers.
  val plane = depthImage.planes[0]
  val byteIndex = x * plane.pixelStride + y * plane.rowStride
  val buffer = plane.buffer.order(ByteOrder.nativeOrder())
  val depthSample = buffer.getShort(byteIndex)
  return depthSample.toInt()
}

Understand depth values

Given point A on the observed real-world geometry and a 2D point a representing the same point in the depth image, the value given by the Depth API at a is equal to the length of CA projected onto the principal axis. This can also be referred as the z-coordinate of A relative to the camera origin C. When working with the Depth API, it is important to understand that the depth values are not the length of the ray CA itself, but the projection of it.