Google is committed to advancing racial equity for Black communities. See how.

GoogleARCore.Frame.CameraImage

Container for state related to the ARCore camera for the frame.

Summary

Properties
ImageDisplayUvs
Gets UVs that map the orientation and aspect ratio of the image returned by Frame.CameraImage.AcquireCameraImageBytes to that of the device's display.
ImageIntrinsics
Gets the unrotated and uncropped intrinsics for the image (CPU) stream.
Texture
static Texture
Gets a texture used from the device's rear camera.
TextureDisplayUvs
Gets UVs that map the orientation and aspect ratio of Frame.CameraImage.Texture to those of the device's display.
TextureIntrinsics
Gets the unrotated and uncropped intrinsics for the texture (GPU) stream.

Public static functions
AcquireCameraImageBytes()
Attempts to acquire the camera image for CPU access that corresponds to the current frame.
GetCameraProjectionMatrix(float nearClipping, float farClipping)
Matrix4x4
Gets the projection matrix for the frame.
TransformCoordinate(Vector2 coordinate, DisplayUvCoordinateType sourceType, DisplayUvCoordinateType targetType)
Vector2
Transforms a coordinate between the source and target display UV coordinate types.
UpdateDepthTexture(ref Texture2D depthTexture)
Updates the input texture with the latest depth data from ARCore.
UpdateRawDepthConfidenceTexture(ref Texture2D confidenceTexture)
Attempts to acquire the confidence image corresponding to the raw depth image of the current frame.
UpdateRawDepthTexture(ref Texture2D depthTexture)
Attempts to acquire a "raw", mostly unfiltered, depth image that corresponds to the current frame.

Properties

ImageDisplayUvs

static DisplayUvCoords ImageDisplayUvs

Gets UVs that map the orientation and aspect ratio of the image returned by Frame.CameraImage.AcquireCameraImageBytes to that of the device's display.

ImageIntrinsics

static CameraIntrinsics ImageIntrinsics

Gets the unrotated and uncropped intrinsics for the image (CPU) stream.

Texture

static Texture Texture

Gets a texture used from the device's rear camera.

TextureDisplayUvs

static DisplayUvCoords TextureDisplayUvs

Gets UVs that map the orientation and aspect ratio of Frame.CameraImage.Texture to those of the device's display.

TextureIntrinsics

static CameraIntrinsics TextureIntrinsics

Gets the unrotated and uncropped intrinsics for the texture (GPU) stream.

Public static functions

AcquireCameraImageBytes

GoogleARCore.CameraImageBytes AcquireCameraImageBytes()

Attempts to acquire the camera image for CPU access that corresponds to the current frame.

Depending on device performance, this can fail for several frames after session start, and for a few frames at a time while the session is running.

Details
Returns
A CameraImageBytes struct with IsAvailable property set to true if successful and false if the image could not be acquired.

GetCameraProjectionMatrix

Matrix4x4 GetCameraProjectionMatrix(
  float nearClipping,
  float farClipping
)

Gets the projection matrix for the frame.

Details
Parameters
nearClipping
The near clipping plane for the projection matrix.
farClipping
The far clipping plane for the projection matrix.
Returns
The projection matrix for the frame.

TransformCoordinate

Vector2 TransformCoordinate(
  Vector2 coordinate,
  DisplayUvCoordinateType sourceType,
  DisplayUvCoordinateType targetType
)

Transforms a coordinate between the source and target display UV coordinate types.

This can be used for the conversion of coordinates accessed in the same Unity update.

Details
Parameters
coordinate
The coordinate to transform.
sourceType
The source type of the desired transformation matrix.
targetType
The target type of the desired transformation matrix.
Returns
A corresponding position in the target frame.

UpdateDepthTexture

DepthStatus UpdateDepthTexture(
  ref Texture2D depthTexture
)

Updates the input texture with the latest depth data from ARCore.

If there is no new data, or an error occurs, the contents of the texture will remain unchanged. See DepthStatus for a complete list of reasons.

The actual resolution of the depth image depends on the device and its display aspect ratio, with sizes typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future.

Details
Parameters
depthTexture
The texture to hold the depth data.
Returns
DepthStatus.Success if successful.

UpdateRawDepthConfidenceTexture

DepthStatus UpdateRawDepthConfidenceTexture(
  ref Texture2D confidenceTexture
)

Attempts to acquire the confidence image corresponding to the raw depth image of the current frame.

Each pixel is an 8-bit unsigned integer representing the estimated confidence of the corresponding pixel in the raw depth image. The confidence value is between 0 and 255, inclusive, with 0 representing the lowest confidence and 255 representing the highest confidence in the measured depth value. Pixels without a valid depth estimate have a confidence value of 0 and a corresponding depth value of 0 (see UpdateRawDepthTexture).

The actual resolution of the depth image depends on the device and its display aspect ratio, with sizes typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future. The resolution of the textures set by UpdateDepthTexture, UpdateRawDepthTexture and UpdateRawDepthConfidenceTexture will be the same on a device.

Details
Parameters
confidenceTexture
The texture to hold the depth data.
Returns
DepthStatus.Success if successful.

UpdateRawDepthTexture

DepthStatus UpdateRawDepthTexture(
  ref Texture2D depthTexture
)

Attempts to acquire a "raw", mostly unfiltered, depth image that corresponds to the current frame.

The raw depth image is sparse and does not provide valid depth for all pixels. Pixels without a valid depth estimate have a pixel value of 0 and a corresponding confidence value of 0 (see UpdateRawDepthConfidenceTexture).

The depth image has a single 16-bit plane at index 0, stored in little-endian format. Each pixel contains the distance in millimeters to the camera plane. Currently, the three most significant bits are always set to 000. The remaining thirteen bits express values ranging from 0 millimeters to 8191 millimeters.

The actual resolution of the depth image depends on the device and its display aspect ratio, with sizes typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future. The resolution of the textures set by UpdateDepthTexture, UpdateRawDepthTexture and UpdateRawDepthConfidenceTexture will be the same on a device.

The output depth image can express depth values from 0 millimeters to 8191 millimeters. Optimal depth accuracy is achieved between 50 millimeters and 5000 millimeters from the camera. Error increases quadratically as distance from the camera increases. Depth captured on each frame is used to improve the accuracy for subsequent frames, reducing the error introduced by camera distance.

If an up-to-date depth image isn't ready for the current frame, a 3D reprojection of the most recent depth image to the current frame will be given instead. An up-to-date depth image should typically become available again within a few frames.

The timestamp of this image is equal to the timestamp of the latest camera image that contributed to the depth estimation. If the timestamp of the raw depth image is different from the timestamp of the frame, it means that the camera image of this frame did not take part in the depth estimation and the depth has been reprojected from a previous frame.

Details
Parameters
depthTexture
The texture to hold the depth data.
Returns
DepthStatus.Success if successful.