Google is committed to advancing racial equity for Black communities. See how.

Recording and Playback developer guide for Android

Testing AR experiences in the real world and across different devices can be challenging, often involving repetitive manual trials at a physical location. With the Recording and Playback APIs, you can record video and AR data once within a given environment and use that content to replace a live camera session for testing purposes. This allows you to replay that content indefinitely to try out different AR effects without having to return to the field.

How video and AR data are recorded for playback

ARCore saves recorded sessions to MP4 files that contain multiple video tracks and other miscellaneous data on the device. You can then point your app to use this data in place of a live camera session.

Recorded data

It’s a good idea to film several "takes" of your environment. Use different angles and film around different parts of a given scene to allow for multiple testing permutations later.

Data accessible through MP4-compatible video player

ARCore captures the following data in H.264 video. Access it on an MP4-compatible video player that is capable of switching tracks. The highest-resolution track is the first in the list because some MP4-compatible video players automatically play the first track in the list and do not allow you to choose which video track to play.

Primary video track (CPU image track)

ARCore records a 640x480 video for motion tracking. This is the primary video file that records the environment or scene for later playback. If you don’t select a camera config with a high CPU image resolution, this video will be the first track in the file and will play by default, regardless of which video player you use.

Camera depth map visualization

This is a video file representing the camera’s depth map, recorded from the device’s time-of-flight (ToF) sensors, and converted to RGB channel values. This video should only be used for preview purposes.

Additional parameters captured by ARCore (not accessible)

ARCore also captures the following parameters, which are not accessible at this time.

Camera metadata

All video tracks are followed by data tracks containing metadata for each recorded frame.

Depth sensor measurement (raw depth data)

When a camera config with an enabled active depth sensor is selected, ARCore records this as a metadata track using information from its ToF sensor.

API call events

ARCore records select API calls and callbacks.

IMU sensor measurements

ARCore records measurements from the device’s gyrometer and accelerometer sensors.

Other data captured by ARCore

ARCore also records other data, some of which may be sensitive:

  • Dataset format versions
  • ARCore SDK version
  • Google Play Services for AR version
  • Device fingerprint (the output of adb shell getprop ro.build.fingerprint)
  • Additional information about sensors used for AR tracking

Recording

Start, stop, and check the status of an ARCore session recording.

Record an ARCore session

To record an entire ARCore session, call session.startRecording() before the first call to session.resume(). Recording automatically starts when the session resumes. To automatically stop recording when the session is paused, call RecordingConfig.setAutoStopOnPause(). To record a partial session, call session.startRecording() while the session is running.

Java

// Configure the ARCore session.
Session session = new Session(context);
String destination = new File(context.getFilesDir(), "recording.mp4").getAbsolutePath();
RecordingConfig recordingConfig =
        new RecordingConfig(session)
        .setMp4DatasetFilePath(destination)
        .setAutoStopOnPause(true);
try {
  // Prepare the session for recording, but do not start recording yet.
  session.startRecording(recordingConfig);
} catch (RecordingFailedException e) {
  Log.e(TAG, "Failed to start recording", e);
}

// Resume the ARCore session to start recording.
session.resume();

Kotlin

// Configure the ARCore session.
val session = Session(context)
val destination = File(context.getFilesDir(), "recording.mp4").getAbsolutePath()
val recordingConfig = RecordingConfig(session)
  .setMp4DatasetFilePath(destination)
  .setAutoStopOnPause(true)
session.startRecording(recordingConfig)

// Resume the ARCore session to start recording.
session.resume()

Stop a recording

To stop recording without pausing the currently running AR session, call session.stopRecording().

Java

try {
  session.stopRecording();  // Stop recording.
} catch (RecordingFailedException e) {
  Log.e(TAG, "Failed to stop recording", e);
}

Kotlin

session.stopRecording()

Check recording status

session.getRecordingStatus() can be used at any time to determine the current RecordingStatus.

Java

// Use any time to determine current RecordingStatus.
if (session.getRecordingStatus() == RecordingStatus.OK) {
  // Update the UI to show that the session is currently being recorded.
}

Kotlin

// Use any time to determine current RecordingStatus.
if (session.recordingStatus == RecordingStatus.OK) {
  // Update the UI to show that the session is currently being recorded.
}

Playback

Play back previously recorded AR sessions.

Play back a previously recorded session

To play back a previously recorded session, call session.setPlaybackDataset() before the first call to session.resume().

Java

// Configure the ARCore session.
Session session = new Session(context);

// Specify the previously recorded MP4 file.
String recordingPath = new File(context.getFilesDir(), "recording.mp4").getAbsolutePath();
session.setPlaybackDataset(recordingPath);
…

// Start playback from the beginning of the dataset.
session.resume();
…

// Pause the AR session, but allow playback to silently continue.
session.pause();
…

// Resume the AR session. Playback continues from where the session was paused, with a gap in time.
session.resume();

Kotlin

// Configure the ARCore session.
val session = Session(context)

// Specify the previously recorded MP4 file.
val recordingPath = File(context.filesDir, "recording.mp4").absolutePath
session.playbackDataset = recordingPath
…

// Start playback from the beginning of the dataset.
session.resume()
…

// Pause the AR session, but allow playback to silently continue.
session.pause()
…

// Resume the AR session. Playback continues from where the session was paused, with a gap in time.
session.resume()

Restart playback from the beginning

To restart a playback from the beginning of the dataset, pause the session and call session.setPlaybackDataset(), specifying the same MP4 recording, before resuming the session.

Java

session.pause();
// Pause and specify the SAME dataset:
session.setPlaybackDataset(previousRecordingPath);
session.resume();  // Playback starts from the BEGINNING of the dataset.

Kotlin

session.pause()
// Pause and specify the SAME dataset:
session.playbackDataset = previousRecordingPath
session.resume()  // Playback starts from the BEGINNING of the dataset.

Play back a different session

To play back a different dataset, pause the session and specify the new dataset before resuming the session.

Java

// Switch to a different dataset.
session.pause();   // Pause the playback of the first dataset.
// Specify a different dataset to use.
session.setPlaybackDataset(newRecordingPath);
session.resume();  // Start playback from the beginning of the new dataset.

Kotlin

// Switch to a different dataset.
session.pause()   // Pause the playback of the first dataset.
// Specify a different dataset to use.
session.playbackDataset = newRecordingPath
session.resume()  // Start playback from the beginning of the new dataset.

Check playback status

You can use session.getPlaybackStatus() at any time to determine the current PlaybackStatus.

Java

// Use any time to determine current PlaybackStatus.
if (session.getPlaybackStatus() != PlaybackStatus.OK) {
  // Update the UI to show that the session playback has finished.
}

Kotlin

// Use any time to determine current PlaybackStatus.
if (session.playbackStatus != PlaybackStatus.OK) {
  // Update the UI to show that the session playback has finished.
}