Testing AR experiences in the real world and across different devices can be challenging, often involving repetitive manual trials at a physical location. With the Recording and Playback APIs, you can record video and AR data once within a given environment and use that content to replace a live camera session for testing purposes. This allows you to replay that content indefinitely to try out different AR effects without having to return to the field.
How video and AR data are recorded for playback
ARCore saves recorded sessions to MP4 files that contain multiple video tracks and other miscellaneous data on the device. You can then point your app to use this data in place of a live camera session.
Recorded data
It’s a good idea to film several "takes" of your environment. Use different angles and film around different parts of a given scene to allow for multiple testing permutations later.
Data accessible through MP4-compatible video player
ARCore captures the following data in H.264 video. Access it on an MP4-compatible video player that is capable of switching tracks. The highest-resolution track is the first in the list because some MP4-compatible video players automatically play the first track in the list and do not allow you to choose which video track to play.
Primary video track (CPU image track)
ARCore records a 640x480 video for motion tracking. This is the primary video file that records the environment or scene for later playback. If you don’t select a camera config with a high CPU image resolution, this video will be the first track in the file and will play by default, regardless of which video player you use.
Camera depth map visualization
This is a video file representing the camera’s depth map, recorded from the device’s time-of-flight (ToF) sensors, and converted to RGB channel values. This video should only be used for preview purposes.
Additional parameters captured by ARCore (not accessible)
ARCore also captures the following parameters, which are not accessible at this time.
Camera metadata
All video tracks are followed by data tracks containing metadata for each recorded frame.
Depth sensor measurement (raw depth data)
When a camera config with an enabled active depth sensor is selected, ARCore records this as a metadata track using information from its ToF sensor.
API call events
ARCore records select API calls and callbacks.
IMU sensor measurements
ARCore records measurements from the device’s gyrometer and accelerometer sensors.
Other data captured by ARCore
ARCore also records other data, some of which may be sensitive:
- Dataset format versions
- ARCore SDK version
- Google Play Services for AR version
- Device fingerprint (the output of
adb shell getprop ro.build.fingerprint) - Additional information about sensors used for AR tracking
Recording
Start, stop, and check the status of an ARCore session recording.
Record an ARCore session
To record an entire ARCore session, call
Session.StartRecording().
This method will fail if a recording is already in progress. Recording may
continue even if the session is paused. If this happens, ARCore will continue to
capture sensor data, while recording the camera feed as a black screen.
ARCoreRecordingConfig config = new ARCoreRecordingConfig();
config.Mp4DatasetFilepath = "path/to/file.mp4";
Session.StartRecording(config);
Stop a recording
To stop recording, call
Session.StopRecording().
Session.StopRecording();
Check recording status
ARRecordingManager.RecordingStatus
can be used at any time to determine the recorder's current state.
Debug.Log("Current Recording Status: " + Session.RecordingStatus);
Playback
Play back previously recorded AR sessions.
Play back a previously recorded session
To play back a previously recorded session, call
Session.SetPlaybackDataset()
and provide a filepath for the dataset you wish to play back. You must pause the
session to use this method. Resume the session for the change to take effect.
// Pause the current session when providing the path.
session.enabled = false;
// In the next frame, provide a filepath for the dataset you wish to play back.
Session.SetPlaybackDataset("path/to/file.mp4");
// In the frame after that, resume the session.
session.enabled = true;
Stop a recording
To stop a playback, call Session.SetPlaybackDataset() and set the dataset filepath to
null.
// The following steps must be completed in separate frames.
session.enabled = false;
Session.SetPlaybackDataset(null);
session.enabled = true;
Restart playback from the beginning
To restart a playback from the beginning of the dataset,
call Session.SetPlaybackDataset() and specify the same MP4 recording before resuming
the session.
// Pause the current session when re-setting the path.
session.enabled = false;
// In the next frame, specify the same dataset.
Session.SetPlaybackDataset(filepath); // filepath is the same path you used before
// In the frame after that, resume playback from the beginning of the dataset.
session.enabled = true;
Play back a different session
To play back a different dataset, call Session.SetPlaybackDataset() and specify a new
dataset before resuming the session.
// Pause the current session when re-setting the path.
session.enabled = false;
// In the next frame, specify a new dataset.
Session.SetPlaybackDataset(new_filepath); // new_filepath is a different path than the one you used before
// In the frame after that, resume playback from the beginning of the new dataset.
session.enabled = true;
Check playback status
ARPlaybackManager.PlaybackStatus
can be used at any time to determine the current playback status.
Debug.Log("Current Playback Status: " + Session.PlaybackStatus);