Spatial Audio API
This is the GVR Audio C API, a spatial audio rendering engine, optimized for mobile VR.
Summary
It allows the user to spatialize sound sources in 3D space, including distance and elevation cues. Specifically, the API is capable of playing back spatial sound in three ways:
- Sound object rendering: This allows the user to create a virtual sound source in 3D space. These sources, while spatialized, are fed with mono audio data.
- Ambisonic soundfields: Ambisonic recordings are multi-channel audio files which are spatialized all around the listener in 360 degrees. These can be thought of as recorded or pre-baked soundfields. They can be of great use for background effects which sound perfectly spatial. Examples include rain noise, crowd noise or even the sound of the ocean off to one side.
- Stereo Sounds: This allows the user to directly play back non-spatialized mono or stereo audio files. This is useful for music and other such audio.
Initialization
gvr_audio_context* gvr_audio_create(int32_t rendering_mode);
The rendering_mode argument corresponds to a gvr_audio_rendering_mode
enum value, which specifies a rendering configuration setting:
GVR_AUDIO_RENDERING_STEREO_PANNING
: Stereo panning of all sound objects. This disables HRTF-based rendering.GVR_AUDIO_RENDERING_BINAURAL_LOW_QUALITY
: This renders sound objects over a virtual array of 8 loudspeakers arranged in a cube configuration around the listener’s head. HRTF-based rendering is enabled.GVR_AUDIO_RENDERING_BINAURAL_HIGH_QUALITY
: This renders sound objects over a virtual array of 16 loudspeakers arranged in an approximate equidistribution about the listener’s head. HRTF-based rendering is enabled.
For most modern phones, the high quality mode offers a good balance between performance and audio quality. To optimize the rendering performance for headphones and speaker playback, the stereo speaker mode can be enabled which automatically switches to stereo panning when no headphone is plugin. Note that this can lead to varying CPU usage based on headphone and speaker playback.
Sound engine control
Audio playback on the default audio device can be started and stopped by calling the following two methods:
void gvr_audio_pause(gvr_audio_context* api); void gvr_audio_resume(gvr_audio_context* api);
Note that:
void gvr_audio_update(gvr_audio_context* api);
must be called from the main thread at a regular rate. It is used to execute background operations outside of the audio thread.
Listener position and rotation
To ensure that the audio in your application reacts to listener head movement, it is important to update the listener's head orientation in the graphics callback using the head orientation matrix.
The following method is used to control the listener’s head position and orientation with the audio engine:
void gvr_audio_set_head_pose(gvr_audio_context* api, gvr_mat4f head_pose_matrix);
Preloading Sounds
Both mono sound files for use with Sound Objects and multi-channel Ambisonic soundfield files can be preloaded into memory before playback or alternatively streamed during playback. Preloading can be useful to reduce CPU usage especially if the same audio clip is likely to be played back many times. In this case playback latency is also reduced.
Sound files can be preloaded into memory by calling:
bool gvr_audio_preload_soundfile(gvr_audio_context* api, const char* filename);
Unused sound files can be unloaded with a call to:
void gvr_audio_unload_soundfile(gvr_audio_context* api, const char* filename);
NOTE: If a sound object, soundfield or stereo sound is created with a file that has not been preloaded, that audio will be streamed.
Spatialization of sound objects
The GVR Audio System allows the user to create virtual sound objects which can be placed anywhere in space around the listener.
To create a new sound object, call:
gvr_audio_source_id gvr_audio_create_sound_object(gvr_audio_context* api, const char* filename);
This returns a handle that can be used to set properties such as the position and the volume of the sound object via calls to the following two functions:
void gvr_audio_set_sound_object_position(gvr_audio_context* api, gvr_audio_source_id sound_object_id, float x, float y, float z); void gvr_audio_set_sound_volume(gvr_audio_context* api, gvr_audio_source_id source_id, float volume);
The GVR Audio System also support directivity patterns per sound source which define a shape or pattern that describes the way in which sound emanates from a source in different directions. The directivity pattern takes two values: alpha and order. Alpha is a weighting balance between a figure-8 pattern and omnidirectional pattern for source emission in the range [0, 1]. A value of 0.5 results in a cardioid pattern. Order is applied to computed directivity. Higher values will result in narrower and sharper directivity patterns, with a range of [1, inf).
void gvr_audio_set_sound_object_directivity( gvr_audio_context* api, gvr_audio_source_id sound_object_id, float alpha, float order); void gvr_audio_set_sound_object_rotation( gvr_audio_context* api, gvr_audio_source_id sound_object_id, gvr_quatf sound_object_rotation);
The behavior of Sound Objects with respect to their distance from the listener can be controlled via calls to the following method:
void gvr_audio_set_sound_object_distance_rolloff_model( gvr_audio_context* api, gvr_audio_source_id sound_object_id, int32_t rolloff_model, float min_distance, float max_distance);
This enables a user to choose between logarithmic and linear distance rolloff methods, or to completely disable distance rolloff effects.
The spatialized playback of a sound object can be triggered with a call to:
void gvr_audio_play_sound(gvr_audio_context* api, gvr_audio_source_id source_id, bool looping_enabled);
and stopped with a call to:
void gvr_audio_stop_sound(gvr_audio_context* api, gvr_audio_source_id source_id);
Note that the sound object handle destroys itself at the moment the sound playback has stopped. This way, no clean up of sound object handles is needed. On subsequent calls to this function the corresponding gvr_audio_source_id no longer refers to a valid sound object.
The following function can be used to check if a sound object is currently active:
bool gvr_audio_is_sound_playing(const gvr_audio_context* api, gvr_audio_source_id source_id);
Rendering of ambisonic soundfields
The GVR Audio System also provides the user with the ability to play back ambisonic soundfields. Ambisonic soundfields are captured or pre-rendered 360 degree recordings. It is best to think of them as equivalent to 360 degree video. While they envelop and surround the listener, they only react to the listener's rotational movement. That is, one cannot walk towards features in the soundfield. Soundfields are ideal for accompanying 360 degree video playback, for introducing background and environmental effects such as rain or crowd noise, or even for pre baking 3D audio to reduce rendering costs. The GVR Audio System supports full 3D First Order Ambisonic recordings using ACN channel ordering and SN3D normalization. For more information please see our Spatial Audio specification at: https://github.com/google/spatial-media/blob/master/docs/spatial-audio-rfc.md#semantics
Note that Soundfield playback is directly streamed from the sound file and no sound file preloading is needed.
To obtain a soundfield handler, call:
gvr_audio_source_id gvr_audio_create_soundfield(gvr_audio_context* api, const char* filename);
This returns a gvr_audio_source_id handle that allows the user to begin playback of the soundfield, to alter the soundfield’s volume or to stop soundfield playback and as such destroy the object. These actions can be achieved with calls to the following functions:
void gvr_audio_play_sound(gvr_audio_context* api, gvr_audio_source_id source_id, bool looping_enabled); void gvr_audio_set_sound_volume(gvr_audio_context* api, gvr_audio_source_id source_id, float volume); void gvr_audio_stop_sound(gvr_audio_context* api, gvr_audio_source_id source_id);
Ambisonic soundfields can also be rotated about the listener's head in order to align the components of the soundfield with the visuals of the game/app.
void gvr_audio_set_soundfield_rotation(gvr_audio_context* api, gvr_audio_source_id soundfield_id, gvr_quatf soundfield_rotation);
Direct Playback of Stereo or Mono Sounds
The GVR Audio System allows the direct non-spatialized playback of both stereo and mono audio. Such audio is often used for music or sound effects that should not be spatialized.
A stereo sound can be created with a call to:
gvr_audio_source_id gvr_audio_create_stereo_sound(gvr_audio_context* api, const char* filename);
Paused Sounds and Stopped Sounds
When using sound sources of any of the above types, the user can ensure that the given source is currently playing before calling.
bool gvr_audio_is_sound_playing(gvr_audio_source_id source_id);
This method will return false if the source has been either paused or stopped, and true if the source is currently playing.
Once one is finished with a Sound Object and wish to remove it, a call can be placed to:
void gvr_audio_stop_sound(gvr_audio_source_id source_id);
Once a source has been stopped it is destroyed and the corresponding gvr_audio_source_id will be invalid. Sources which have been played with the |looping_enabled| parameter disabled will also be destroyed once playback of the full audio clip has completed.
To check whether a given gvr_audio_source_id corresponds to a valid source which exists and is in a playable state, a call can be made to:
bool gvr_audio_is_source_id_valid(gvr_audio_source_id source_id);
By using this pair of methods a user can differentiate between sources which have been paused and those which have ceased.
Room effects
The GVR Audio System provides a powerful reverb engine which can be used to create customized room effects by specifying the size of a room and a material for each surface of the room from the gvr_audio_material_name enum. Each of these surface materials has unique absorption properties which differ with frequency. The room created will be centered around the listener. Note that the Google VR Audio System uses meters as the unit of distance throughout.
The following methods are used to control room effects:
void gvr_audio_enable_room(gvr_audio_context* api, bool enable);
enables or disables room effects with smooth transitions.
and
void gvr_audio_set_room_properties(gvr_audio_context* api, float size_x, float size_y, float size_z, gvr_audio_material_name wall_material, gvr_audio_material_name ceiling_material, gvr_audio_material_name floor_material);
allows the user to describe the room based on its dimensions and its surface properties. For example, one can expect very large rooms to be more reverberant than smaller rooms, and a room with hard surface materials such as brick to be more reverberant than one with soft absorbent materials such as heavy curtains on every surface.
Note that when a sound source is located outside of the listener's room, it will sound different from sources located within the room due to attenuation of both the direct sound and the reverb on that source. Sources located far outside of the listener's room will not be audible to the listener.
The following method can be used to subtly adjust the reverb in a room by changing the gain/attenuation on the reverb, setting a multiplier on the reverberation time to control the reverb's length, or adjusting the balance between the low and high frequency components of the reverb.
void gvr_audio_set_room_reverb_adjustments(gvr_audio_context* api, float gain, float time_adjust, float brightness_adjust);
If you are writing C++ code, you might prefer to use the C++ wrapper rather than implement this C API directly.
Example usage (C++ API)
Construction:
std::unique_ptr<gvr::AudioApi> gvr_audio_api(new gvr::AudioApi); gvr_audio_api->Init(GVR_AUDIO_RENDERING_BINAURAL_HIGH_QUALITY);
Update head rotation in DrawFrame():
head_pose_ = gvr_api_->GetHeadSpaceFromStartSpaceRotation(target_time); gvr_audio_api_->SetHeadPose(head_pose_); gvr_audio_api_->Update();
Preload sound file, create sound handle and start playback:
gvr_audio_api->PreloadSoundfile(kSoundFile); AudioSourceId source_id = gvr_audio_api_->CreateSoundObject("sound.wav"); gvr_audio_api->SetSoundObjectPosition(source_id, position_x, position_y, position_z); gvr_audio_api->PlaySound(source_id, /*looped playback=
Typedefs |
|
---|---|
gvr_audio_context
|
typedeftypedef **struct gvr_audio_context_
|
Functions |
|
---|---|
gvr_audio_create(int32_t rendering_mode)
|
gvr_audio_context *
Creates and initializes a gvr_audio_context.
|
gvr_audio_create_sound_object(gvr_audio_context *api, const char *filename)
|
Returns a new sound object.
|
gvr_audio_create_soundfield(gvr_audio_context *api, const char *filename)
|
Returns a new ambisonic sound field.
|
gvr_audio_create_stereo_sound(gvr_audio_context *api, const char *filename)
|
Returns a new stereo non-spatialized source, which directly plays back mono or stereo audio.
|
gvr_audio_destroy(gvr_audio_context **api)
|
void
Destroys a gvr_audio_context that was previously created with gvr_audio_create or gvr_audio_create_android.
|
gvr_audio_enable_room(gvr_audio_context *api, bool enable)
|
void
Turns on/off the room reverberation effect.
|
gvr_audio_enable_stereo_speaker_mode(gvr_audio_context *api, bool enable)
|
void
Enables the stereo speaker mode.
|
gvr_audio_is_sound_playing(const gvr_audio_context *api, gvr_audio_source_id source_id)
|
bool
Checks if a sound is playing.
|
gvr_audio_is_source_id_valid(const gvr_audio_context *api, gvr_audio_source_id source_id)
|
bool
Checks if a |source_id| is valid, and that the corresponding source is in a playable state.
|
gvr_audio_pause(gvr_audio_context *api)
|
void
Pauses the VR Audio system.
|
gvr_audio_pause_sound(gvr_audio_context *api, gvr_audio_source_id source_id)
|
void
Pauses the playback of a sound.
|
gvr_audio_play_sound(gvr_audio_context *api, gvr_audio_source_id source_id, bool looping_enabled)
|
void
Starts the playback of a sound.
|
gvr_audio_preload_soundfile(gvr_audio_context *api, const char *filename)
|
bool
Preloads a local sound file.
|
gvr_audio_resume(gvr_audio_context *api)
|
void
Resumes the VR Audio system.
|
gvr_audio_resume_sound(gvr_audio_context *api, gvr_audio_source_id source_id)
|
void
Resumes the playback of a sound.
|
gvr_audio_set_head_pose(gvr_audio_context *api, gvr_mat4f head_pose_matrix)
|
void
Sets the head pose from a matrix representation of the same.
|
gvr_audio_set_master_volume(gvr_audio_context *api, float volume)
|
void
Changes the master volume.
|
gvr_audio_set_room_properties(gvr_audio_context *api, float size_x, float size_y, float size_z, int32_t wall_material, int32_t ceiling_material, int32_t floor_material)
|
void
Sets the room properties describing the dimensions and surface materials of a given room.
|
gvr_audio_set_room_reverb_adjustments(gvr_audio_context *api, float gain, float time_adjust, float brightness_adjust)
|
void
Adjusts the properties of the current reverb, allowing changes to the reverb's gain, duration and low/high frequency balance.
|
gvr_audio_set_sound_object_directivity(gvr_audio_context *api, gvr_audio_source_id sound_object_id, float alpha, float order)
|
void
Sets the sound object directivity constants for an existing sound object.
|
gvr_audio_set_sound_object_distance_rolloff_model(gvr_audio_context *api, gvr_audio_source_id sound_object_id, int32_t rolloff_model, float min_distance, float max_distance)
|
void
Sets the given sound object source's distance attenuation method with minimum and maximum distances.
|
gvr_audio_set_sound_object_position(gvr_audio_context *api, gvr_audio_source_id sound_object_id, float x, float y, float z)
|
void
Repositions an existing sound object.
|
gvr_audio_set_sound_object_rotation(gvr_audio_context *api, gvr_audio_source_id sound_object_id, gvr_quatf sound_object_rotation)
|
void
Sets the given sound object's rotation.
|
gvr_audio_set_sound_volume(gvr_audio_context *api, gvr_audio_source_id source_id, float volume)
|
void
Changes the volume of an existing sound.
|
gvr_audio_set_soundfield_rotation(gvr_audio_context *api, gvr_audio_source_id soundfield_id, gvr_quatf soundfield_rotation)
|
void
Sets the given ambisonic soundfields's rotation.
|
gvr_audio_stop_sound(gvr_audio_context *api, gvr_audio_source_id source_id)
|
void
Stops the playback of a sound and destroys the corresponding sound object or Soundfield.
|
gvr_audio_unload_soundfile(gvr_audio_context *api, const char *filename)
|
void
Unloads a previously preloaded sample from memory.
|
gvr_audio_update(gvr_audio_context *api)
|
void
This method must be called from the main thread at a regular rate.
|
Typedefs
gvr_audio_context
typedef **struct gvr_audio_context_ gvr_audio_context
Functions
gvr_audio_create
gvr_audio_context * gvr_audio_create( int32_t rendering_mode )
Creates and initializes a gvr_audio_context.
This call also initializes the audio interface and starts the audio engine. Note that the returned instance must be deleted with gvr_audio_destroy.
Details | |||
---|---|---|---|
Parameters |
|
||
Returns |
gvr_audio_context instance.
|
gvr_audio_create_sound_object
gvr_audio_source_id gvr_audio_create_sound_object( gvr_audio_context *api, const char *filename )
Returns a new sound object.
Note that the sample should only contain a single audio channel (stereo sources are automatically downmixed to mono). The handle automatically destroys itself at the moment the sound playback has stopped.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
||||
Returns |
Id of new sound object. Returns GVR_AUDIO_INVALID_SOURCE_ID if the sound file has not been preloaded or if the number of input channels is > 1.
|
gvr_audio_create_soundfield
gvr_audio_source_id gvr_audio_create_soundfield( gvr_audio_context *api, const char *filename )
Returns a new ambisonic sound field.
Note that the sample needs to be preloaded and must have 4 separate audio channels. The handle automatically destroys itself at the moment the sound playback has stopped.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
||||
Returns |
Id of new soundfield. Returns GVR_AUDIO_INVALID_SOURCE_ID if the sound file has not been preloaded or if the number of input channels does not match that required.
|
gvr_audio_create_stereo_sound
gvr_audio_source_id gvr_audio_create_stereo_sound( gvr_audio_context *api, const char *filename )
Returns a new stereo non-spatialized source, which directly plays back mono or stereo audio.
Note the sample needs to be preloaded and may contain only one (mono) or two (stereo) audio channels.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
||||
Returns |
Id of new stereo non-spatialized source. Returns GVR_AUDIO_INVALID_SOURCE_ID if the sound file has not been preloaded or if the number of input channels is > 2;
|
gvr_audio_destroy
void gvr_audio_destroy( gvr_audio_context **api )
Destroys a gvr_audio_context that was previously created with gvr_audio_create or gvr_audio_create_android.
Details | |||
---|---|---|---|
Parameters |
|
gvr_audio_enable_room
void gvr_audio_enable_room( gvr_audio_context *api, bool enable )
Turns on/off the room reverberation effect.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_enable_stereo_speaker_mode
void gvr_audio_enable_stereo_speaker_mode( gvr_audio_context *api, bool enable )
Enables the stereo speaker mode.
It enforces stereo-panning when headphones are not plugged into the phone. This helps to avoid HRTF-based coloring effects and reduces computational complexity when speaker playback is active. By default the stereo speaker mode optimization is disabled.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_is_sound_playing
bool gvr_audio_is_sound_playing( const gvr_audio_context *api, gvr_audio_source_id source_id )
Checks if a sound is playing.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
||||
Returns |
True if the sound is being played.
|
gvr_audio_is_source_id_valid
bool gvr_audio_is_source_id_valid( const gvr_audio_context *api, gvr_audio_source_id source_id )
Checks if a |source_id| is valid, and that the corresponding source is in a playable state.
Sources that have been stopped will be reported as invalid.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
||||
Returns |
True if the source exists and is in a playable state.
|
gvr_audio_pause
void gvr_audio_pause( gvr_audio_context *api )
Pauses the VR Audio system.
Calling this when already paused is a no-op. Thread-safe (call from any thread).
Details | |||
---|---|---|---|
Parameters |
|
gvr_audio_pause_sound
void gvr_audio_pause_sound( gvr_audio_context *api, gvr_audio_source_id source_id )
Pauses the playback of a sound.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_play_sound
void gvr_audio_play_sound( gvr_audio_context *api, gvr_audio_source_id source_id, bool looping_enabled )
Starts the playback of a sound.
Details | |||||||
---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_preload_soundfile
bool gvr_audio_preload_soundfile( gvr_audio_context *api, const char *filename )
Preloads a local sound file.
Note that the local file access method depends on the target platform.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
||||
Returns |
True on success or if file has already been preloaded.
|
gvr_audio_resume
void gvr_audio_resume( gvr_audio_context *api )
Resumes the VR Audio system.
Call this when your app/game loses focus. Calling this when not paused is a no-op. Thread-safe (call from any thread).
Details | |||
---|---|---|---|
Parameters |
|
gvr_audio_resume_sound
void gvr_audio_resume_sound( gvr_audio_context *api, gvr_audio_source_id source_id )
Resumes the playback of a sound.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_head_pose
void gvr_audio_set_head_pose( gvr_audio_context *api, gvr_mat4f head_pose_matrix )
Sets the head pose from a matrix representation of the same.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_master_volume
void gvr_audio_set_master_volume( gvr_audio_context *api, float volume )
Changes the master volume.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_room_properties
void gvr_audio_set_room_properties( gvr_audio_context *api, float size_x, float size_y, float size_z, int32_t wall_material, int32_t ceiling_material, int32_t floor_material )
Sets the room properties describing the dimensions and surface materials of a given room.
Details | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_room_reverb_adjustments
void gvr_audio_set_room_reverb_adjustments( gvr_audio_context *api, float gain, float time_adjust, float brightness_adjust )
Adjusts the properties of the current reverb, allowing changes to the reverb's gain, duration and low/high frequency balance.
Details | |||||||||
---|---|---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_sound_object_directivity
void gvr_audio_set_sound_object_directivity( gvr_audio_context *api, gvr_audio_source_id sound_object_id, float alpha, float order )
Sets the sound object directivity constants for an existing sound object.
Details | |||||||||
---|---|---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_sound_object_distance_rolloff_model
void gvr_audio_set_sound_object_distance_rolloff_model( gvr_audio_context *api, gvr_audio_source_id sound_object_id, int32_t rolloff_model, float min_distance, float max_distance )
Sets the given sound object source's distance attenuation method with minimum and maximum distances.
Maximum distance must be greater than the minimum distance for the method to be set.
Details | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_sound_object_position
void gvr_audio_set_sound_object_position( gvr_audio_context *api, gvr_audio_source_id sound_object_id, float x, float y, float z )
Repositions an existing sound object.
Details | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_sound_object_rotation
void gvr_audio_set_sound_object_rotation( gvr_audio_context *api, gvr_audio_source_id sound_object_id, gvr_quatf sound_object_rotation )
Sets the given sound object's rotation.
Only applies if gvr_audio_set_sound_object_directivity has been called on the sound.
Details | |||||||
---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_sound_volume
void gvr_audio_set_sound_volume( gvr_audio_context *api, gvr_audio_source_id source_id, float volume )
Changes the volume of an existing sound.
Details | |||||||
---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_set_soundfield_rotation
void gvr_audio_set_soundfield_rotation( gvr_audio_context *api, gvr_audio_source_id soundfield_id, gvr_quatf soundfield_rotation )
Sets the given ambisonic soundfields's rotation.
Details | |||||||
---|---|---|---|---|---|---|---|
Parameters |
|
gvr_audio_stop_sound
void gvr_audio_stop_sound( gvr_audio_context *api, gvr_audio_source_id source_id )
Stops the playback of a sound and destroys the corresponding sound object or Soundfield.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_unload_soundfile
void gvr_audio_unload_soundfile( gvr_audio_context *api, const char *filename )
Unloads a previously preloaded sample from memory.
Note that if the sample is currently used, the memory is freed at the moment playback stops.
Details | |||||
---|---|---|---|---|---|
Parameters |
|
gvr_audio_update
void gvr_audio_update( gvr_audio_context *api )
This method must be called from the main thread at a regular rate.
It is used to execute background operations outside of the audio thread.
Details | |||
---|---|---|---|
Parameters |
|