Handle timed metadata in linear DAI streams

The Interactive Media Ads (IMA) Dynamic Ad Insertion SDK (DAI) relies on metadata information embedded in the stream's media segments (in-band metadata), or in the streaming manifest file (in-manifest metadata) to track viewers’ positions and client-side ad events. Metadata is sent in different formats, depending on the type of stream being played.

The video player receives timed metadata in batches. Depending on the player, metadata can be surfaced at the scheduled time, or in batches. Each metadata string has an associated presentation timestamp (PTS) for when it should be triggered.

Your app is responsible for capturing metadata and forwarding it to the IMA DAI SDK. The SDK offers the following methods to pass this information:

onTimedMetadata

This method forwards metadata strings that are ready to be processed to the IMA DAI SDK. It takes a single argument:

• metadata: an object containing a key of TXXX with an associated string value that is prefixed by google_.

processMetadata

This method schedules metadata strings to be processed by the SDK after the specified PTS. It takes the following arguments:

• type: a string containing the type of event being processed. Accepted values are ID3 for HLS or urn:google:dai:2018 for DASH
• data: either a string value prefixed by google_ or a byte array that decodes to such a string.
• timestamp: the timestamp in seconds when data should be processed.

Each stream type supported by the IMA DAI SDK uses a unique form of timed metadata, as described in the following sections.

HLS MPEG2TS streams

Linear DAI HLS streams using the MPEG2TS segments pass timed metadata to the video player through in-band ID3 tags. These ID3 tags are embedded within the MPEG2TS segments and are given the TXXX field name (for custom user-defined text content).

Native playback via VIDEO element (Safari)

Safari processes ID3 tags automatically, as a hidden track, so cuechange events fire at the correct time to process each piece of metadata. It’s alright to pass all metadata to the IMA SDK, regardless of content or type. Irrelevant metadata is filtered out automatically.

Here's an example:

...

videoElement.textTracks.addEventListener('addtrack', (e) => {
  const track = e.track;
  if (track.kind === 'metadata') {
    track.mode = 'hidden';
    track.addEventListener('cuechange', () => {
      for (const cue of track.activeCues) {
        const metadata = {};
        metadata[cue.value.key] = cue.value.data;
        streamManager.onTimedMetadata(metadata);
      }
    });
  }
});

...

HLS.js

HLS.js provides ID3 tags in batches through the FRAG_PARSING_METADATA event, as an array of samples. HLS.js doesn’t translate the ID3 data from byte arrays to strings and doesn’t offset events to their corresponding PTS. It isn’t necessary to decode the sample data from byte array to string, or to filter out irrelevant ID3 tags, as the IMA DAI SDK performs this decoding and filtering automatically.

Here's an example:

...

hls.on(Hls.Events.FRAG_PARSING_METADATA, (e, data) => {
  if (streamManager && data) {
    data.samples.forEach((sample) => {
      streamManager.processMetadata('ID3', sample.data, sample.pts);
    });
  }
});

...

HLS CMAF streams

Linear DAI HLS streams using the Common Media Application Framework (CMAF) pass timed metadata through in-band eMSGv1 boxes following the ID3 through CMAF standard. These eMSG boxes are embedded at the beginning of each media segment, with each ID3 eMSG containing a PTS relative to the last discontinuity in the stream.

Luckily, as of the 1.2.0 release of HLS.js, both of our suggested players pass ID3 through CMAF to the user as if they were traditional ID3 tags. For this reason, the examples below are the same as for HLS MPEG2TS streams. However, this may not be the case with all players, so implementing support for HLS CMAF streams may require unique code to parse ID3 through eMSG.

Native Playback (Safari)

Safari treats ID3 through eMSG metadata as pseudo ID3 events, providing them in batches, automatically, as a hidden track, such that cuechange events are fired at the correct time to process each piece of metadata. It is alright to pass all metadata to the IMA SDK, whether relevant to timing or not. Any non-IMA-related metadata will be filtered out automatically.

Here's an example:

...

videoElement.textTracks.addEventListener('addtrack', (e) => {
  const track = e.track;
  if (track.kind === 'metadata') {
    track.mode = 'hidden';
    track.addEventListener('cuechange', () => {
      for (const cue of track.activeCues) {
        const metadata = {};
        metadata[cue.value.key] = cue.value.data;
        streamManager.onTimedMetadata(metadata);
      }
    });
  }
});

...

HLS.js

As of version 1.2.0, HLS.js treats ID3 through eMSG metadata as pseudo ID3 events, providing them in batches, through the FRAG_PARSING_METADATA event, as an array of samples. HLS.js does not translate the ID3 data from byte arrays to strings and does not offset events to their corresponding PTS. It isn’t necessary to decode the sample data from byte array to string, as the IMA DAI SDK performs this decoding automatically.

Here's an example:

...

hls.on(Hls.Events.FRAG_PARSING_METADATA, (e, data) => {
  if (streamManager && data) {
    data.samples.forEach((sample) => {
      streamManager.processMetadata('ID3', sample.data, sample.pts);
    });
  }
});

...

DASH streams

Linear DAI DASH streams pass metadata as manifest events in an event stream with the custom schemeIdUri value urn:google:dai:2018. Each event in these streams contains a text payload, and the PTS.

DASH.js

Dash.js provides custom event handlers named after the schemeIdUri value of each event stream. These custom handlers fire in batches, leaving it up to you to process the PTS value to properly time the event. The IMA SDK can handle this for you, with the streamManager method, processMetadata().

Here's an example:

...

const dash = dashjs.MediaPlayer().create();
dash.on('urn:google:dai:2018', (payload) => {
  const mediaId = payload.event.messageData;
  const pts = payload.event.calculatedPresentationTime;
  streamManager.processMetadata('urn:google:dai:2018', mediaId, pts);
});

...

Shaka Player

Note: If you're using Shaka Player's integrated DAI support, you don't need to manually process timed metadata. This advice applies to advanced users who are implementing the IMA DAI SDK manually with Shaka player.

Shaka Player surfaces events as a part of their timelineregionenter event. Due to a formatting incompatibility with Shaka Player, the metadata value must be retrieved raw, through the detail property eventElement.attributes['messageData'].value.

Here's an example:

...

player.addEventListener('timelineregionenter', function(event) {
  const detail = event.detail;
  if ( detail.eventElement.attributes &&
       detail.eventElement.attributes['messageData'] &&
       detail.eventElement.attributes['messageData'].value) {
    const mediaId = detail.eventElement.attributes['messageData'].value;
    const pts = detail.startTime;
    streamManager.processMetadata("urn:google:dai:2018", mediaId, pts);
  }
});

...