Audio Classifier

The MediaPipe Audio Classifier task lets you classify audio clips into a set of defined categories, such as guitar music, a train whistle, or a bird's song. The categories are defined during the training of the model. This task operates on audio data with a machine learning (ML) model as independent audio clips or a continuous stream and outputs a list of potential categories ranked by descending probability score.

Open in Web DemosEdit in StackBlitz

Quickstart

Install the SDK for your platform, prepare audio input data, choose an audio running mode, and run inference.

Web

npmnpm install @mediapipe/tasks-audioWeb demo·Edit in StackBlitz·Setup guide

Audio clips

AUDIO_CLIPSSync · complete clipSynchronous classification across every window of a complete prerecorded audio clip.
import { AudioClassifier, FilesetResolver } from "@mediapipe/tasks-audio";

const MODEL_URL = "https://storage.googleapis.com/mediapipe-models/audio_classifier/yamnet/float32/latest/yamnet.tflite";

const audio = await FilesetResolver.forAudioTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-audio@latest/wasm"
);
const audioClassifier = await AudioClassifier.createFromOptions(audio, {
  baseOptions: { modelAssetPath: MODEL_URL },
  maxResults: 5,
});

// Synchronous classification across every window of a complete decoded clip
const audioCtx = new AudioContext({ sampleRate: 16000 });
const response = await fetch("audio_clip.wav");
const audioBuffer = await audioCtx.decodeAudioData(await response.arrayBuffer());
const results = audioClassifier.classify(audioBuffer.getChannelData(0), audioBuffer.sampleRate);

Audio stream

Web Audio streamLive mic chunksClassifies live microphone PCM buffers from the Web Audio API synchronously as each chunk arrives.
import { AudioClassifier, FilesetResolver } from "@mediapipe/tasks-audio";

const MODEL_URL = "https://storage.googleapis.com/mediapipe-models/audio_classifier/yamnet/float32/latest/yamnet.tflite";

const audio = await FilesetResolver.forAudioTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-audio@latest/wasm"
);
const audioClassifier = await AudioClassifier.createFromOptions(audio, {
  baseOptions: { modelAssetPath: MODEL_URL },
  maxResults: 5,
});

// Live microphone stream: classify contiguous PCM windows posted by a 15-line
// AudioWorklet processor (pcm-chunk-processor.js) — see the Prepare input data guide.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const audioCtx = new AudioContext({ sampleRate: 16000 });
await audioCtx.audioWorklet.addModule("pcm-chunk-processor.js");
const source = audioCtx.createMediaStreamSource(stream);
const worklet = new AudioWorkletNode(audioCtx, "pcm-chunk-processor");
worklet.port.onmessage = ({ data: pcmSamples }) => {
  const results = audioClassifier.classify(pcmSamples, audioCtx.sampleRate);
  console.log(results[0]?.classifications[0]?.categories);
};
source.connect(worklet);

Android (Kotlin)

Gradleimplementation("com.google.mediapipe:tasks-audio:latest.release")Sample code·Setup guide

Audio clips

AUDIO_CLIPSSync · complete clipSynchronous classification across every window of a complete prerecorded audio clip.
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier.AudioClassifierOptions
import com.google.mediapipe.tasks.audio.core.RunningMode
import com.google.mediapipe.tasks.components.containers.AudioData
import com.google.mediapipe.tasks.components.containers.AudioData.AudioDataFormat
import com.google.mediapipe.tasks.core.BaseOptions

val baseOptions = BaseOptions.builder()
    .setModelAssetPath("yamnet.tflite")
    .build()
val options = AudioClassifierOptions.builder()
    .setBaseOptions(baseOptions)
    .setRunningMode(RunningMode.AUDIO_CLIPS)
    .setMaxResults(5)
    .build()

val audioClassifier = AudioClassifier.createFromOptions(context, options)

// Synchronous classification over a complete prerecorded AudioData clip
val format = AudioDataFormat.builder().setSampleRate(16000f).build()
val audioData = AudioData.create(format, clipSamples.size)
audioData.load(clipSamples)
val result = audioClassifier.classify(audioData)

Audio stream

AUDIO_STREAMAsync · result callbackAsynchronous microphone pipeline with monotonic timestamps and a result callback. Accumulates contiguous PCM blocks into fixed-length spectrogram windows.
import android.os.SystemClock
import android.util.Log
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier.AudioClassifierOptions
import com.google.mediapipe.tasks.audio.core.RunningMode
import com.google.mediapipe.tasks.components.containers.AudioData
import com.google.mediapipe.tasks.core.BaseOptions

val baseOptions = BaseOptions.builder()
    .setModelAssetPath("yamnet.tflite")
    .build()
val options = AudioClassifierOptions.builder()
    .setBaseOptions(baseOptions)
    .setRunningMode(RunningMode.AUDIO_STREAM)
    .setMaxResults(5)
    .setResultListener { result ->
        val categories = result.classificationResults().firstOrNull()?.classifications()?.firstOrNull()?.categories()
    }
    .setErrorListener { error -> Log.e("MediaPipe", error.message.orEmpty()) }
    .build()

val audioClassifier = AudioClassifier.createFromOptions(context, options)

// Async: accumulates contiguous microphone PCM chunks into spectrogram windows
if (audioData.load(audioRecord) > 0) {
    audioClassifier.classifyAsync(audioData, SystemClock.uptimeMillis())
}

Android (Java)

Gradleimplementation 'com.google.mediapipe:tasks-audio:latest.release'Sample code·Setup guide

Audio clips

AUDIO_CLIPSSync · complete clipSynchronous classification across every window of a complete prerecorded audio clip.
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier;
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier.AudioClassifierOptions;
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifierResult;
import com.google.mediapipe.tasks.audio.core.RunningMode;
import com.google.mediapipe.tasks.components.containers.AudioData;
import com.google.mediapipe.tasks.components.containers.AudioData.AudioDataFormat;
import com.google.mediapipe.tasks.core.BaseOptions;

BaseOptions baseOptions = BaseOptions.builder()
    .setModelAssetPath("yamnet.tflite")
    .build();
AudioClassifierOptions options = AudioClassifierOptions.builder()
    .setBaseOptions(baseOptions)
    .setRunningMode(RunningMode.AUDIO_CLIPS)
    .setMaxResults(5)
    .build();

AudioClassifier audioClassifier = AudioClassifier.createFromOptions(context, options);

// Synchronous classification over a complete prerecorded AudioData clip
AudioDataFormat format = AudioDataFormat.builder().setSampleRate(16000f).build();
AudioData audioData = AudioData.create(format, clipSamples.length);
audioData.load(clipSamples);
AudioClassifierResult result = audioClassifier.classify(audioData);

Audio stream

AUDIO_STREAMAsync · result callbackAsynchronous microphone pipeline with monotonic timestamps and a result callback. Accumulates contiguous PCM blocks into fixed-length spectrogram windows.
import android.os.SystemClock;
import android.util.Log;
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier;
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier.AudioClassifierOptions;
import com.google.mediapipe.tasks.audio.core.RunningMode;
import com.google.mediapipe.tasks.components.containers.AudioData;
import com.google.mediapipe.tasks.core.BaseOptions;

BaseOptions baseOptions = BaseOptions.builder()
    .setModelAssetPath("yamnet.tflite")
    .build();
AudioClassifierOptions options = AudioClassifierOptions.builder()
    .setBaseOptions(baseOptions)
    .setRunningMode(RunningMode.AUDIO_STREAM)
    .setMaxResults(5)
    .setResultListener(result -> { var results = result.classificationResults(); })
    .setErrorListener(error -> Log.e("MediaPipe", String.valueOf(error.getMessage())))
    .build();

AudioClassifier audioClassifier = AudioClassifier.createFromOptions(context, options);

// Async: accumulates contiguous microphone PCM chunks into spectrogram windows
if (audioData.load(audioRecord) > 0) {
    audioClassifier.classifyAsync(audioData, SystemClock.uptimeMillis());
}

iOS

SwiftPMhttps://github.com/google-ai-edge/mediapipe → MediaPipeTasksAudioPackage.swift·Sample code·Setup guide

Audio clips

AUDIO_CLIPSSync · complete clipSynchronous classification across every window of a complete prerecorded audio clip.
import Foundation
import MediaPipeTasksAudio

guard let modelPath = Bundle.main.path(forResource: "yamnet", ofType: "tflite") else { return }

let options = AudioClassifierOptions()
options.baseOptions.modelAssetPath = modelPath
options.runningMode = .audioClips
options.maxResults = 5

let audioClassifier = try AudioClassifier(options: options)

// Synchronous classification across every window of a complete audio clip
let result = try audioClassifier.classify(audioClip: audioData)

Audio stream

AUDIO_STREAMAsync · result callbackAsynchronous microphone pipeline with monotonic timestamps and a result callback. Accumulates contiguous PCM blocks into fixed-length spectrogram windows.
import Foundation
import MediaPipeTasksAudio

final class AudioClassifierStreamService: NSObject {
    private var audioClassifier: AudioClassifier?

    func setUp() throws {
        guard let modelPath = Bundle.main.path(forResource: "yamnet", ofType: "tflite") else { return }

        let options = AudioClassifierOptions()
        options.baseOptions.modelAssetPath = modelPath
        options.runningMode = .audioStream
        options.maxResults = 5
        options.audioClassifierStreamDelegate = self

        audioClassifier = try AudioClassifier(options: options)
    }

    func classifyAsync(audioData: AudioData, timestampMs: Int) throws {
        // Async: accumulates contiguous microphone PCM blocks into spectrogram windows
        try audioClassifier?.classifyAsync(audioBlock: audioData, timestampInMilliseconds: timestampMs)
    }
}

extension AudioClassifierStreamService: AudioClassifierStreamDelegate {
    func audioClassifier(
        _ audioClassifier: AudioClassifier,
        didFinishClassification result: AudioClassifierResult?,
        timestampInMilliseconds: Int,
        error: (any Error)?
    ) {
        guard let result else { return }
        DispatchQueue.main.async {
            let categories = result.classificationResult.classifications.first?.categories
        }
    }
}

Python

pippip install mediapipeSample code·Setup guide

Audio clips

AUDIO_CLIPSSync · complete clipSynchronous classification across every window of a complete prerecorded audio clip.
import mediapipe as mp

BaseOptions = mp.tasks.BaseOptions
AudioClassifier = mp.tasks.audio.AudioClassifier
AudioClassifierOptions = mp.tasks.audio.AudioClassifierOptions
AudioRunningMode = mp.tasks.audio.RunningMode
AudioData = mp.tasks.components.containers.AudioData

options = AudioClassifierOptions(
    base_options=BaseOptions(model_asset_path="yamnet.tflite"),
    running_mode=AudioRunningMode.AUDIO_CLIPS,
    max_results=5,
)

audio_data = AudioData.create_from_array(clip_samples, sample_rate=16000)
with AudioClassifier.create_from_options(options) as classifier:
    # Synchronous: returns timestamped results for every window across the clip
    result_list = classifier.classify(audio_data)

Audio stream

AUDIO_STREAMAsync · result callbackAsynchronous microphone pipeline with monotonic timestamps and a result callback. Accumulates contiguous PCM blocks into fixed-length spectrogram windows.
import mediapipe as mp

BaseOptions = mp.tasks.BaseOptions
AudioClassifier = mp.tasks.audio.AudioClassifier
AudioClassifierOptions = mp.tasks.audio.AudioClassifierOptions
AudioRunningMode = mp.tasks.audio.RunningMode
AudioData = mp.tasks.components.containers.AudioData

def on_result(result, timestamp_ms: int):
    print(result.classifications[0].categories)

options = AudioClassifierOptions(
    base_options=BaseOptions(model_asset_path="yamnet.tflite"),
    running_mode=AudioRunningMode.AUDIO_STREAM,
    max_results=5,
    result_callback=on_result,
)

with AudioClassifier.create_from_options(options) as classifier:
    # Async: accumulates contiguous PCM blocks into spectrogram windows
    audio_data = AudioData.create_from_array(pcm_chunk, sample_rate=16000)
    classifier.classify_async(audio_data, timestamp_ms)

Task details

This section describes the capabilities, inputs, outputs, and configuration options of this task.

Features

  • Input audio processing - Processing includes audio resampling, buffering, framing, and fourier transform.
  • Label map locale - Set the language used for display names
  • Score threshold - Filter results based on prediction scores.
  • Top-k classification - Limit the number of classification results.
  • Label allowlist and denylist - Specify the categories detected.
Inputs
The Audio Classifier accepts an input of one of the following data types:
  • Audio clips
  • Audio stream

See Prepare audio input data for AudioData, AudioRecord, and PCM buffer examples.

Outputs
The Audio Classifier outputs a list of categories containing:
  • Category index: the index of the category in the model outputs
  • Score: the confidence score for this category, usually a probability in [0,1]
  • Category name (optional): the name of the category as specified in the LiteRT model metadata, if available
  • Category display name (optional): a display name for the category as specified in the LiteRT model metadata, in the language specified through display names locale options, if available

Configuration options

This task has the following configuration options:

Web

Option Name Description Value Range Default Value
runningMode Sets the running mode for the task:

AUDIO_CLIPS: Independent audio clips.
AUDIO_STREAM: Continuous audio stream.		 {AUDIO_CLIPS, AUDIO_STREAM} AUDIO_CLIPS
displayNamesLocale Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. Locale code "en"
maxResults Maximum number of top-scored classification results to return (-1 returns all results). Integer -1
scoreThreshold Prediction score threshold overriding the model metadata default. Classifications lower than this score are rejected. Float [0.0, 1.0] Not set
categoryAllowlist Optional list of allowed category names. Mutually exclusive with categoryDenylist. List of strings Not set
categoryDenylist Optional list of excluded category names. Mutually exclusive with categoryAllowlist. List of strings Not set

Android

Option Name Description Value Range Default Value
runningMode Sets the running mode for the task:

AUDIO_CLIPS: Independent audio clips.
AUDIO_STREAM: Continuous audio livestream with asynchronous callback/delegate.		 {AUDIO_CLIPS, AUDIO_STREAM} AUDIO_CLIPS
displayNamesLocale Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. Locale code "en"
maxResults Maximum number of top-scored classification results to return (-1 returns all results). Integer -1
scoreThreshold Prediction score threshold overriding the model metadata default. Classifications lower than this score are rejected. Float [0.0, 1.0] Not set
categoryAllowlist Optional list of allowed category names. Mutually exclusive with categoryDenylist. List of strings Not set
categoryDenylist Optional list of excluded category names. Mutually exclusive with categoryAllowlist. List of strings Not set
resultListener Asynchronous callback for AUDIO_STREAM mode results. OutputHandler.ResultListener<AudioClassifierResult, Long> Not set
errorListener Optional error listener for AUDIO_STREAM mode. ErrorListener Not set

iOS

Option Name Description Value Range Default Value
runningMode Sets the running mode for the task:

AUDIO_CLIPS: Independent audio clips.
AUDIO_STREAM: Continuous audio livestream with asynchronous callback/delegate.		 {AUDIO_CLIPS, AUDIO_STREAM} AUDIO_CLIPS
displayNamesLocale Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. Locale code "en"
maxResults Maximum number of top-scored classification results to return (-1 returns all results). Integer -1
scoreThreshold Prediction score threshold overriding the model metadata default. Classifications lower than this score are rejected. Float [0.0, 1.0] Not set
categoryAllowlist Optional list of allowed category names. Mutually exclusive with categoryDenylist. List of strings Not set
categoryDenylist Optional list of excluded category names. Mutually exclusive with categoryAllowlist. List of strings Not set
audioClassifierStreamDelegate Asynchronous delegate for AUDIO_STREAM mode results (invoked on a serial background queue). AudioClassifierStreamDelegate Not set

Python

Option Name Description Value Range Default Value
running_mode Sets the running mode for the task:

AUDIO_CLIPS: Independent audio clips.
AUDIO_STREAM: Continuous audio livestream with result_callback.		 {AUDIO_CLIPS, AUDIO_STREAM} AUDIO_CLIPS
display_names_locale Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. Locale code "en"
max_results Maximum number of top-scored classification results to return (-1 returns all results). Integer -1
score_threshold Prediction score threshold overriding the model metadata default. Classifications lower than this score are rejected. Float [0.0, 1.0] Not set
category_allowlist Optional list of allowed category names. Mutually exclusive with category_denylist. List of strings Not set
category_denylist Optional list of excluded category names. Mutually exclusive with category_allowlist. List of strings Not set
result_callback Asynchronous callback function for AUDIO_STREAM mode results. Callable Not set

Task output

Each AudioClassifierResult returns timestamped classification windows (for example, 975 ms YAMNet windows across an audio clip or stream):

Field Type / Range Description
timestampMs number (milliseconds) Start timestamp of the classified audio window (timestamp_ms in Python).
classifications Classifications[] Per-head classification results (headIndex, headName, categories).
categories Category[] (score in [0.0, 1.0]) Ranked acoustic event classes (categoryName, displayName, index, score).

The following example shows the top acoustic event predictions for the first audio window (timestampMs: 0):

[
  {
    timestampMs: 0,
    classifications: [
      {
        headIndex: 0,
        categories: [
          { index: 0, score: 0.6000, categoryName: "Speech" },
          { index: 137, score: 0.2000, categoryName: "Music" }
        ]
      }
    ]
  }
]

Models

Audio Classifier pipeline overview

The Audio Classifier requires an audio classification model to be downloaded and stored in your project directory. Start with the default, recommended model for your target platform when you start developing with this task. The other available models typically make trade-offs between performance, accuracy, resolution, and resource requirements, and in some cases, include additional features.

The Yamnet model is an audio event classifier trained on the AudioSet dataset to predict audio events defined in the AudioSet data. For information on the audio events recognized by this model, see the model labels list.

Model name Input shape Quantization type Versions
YamNet 1 x 15600 None (float32) Latest

Task benchmarks

Here are the task benchmarks for the whole pipeline based on the preceding pre-trained models. The latency result is the average latency on Pixel 6 using CPU / GPU.

Model name Device CPU latency GPU latency
YamNet Pixel 6 12.29ms -