Introducing Google AI Edge Portal: Benchmark Edge AI at scale. Sign-up to request access during private preview.

Audio classification guide for iOS

The MediaPipe Audio Classifier task lets you perform classification on audio data. You can use this task to identify sound events from a set of trained categories. These instructions show you how to use the Audio Classifier in iOS apps. The code sample described in these instructions is available on GitHub.

For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The MediaPipe Tasks example code is a basic implementation of a Audio Classifier app for iOS. The example uses the microphone on a physical iOS device to continuously classify sounds, and can also run the classifier on sound files stored on the device.

You can use the app as a starting point for your own iOS app, or refer to it when modifying an existing app. You can refer to the Audio Classifier example code on GitHub.

Download the code

The following instructions show you how to create a local copy of the example code using the git command line tool.

To download the example code:

Clone the git repository using the following command:

git clone https://github.com/google-ai-edge/mediapipe-samples

Optionally, configure your git instance to use sparse checkout, so you have only the files for the Audio Classifier example app:
```
cd mediapipe-samples
git sparse-checkout init --cone
git sparse-checkout set examples/audio_classifier/ios/
```

After creating a local version of the example code, you can install the MediaPipe task library, open the project using Xcode, and run the app. For instructions, see the Setup Guide for iOS.

Key components

The following files contain the crucial code for the Audio Classifier example application:

AudioClassifierHelper.swift: Initializes the audio classifier and handles the model selection.
ViewController.swift: Implements the UI and formats the results.

Setup

This section describes key steps for setting up your development environment and code projects to use Audio Classifier. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for iOS.

Dependencies

Audio Classifier uses the MediaPipeTasksAudio library, which must be installed using CocoaPods. The library is compatible with both Swift and Objective-C apps and does not require any additional language-specific setup.

For instructions to install CocoaPods on macOS, refer to the CocoaPods installation guide. For instructions on how to create a Podfile with the necessary pods for your app, refer to Using CocoaPods.

Add the MediaPipeTasksAudio pod in the Podfile using the following code:

target 'MyAudioClassifierApp' do
  use_frameworks!
  pod 'MediaPipeTasksAudio'
end

If your app includes unit test targets, refer to the Set Up Guide for iOS for additional information on setting up your Podfile.

Model

The MediaPipe Audio Classifier task requires a trained model that is compatible with this task. For more information about the available trained models for Audio Classifier, see the task overview Models section.

Select and download a model, and add it to your project directory using Xcode. For instructions on how to add files to your Xcode project, refer to Managing files and folders in your Xcode project.

Use the BaseOptions.modelAssetPath property to specify the path to the model in your app bundle. For a code example, see the next section.

Create the task

You can create the Audio Classifier task by calling one of its initializers. The AudioClassifier(options:) initializer sets values for the configuration options.

If you don't need a Audio Classifier initialized with customized configuration options, you can use the AudioClassifier(modelPath:) initializer to create a Audio Classifier with the default options. For more information about configuration options, see Configuration Overview.

The Audio Classifier task supports two running modes: audio clips and audio streams. You need to specify the running mode corresponding to your input data type when creating the task.

Choose the tab corresponding to your language and running mode to see how to create the task and run inference.

Swift

Audio clips

import MediaPipeTasksAudio

let modelPath = Bundle.main.path(forResource: "yamnet",
                                      ofType: "tflite")

let options = AudioClassifierOptions()
options.baseOptions.modelAssetPath = modelPath
options.runningMode = .audioClips
options.maxResults = 5

let audioClassifier = try AudioClassifier(options: options)

Audio stream

import MediaPipeTasksAudio

let modelPath = Bundle.main.path(forResource: "yamnet",
                                      ofType: "tflite")

let options = AudioClassifierOptions()
options.baseOptions.modelAssetPath = modelPath
options.runningMode = .audioStream
options.maxResults = 5
options.audioClassifierStreamDelegate = self

let audioClassifier = try AudioClassifier(options: options)

// Define the delegate class or extension
extension ViewController: AudioClassifierStreamDelegate {
  func audioClassifier(
    _ audioClassifier: AudioClassifier,
    didFinishClassification result: AudioClassifierResult?,
    timestampInMilliseconds: Int,
    error: Error?
  ) {
    // Process the classification result here
  }
}

Objective-C

Audio clips

@import MediaPipeTasksAudio;

NSString *modelPath = [[NSBundle mainBundle] pathForResource:@"yamnet"
                                                      ofType:@"tflite"];

MPPAudioClassifierOptions *options = [[MPPAudioClassifierOptions alloc] init];
options.baseOptions.modelAssetPath = modelPath;
options.runningMode = MPPAudioRunningModeAudioClips;
options.maxResults = 5;

MPPAudioClassifier *audioClassifier =
      [[MPPAudioClassifier alloc] initWithOptions:options error:nil];

Audio stream

@import MediaPipeTasksAudio;

NSString *modelPath = [[NSBundle mainBundle] pathForResource:@"yamnet"
                                                      ofType:@"tflite"];

MPPAudioClassifierOptions *options = [[MPPAudioClassifierOptions alloc] init];
options.baseOptions.modelAssetPath = modelPath;
options.runningMode = MPPAudioRunningModeAudioStream;
options.maxResults = 5;
options.audioClassifierStreamDelegate = self;

MPPAudioClassifier *audioClassifier =
      [[MPPAudioClassifier alloc] initWithOptions:options error:nil];

// Conform to MPPAudioClassifierStreamDelegate in interface or category
- (void)audioClassifier:(MPPAudioClassifier *)audioClassifier
    didFinishClassificationWithResult:(nullable MPPAudioClassifierResult *)result
              timestampInMilliseconds:(NSInteger)timestampInMilliseconds
                                error:(nullable NSError *)error {
  // Process the classification result here
}

Configuration options

This task has the following configuration options for iOS apps:

Option Name	Description	Value Range	Default Value
`runningMode`	Sets the running mode for the task. Audio Classifier has two modes: AUDIO_CLIPS: The mode for running the audio task on independent audio clips. AUDIO_STREAM: The mode for running the audio task on an audio stream, such as from microphone. In this mode, resultListener must be called to set up a listener to receive the classification results asynchronously.	{`AUDIO_CLIPS, AUDIO_STREAM`}	`AUDIO_CLIPS`
`displayNamesLocale`	Sets the language of labels to use for display names provided in the metadata of the task's model, if available. Default is `en` for English. You can add localized labels to the metadata of a custom model using the TensorFlow Lite Metadata Writer API.	Locale code	en
`maxResults`	Sets the optional maximum number of top-scored classification results to return. If < 0, all available results will be returned.	Any positive numbers	`-1`
`scoreThreshold`	Sets the prediction score threshold that overrides the one provided in the model metadata (if any). Results below this value are rejected.	[0.0, 1.0]	Not set
`categoryAllowlist`	Sets the optional list of allowed category names. If non-empty, classification results whose category name is not in this set will be filtered out. Duplicate or unknown category names are ignored. This option is mutually exclusive with `categoryDenylist` and using both results in an error.	Any strings	Not set
`categoryDenylist`	Sets the optional list of category names that are not allowed. If non-empty, classification results whose category name is in this set will be filtered out. Duplicate or unknown category names are ignored. This option is mutually exclusive with `categoryAllowlist` and using both results in an error.	Any strings	Not set
`audioClassifierStreamDelegate`	Sets the result listener to receive the classification results asynchronously when the Audio Classifier is in the audio stream mode. Can only be used when running mode is set to `AUDIO_STREAM`	N/A	Not set

Prepare data

Audio Classifier works with audio clips and audio streams. The task handles the data input preprocessing, including resampling, buffering, and framing. However, you must convert the input audio data to a AudioData object before passing it to the Audio Classifier task.

Swift

Audio clips

import MediaPipeTasksAudio

// Create AudioDataFormat
let format = AudioDataFormat(channelCount: 1, sampleRate: 16000)

// Create AudioData with format and sampleCount
let audioData = AudioData(format: format, sampleCount: floatArray.count)

// Convert float array to FloatBuffer
let floatBuffer = FloatBuffer(data: floatArray, length: floatArray.count)

// Load the float buffer into AudioData
try audioData.load(buffer: floatBuffer, offset: 0, length: floatArray.count)

Audio stream

import MediaPipeTasksAudio

// Create the AudioRecord instance
let audioRecord = try AudioClassifier.createAudioRecord(
  channelCount: 1,
  sampleRate: 16000,
  bufferLength: 8000
)

// Acquire permission and start recording
try audioRecord.startRecording()

...

// To load the audio samples from the microphone:
let format = AudioDataFormat(channelCount: 1, sampleRate: 16000)
let audioData = AudioData(format: format, sampleCount: 16000)
try audioData.load(audioRecord: audioRecord)

Objective-C

Audio clips

@import MediaPipeTasksAudio;

// Create MPPAudioDataFormat
MPPAudioDataFormat *format = [[MPPAudioDataFormat alloc] initWithChannelCount:1 sampleRate:16000];

// Create MPPAudioData with format and sampleCount
MPPAudioData *audioData = [[MPPAudioData alloc] initWithFormat:format sampleCount:length];

// Convert float array to MPPFloatBuffer
MPPFloatBuffer *floatBuffer = [[MPPFloatBuffer alloc] initWithData:floatData length:length];

// Load the float buffer into MPPAudioData
[audioData loadBuffer:floatBuffer offset:0 length:length error:nil];

Audio stream

@import MediaPipeTasksAudio;

NSError *error = nil;

// Create the MPPAudioRecord instance
MPPAudioRecord *audioRecord = [MPPAudioClassifier createAudioRecordWithChannelCount:1
                                                                         sampleRate:16000
                                                                       bufferLength:8000
                                                                              error:&error];

// Start recording
[audioRecord startRecordingWithError:&error];

...

// To load the audio samples from the microphone:
MPPAudioDataFormat *format = [[MPPAudioDataFormat alloc] initWithChannelCount:1 sampleRate:16000];
MPPAudioData *audioData = [[MPPAudioData alloc] initWithFormat:format sampleCount:16000];
[audioData loadAudioRecord:audioRecord error:&error];

Run the task

To run the Audio Classifier, call the classify function corresponding to your running mode.

Swift

Audio clips

let result = try audioClassifier.classify(audioClip: audioData)

Audio stream

// Run classification asynchronously. The results are delivered
// through the `audioClassifierStreamDelegate` callback.
try audioClassifier.classifyAsync(
  audioBlock: audioData,
  timestampInMilliseconds: timestampMs
)

Objective-C

Audio clips

MPPAudioClassifierResult *result = [audioClassifier classifyAudioClip:audioData
                                                                error:nil];

Audio stream

// Run classification asynchronously. The results are delivered
// through the `audioClassifierStreamDelegate` callback.
[audioClassifier classifyAsyncAudioBlock:audioData
                 timestampInMilliseconds:timestampMs
                                   error:nil];

Note: When running in .audioClips mode, the task blocks the current thread until it finishes running inference. When running in .audioStream mode, the task returns immediately and does not block the current thread.

Handle and display results

Upon running inference, the Audio Classifier task returns an AudioClassifierResult object which contains a list of predicted sound categories.

The following shows an example of the output data from this task:

AudioClassifierResult:
  Timestamp in microseconds: 100
  ClassificationResult #0:
    Timestamp in microseconds: 100  
    Classifications #0 (single classification head):
      head index: 0
      category #0:
        category name: "Speech"
        score: 0.6
        index: 0
      category #1:
        category name: "Music"
        score: 0.2
        index: 1

The ViewController.swift file in the example code demonstrates how to display the classification results returned from the task.

Audio classification guide for iOS Stay organized with collections Save and categorize content based on your preferences.

Code example

Download the code

Key components

Setup

Dependencies

Model

Create the task

Swift

Audio clips

Audio stream

Objective-C

Audio clips

Audio stream

Configuration options

Prepare data

Swift

Audio clips

Audio stream

Objective-C

Audio clips

Audio stream

Run the task

Swift

Audio clips

Audio stream

Objective-C

Audio clips

Audio stream

Handle and display results

Audio classification guide for iOS