The MediaPipe Image Segmenter task lets you divide images into regions based on predefined categories. You can use this functionality to identify specific objects or textures, and then apply visual effects such as background blurring. This task includes several models specifically trained for segmenting people and their features within image data, including:
- Person and background
- Person's hair only
- Person's hair, face, skin, clothing, and accessories
This task operates on image data with a machine learning (ML) model with single images or a continuous video stream. It outputs a list of segmented regions, representing objects or areas in an image, depending on the model you choose.
Quickstart
Install the SDK for your platform, prepare input data, choose a running mode, and run inference.
Web
Image
import { FilesetResolver, ImageSegmenter } from "@mediapipe/tasks-vision"; const vision = await FilesetResolver.forVisionTasks( "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm" ); const imageSegmenter = await ImageSegmenter.createFromOptions(vision, { baseOptions: { modelAssetPath: "deeplab_v3.tflite", delegate: "GPU" }, runningMode: "IMAGE", outputCategoryMask: true, outputConfidenceMasks: false, }); // Synchronous single-frame call (stateless across images) const image = document.getElementById("image") as HTMLImageElement; imageSegmenter.segment(image, (result) => { const mask = result.categoryMask?.getAsUint8Array(); });
Video
import { FilesetResolver, ImageSegmenter } from "@mediapipe/tasks-vision"; const vision = await FilesetResolver.forVisionTasks( "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm" ); const imageSegmenter = await ImageSegmenter.createFromOptions(vision, { baseOptions: { modelAssetPath: "deeplab_v3.tflite", delegate: "GPU" }, runningMode: "VIDEO", outputCategoryMask: true, outputConfidenceMasks: false, }); // Synchronous across decoded frames with monotonic timestamps (never drops frames) for (const { frame, timestampMs } of decodedVideoFrames) { imageSegmenter.segmentForVideo(frame, timestampMs, (result) => { const mask = result.categoryMask?.getAsUint8Array(); }); }
Livestream
import { FilesetResolver, ImageSegmenter } from "@mediapipe/tasks-vision"; const vision = await FilesetResolver.forVisionTasks( "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm" ); // On Web, live camera loops pair runningMode: "VIDEO" with requestAnimationFrame const imageSegmenter = await ImageSegmenter.createFromOptions(vision, { baseOptions: { modelAssetPath: "deeplab_v3.tflite", delegate: "GPU" }, runningMode: "VIDEO", outputCategoryMask: true, outputConfidenceMasks: false, }); const video = document.getElementById("webcam") as HTMLVideoElement; video.srcObject = await navigator.mediaDevices.getUserMedia({ video: true }); await video.play(); let lastVideoTime = -1; function renderLoop() { if (video.readyState >= 2 && video.currentTime !== lastVideoTime) { lastVideoTime = video.currentTime; imageSegmenter.segmentForVideo(video, performance.now(), (result) => { const mask = result.categoryMask?.getAsUint8Array(); }); } requestAnimationFrame(renderLoop); } renderLoop();
Android (Kotlin)
Image
import com.google.mediapipe.framework.image.BitmapImageBuilder import com.google.mediapipe.framework.image.MPImage import com.google.mediapipe.tasks.core.BaseOptions import com.google.mediapipe.tasks.vision.core.RunningMode import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenter val baseOptions = BaseOptions.builder() .setModelAssetPath("deeplab_v3.tflite") .build() val options = ImageSegmenter.ImageSegmenterOptions.builder() .setBaseOptions(baseOptions) .setRunningMode(RunningMode.IMAGE) .setOutputCategoryMask(true) .setOutputConfidenceMasks(false) .build() val imageSegmenter = ImageSegmenter.createFromOptions(context, options) // Synchronous single-image call (no temporal tracking across calls) val mpImage = BitmapImageBuilder(bitmap).build() val result = imageSegmenter.segment(mpImage)
Video
import com.google.mediapipe.framework.image.BitmapImageBuilder import com.google.mediapipe.framework.image.MPImage import com.google.mediapipe.tasks.core.BaseOptions import com.google.mediapipe.tasks.vision.core.RunningMode import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenter val baseOptions = BaseOptions.builder() .setModelAssetPath("deeplab_v3.tflite") .build() val options = ImageSegmenter.ImageSegmenterOptions.builder() .setBaseOptions(baseOptions) .setRunningMode(RunningMode.VIDEO) .setOutputCategoryMask(true) .setOutputConfidenceMasks(false) .build() val imageSegmenter = ImageSegmenter.createFromOptions(context, options) // Synchronous: processes every frame in order with monotonic timestamps (no frame drops) val mpImage = BitmapImageBuilder(frameBitmap).build() val result = imageSegmenter.segmentForVideo(mpImage, frameTimestampMs)
Livestream
import android.os.SystemClock import android.util.Log import com.google.mediapipe.framework.image.BitmapImageBuilder import com.google.mediapipe.framework.image.MPImage import com.google.mediapipe.tasks.core.BaseOptions import com.google.mediapipe.tasks.vision.core.RunningMode import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenter val baseOptions = BaseOptions.builder() .setModelAssetPath("deeplab_v3.tflite") .build() val options = ImageSegmenter.ImageSegmenterOptions.builder() .setBaseOptions(baseOptions) .setRunningMode(RunningMode.LIVE_STREAM) .setOutputCategoryMask(true) .setOutputConfidenceMasks(false) .setResultListener { result, _ -> val categoryMask = result.categoryMask() } .setErrorListener { error -> Log.e("MediaPipe", error.message.orEmpty()) } .build() val imageSegmenter = ImageSegmenter.createFromOptions(context, options) // Async: automatically drops incoming camera frames when busy so results stay current imageSegmenter.segmentAsync(mpImage, SystemClock.uptimeMillis())
Android (Java)
Image
import com.google.mediapipe.framework.image.BitmapImageBuilder; import com.google.mediapipe.framework.image.MPImage; import com.google.mediapipe.tasks.core.BaseOptions; import com.google.mediapipe.tasks.vision.core.RunningMode; import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenter; import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenterResult; BaseOptions baseOptions = BaseOptions.builder() .setModelAssetPath("deeplab_v3.tflite") .build(); ImageSegmenter.ImageSegmenterOptions options = ImageSegmenter.ImageSegmenterOptions.builder() .setBaseOptions(baseOptions) .setRunningMode(RunningMode.IMAGE) .setOutputCategoryMask(true) .setOutputConfidenceMasks(false) .build(); ImageSegmenter imageSegmenter = ImageSegmenter.createFromOptions(context, options); // Synchronous single-image call (no temporal tracking across calls) MPImage mpImage = new BitmapImageBuilder(bitmap).build(); ImageSegmenterResult result = imageSegmenter.segment(mpImage);
Video
import com.google.mediapipe.framework.image.BitmapImageBuilder; import com.google.mediapipe.framework.image.MPImage; import com.google.mediapipe.tasks.core.BaseOptions; import com.google.mediapipe.tasks.vision.core.RunningMode; import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenter; import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenterResult; BaseOptions baseOptions = BaseOptions.builder() .setModelAssetPath("deeplab_v3.tflite") .build(); ImageSegmenter.ImageSegmenterOptions options = ImageSegmenter.ImageSegmenterOptions.builder() .setBaseOptions(baseOptions) .setRunningMode(RunningMode.VIDEO) .setOutputCategoryMask(true) .setOutputConfidenceMasks(false) .build(); ImageSegmenter imageSegmenter = ImageSegmenter.createFromOptions(context, options); // Synchronous: processes every frame in order with monotonic timestamps (no frame drops) MPImage mpImage = new BitmapImageBuilder(frameBitmap).build(); ImageSegmenterResult result = imageSegmenter.segmentForVideo(mpImage, frameTimestampMs);
Livestream
import android.os.SystemClock; import android.util.Log; import com.google.mediapipe.framework.image.BitmapImageBuilder; import com.google.mediapipe.framework.image.MPImage; import com.google.mediapipe.tasks.core.BaseOptions; import com.google.mediapipe.tasks.vision.core.RunningMode; import com.google.mediapipe.tasks.vision.imagesegmenter.ImageSegmenter; BaseOptions baseOptions = BaseOptions.builder() .setModelAssetPath("deeplab_v3.tflite") .build(); ImageSegmenter.ImageSegmenterOptions options = ImageSegmenter.ImageSegmenterOptions.builder() .setBaseOptions(baseOptions) .setRunningMode(RunningMode.LIVE_STREAM) .setOutputCategoryMask(true) .setOutputConfidenceMasks(false) .setResultListener((result, inputImage) -> { var categoryMask = result.categoryMask(); }) .setErrorListener(error -> Log.e("MediaPipe", String.valueOf(error.getMessage()))) .build(); ImageSegmenter imageSegmenter = ImageSegmenter.createFromOptions(context, options); // Async: automatically drops incoming camera frames when busy so results stay current imageSegmenter.segmentAsync(mpImage, SystemClock.uptimeMillis());
iOS
Image
import Foundation import MediaPipeTasksVision guard let modelPath = Bundle.main.path(forResource: "deeplab_v3", ofType: "tflite") else { return } let options = ImageSegmenterOptions() options.baseOptions.modelAssetPath = modelPath options.runningMode = .image options.shouldOutputCategoryMask = true options.shouldOutputConfidenceMasks = false let imageSegmenter = try ImageSegmenter(options: options) // Synchronous single-image call (stateless across images) let mpImage = try MPImage(uiImage: image) let result = try imageSegmenter.segment(image: mpImage)
Video
import Foundation import MediaPipeTasksVision guard let modelPath = Bundle.main.path(forResource: "deeplab_v3", ofType: "tflite") else { return } let options = ImageSegmenterOptions() options.baseOptions.modelAssetPath = modelPath options.runningMode = .video options.shouldOutputCategoryMask = true options.shouldOutputConfidenceMasks = false let imageSegmenter = try ImageSegmenter(options: options) // Synchronous: processes every frame in order with monotonic timestamps (no frame drops) let mpImage = try MPImage(pixelBuffer: pixelBuffer) let result = try imageSegmenter.segment(videoFrame: mpImage, timestampInMilliseconds: timestampMs)
Livestream
import Foundation import MediaPipeTasksVision final class ImageSegmenterStreamService: NSObject { private var imageSegmenter: ImageSegmenter? func setUp() throws { guard let modelPath = Bundle.main.path(forResource: "deeplab_v3", ofType: "tflite") else { return } let options = ImageSegmenterOptions() options.baseOptions.modelAssetPath = modelPath options.runningMode = .liveStream options.shouldOutputCategoryMask = true options.shouldOutputConfidenceMasks = false options.imageSegmenterLiveStreamDelegate = self imageSegmenter = try ImageSegmenter(options: options) } func segmentAsync(mpImage: MPImage, timestampMs: Int) throws { // Async: automatically drops incoming camera frames when busy so results stay current try imageSegmenter?.segmentAsync(image: mpImage, timestampInMilliseconds: timestampMs) } } extension ImageSegmenterStreamService: ImageSegmenterLiveStreamDelegate { func imageSegmenter( _ imageSegmenter: ImageSegmenter, didFinishSegmentation result: ImageSegmenterResult?, timestampInMilliseconds: Int, error: (any Error)? ) { guard let result else { return } DispatchQueue.main.async { let categoryMask = result.categoryMask } } }
Python
Image
import mediapipe as mp BaseOptions = mp.tasks.BaseOptions ImageSegmenter = mp.tasks.vision.ImageSegmenter ImageSegmenterOptions = mp.tasks.vision.ImageSegmenterOptions VisionRunningMode = mp.tasks.vision.RunningMode options = ImageSegmenterOptions( base_options=BaseOptions(model_asset_path="deeplab_v3.tflite"), running_mode=VisionRunningMode.IMAGE, output_category_mask=True, output_confidence_masks=False, ) with ImageSegmenter.create_from_options(options) as segmenter: # Synchronous single-image call (stateless across images) mp_image = mp.Image.create_from_file("input.jpg") result = segmenter.segment(mp_image)
Video
import mediapipe as mp BaseOptions = mp.tasks.BaseOptions ImageSegmenter = mp.tasks.vision.ImageSegmenter ImageSegmenterOptions = mp.tasks.vision.ImageSegmenterOptions VisionRunningMode = mp.tasks.vision.RunningMode options = ImageSegmenterOptions( base_options=BaseOptions(model_asset_path="deeplab_v3.tflite"), running_mode=VisionRunningMode.VIDEO, output_category_mask=True, output_confidence_masks=False, ) with ImageSegmenter.create_from_options(options) as segmenter: # Synchronous: processes every frame in order with monotonic timestamps (no frame drops) for mp_image, timestamp_ms in decoded_video_frames: result = segmenter.segment_for_video(mp_image, timestamp_ms)
Livestream
import mediapipe as mp BaseOptions = mp.tasks.BaseOptions ImageSegmenter = mp.tasks.vision.ImageSegmenter ImageSegmenterOptions = mp.tasks.vision.ImageSegmenterOptions VisionRunningMode = mp.tasks.vision.RunningMode def on_result(result, output_image: mp.Image, timestamp_ms: int): print(f"Category mask shape at {timestamp_ms} ms: {result.category_mask.numpy_view().shape}") options = ImageSegmenterOptions( base_options=BaseOptions(model_asset_path="deeplab_v3.tflite"), running_mode=VisionRunningMode.LIVE_STREAM, output_category_mask=True, output_confidence_masks=False, result_callback=on_result, ) with ImageSegmenter.create_from_options(options) as segmenter: # Async: automatically drops incoming camera frames when busy so results stay current mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame) segmenter.segment_async(mp_image, timestamp_ms)
Task details
This section describes the capabilities, inputs, outputs, and configuration options of this task.
Features
- Input image processing - Processing includes image rotation, resizing, normalization, and color space conversion.
- Still images
- Decoded video frames
- Live video feed
See Prepare vision input data for MPImage, mp.Image, and browser media examples.
CATEGORY_MASK: a list containing a segmented mask as a uint8 format image. Each pixel value indicates if it is part of a specific segment category supported by the model.CONFIDENCE_MASK: a list of channels containing a segmented mask with pixel values in float32 format. Each pixel value indicates the level of confidence that it is part of a specific category supported by the model.
Configuration options
This task has the following configuration options:
Web
| Option Name | Description | Value Range | Default Value |
|---|---|---|---|
runningMode |
Sets the running mode for the task:IMAGE: Single image inputs.VIDEO: Decoded video frames and live camera streams. |
{IMAGE, VIDEO} |
IMAGE |
outputCategoryMask |
If true, outputs a uint8 category mask where each pixel value indicates the winning class index (shouldOutputCategoryMask on iOS). |
Boolean |
false |
outputConfidenceMasks |
If true, outputs per-category floating-point confidence masks in [0.0, 1.0] (shouldOutputConfidenceMasks on iOS). |
Boolean |
true |
displayNamesLocale |
Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. | Locale code | "en" |
Android
| Option Name | Description | Value Range | Default Value |
|---|---|---|---|
runningMode |
Sets the running mode for the task:IMAGE: Single image inputs.VIDEO: Decoded frames of a video.LIVE_STREAM: Livestream of camera input data (requires asynchronous listener/delegate). |
{IMAGE, VIDEO, LIVE_STREAM} |
IMAGE |
outputCategoryMask |
If true, outputs a uint8 category mask where each pixel value indicates the winning class index (shouldOutputCategoryMask on iOS). |
Boolean |
false |
outputConfidenceMasks |
If true, outputs per-category floating-point confidence masks in [0.0, 1.0] (shouldOutputConfidenceMasks on iOS). |
Boolean |
true |
displayNamesLocale |
Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. | Locale code | "en" |
resultListener |
Asynchronous callback for LIVE_STREAM mode results. |
OutputHandler.ResultListener<ImageSegmenterResult, MPImage> |
Not set |
errorListener |
Optional error listener for LIVE_STREAM mode. |
ErrorListener |
Not set |
iOS
| Option Name | Description | Value Range | Default Value |
|---|---|---|---|
runningMode |
Sets the running mode for the task:IMAGE: Single image inputs.VIDEO: Decoded frames of a video.LIVE_STREAM: Livestream of camera input data (requires asynchronous listener/delegate). |
{IMAGE, VIDEO, LIVE_STREAM} |
IMAGE |
shouldOutputCategoryMask |
If true, outputs a uint8 category mask where each pixel value indicates the winning class index (shouldOutputCategoryMask on iOS). |
Boolean |
false |
shouldOutputConfidenceMasks |
If true, outputs per-category floating-point confidence masks in [0.0, 1.0] (shouldOutputConfidenceMasks on iOS). |
Boolean |
true |
displayNamesLocale |
Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. | Locale code | "en" |
imageSegmenterLiveStreamDelegate |
Asynchronous delegate for LIVE_STREAM mode results (invoked on a serial background queue). |
ImageSegmenterLiveStreamDelegate |
Not set |
Python
| Option Name | Description | Value Range | Default Value |
|---|---|---|---|
running_mode |
Sets the running mode for the task:IMAGE: Single image inputs.VIDEO: Decoded frames of a video.LIVE_STREAM: Livestream of camera input data (requires result_callback). |
{IMAGE, VIDEO, LIVE_STREAM} |
IMAGE |
output_category_mask |
If true, outputs a uint8 category mask where each pixel value indicates the winning class index (shouldOutputCategoryMask on iOS). |
Boolean |
False |
output_confidence_masks |
If true, outputs per-category floating-point confidence masks in [0.0, 1.0] (shouldOutputConfidenceMasks on iOS). |
Boolean |
True |
display_names_locale |
Locale code for display names provided in the model metadata, if available. You can add localized labels to custom models with the LiteRT Metadata Writer API. | Locale code | "en" |
result_callback |
Asynchronous callback function for LIVE_STREAM mode results. |
Callable |
Not set |
Task output
Each ImageSegmenterResult provides pixel-aligned masks matching the dimensions of the input image:
| Field | Type / Range | Description |
|---|---|---|
categoryMask |
MPMask (Uint8 in [0, 255]) |
Single-channel mask where each pixel value is the winning class index (0..N-1) with the highest score (category_mask in Python). |
confidenceMasks |
MPMask[] (Float32 in [0.0, 1.0]) |
Per-class float masks where each pixel value represents the probability that the pixel belongs to that category (confidence_masks in Python). |
qualityScores |
float[] ([0.0, 1.0]) |
Optional model quality scores for each segmentation mask (quality_scores in Python). |
Example ImageSegmenterResult payload
The following example shows a 640×480 category mask and per-class confidence masks:
{
categoryMask: {
width: 640,
height: 480,
data: Uint8Array(307200) [0, 0, 15, 15, 15 /* ... winning class index per pixel */]
},
confidenceMasks: [
{
width: 640,
height: 480,
data: Float32Array(307200) [0.992, 0.988, 0.014 /* ... class #0 probability per pixel */]
}
]
}
Models
The Image Segmenter can be used with more than one ML model. Most of the following segmentation models are built and trained to perform segmentation with images of people. However, the DeepLab-v3 model is built as a general purpose image segmenter. Select the model that fits best for your application.
Selfie segmentation model
This model can segment the portrait of a person, and can be used for replacing or modifying the background in an image. The model outputs two categories, background at index 0 and person at index 1. This model has versions with different input shapes, including a square version and a landscape version that may be more efficient for applications where the input is always that shape, such as video calls.
| Model name | Input shape | Quantization type | Model Card | Versions |
|---|---|---|---|---|
| SelfieSegmenter (square) | 256 x 256 | float 16 | info | Latest |
| SelfieSegmenter (landscape) | 144 x 256 | float 16 | info | Latest |
Hair segmentation model
This model takes an image of a person, locates the hair on their head, and outputs an image segmentation map for their hair. You can use this model for recoloring hair or applying other hair effects. The model outputs the following segmentation categories:
| Model name | Input shape | Quantization type | Model Card | Versions |
|---|---|---|---|---|
| HairSegmenter | 512 x 512 | None (float32) | info | Latest |
Multi-class selfie segmentation model
This model takes an image of a person, locates regions for different categories such as hair, skin, and clothing, and outputs an image segmentation map for these items. You can use this model for applying various effects to people in images or video. The model outputs the following segmentation categories:
| Model name | Input shape | Quantization type | Model Card | Versions |
|---|---|---|---|---|
| SelfieMulticlass (256 x 256) | 256 x 256 | None (float32) | info | Latest |
DeepLab-v3 model
This model identifies segments for a number of categories, including background, person, cat, dog, and potted plant. The model uses atrous spatial pyramid pooling to capture longer range information. For more information, see DeepLab-v3.
| Model name | Input shape | Quantization type | Versions |
|---|---|---|---|
| DeepLab-V3 | 257 x 257 | None (float32) | Latest |
Task benchmarks
Here are the task benchmarks for the whole pipeline based on the preceding pre-trained models. The latency result is the average latency on Pixel 6 using CPU / GPU.
| Model name | Device | CPU latency | GPU latency |
|---|---|---|---|
| SelfieSegmenter (square) | Pixel 6 | 33.46ms | 35.15ms |
| SelfieSegmenter (landscape) | Pixel 6 | 34.19ms | 33.55ms |
| HairSegmenter | Pixel 6 | 57.90ms | 52.14ms |
| SelfieMulticlass (256 x 256) | Pixel 6 | 217.76ms | 71.24ms |
| DeepLab-V3 | Pixel 6 | 123.93ms | 103.30ms |
