Universal Embedder

The MediaPipe Universal Embedder task combines the features of individual audio, text, and image embedders into a single unified service, while also adding support for video embeddings. It runs on the on-device LiteRT-LM runtime to produce high-performance multimodal embeddings without any server round-trips. You can use the Universal Embedder to vectorize text, images, and audio samples, or run multimodal embedding extraction on lists of composite contents to capture semantic meaning across modalities.

Open in Web Demos Edit in StackBlitz

Quickstart

Install the SDK for your platform, initialize the task with a model, and run inference.

Web

npmnpm install @mediapipe/tasks-retrievalWeb demo·Edit in StackBlitz·Setup guide

import { FilesetResolver, UniversalEmbedder } from "@mediapipe/tasks-retrieval";

// 1. Create the task
const MODEL_URL = "https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm";

const retrieval = await FilesetResolver.forRetrievalTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-retrieval@latest/wasm"
);
const universalEmbedder = await UniversalEmbedder.createFromOptions(retrieval, {
  baseOptions: { modelAssetPath: MODEL_URL },
});

// 2. Run inference across modalities and compare cosine similarity
const textResult = await universalEmbedder.embedText("A photo of a monument");
const imageBytes = new Uint8Array(await (await fetch("monument.jpg")).arrayBuffer());
const imageResult = await universalEmbedder.embedImage(imageBytes);
const similarity = UniversalEmbedder.cosineSimilarity(
  textResult.embeddings[0],
  imageResult.embeddings[0]
);

Android (Kotlin)

Gradleimplementation("com.google.mediapipe:tasks-retrieval:latest.release")Sample code·Setup guide

import com.google.mediapipe.tasks.core.BaseOptions
import com.google.mediapipe.tasks.core.Delegate
import com.google.mediapipe.tasks.retrieval.universalembedder.UniversalEmbedder
import com.google.mediapipe.tasks.retrieval.universalembedder.UniversalEmbedderOptions

// 1. Create the task
val baseOptions = BaseOptions.builder()
    .setModelAssetPath("embeddinggemma-2-text-vision-440m.litertlm")
    .build()
val options = UniversalEmbedderOptions.builder()
    .setBaseOptions(baseOptions)
    .setTextDelegate(Delegate.GPU)
    .setVisionDelegate(Delegate.GPU)
    .setL2Normalize(true)
    .build()

val universalEmbedder = UniversalEmbedder.createFromOptions(context, options)

// 2. Run inference across modalities and compare cosine similarity
val textResult = universalEmbedder.embedText("A photo of a monument")
val imageResult = universalEmbedder.embedImage(mpImage)
val similarity = UniversalEmbedder.cosineSimilarity(
    textResult.embeddings().first(),
    imageResult.embeddings().first()
)

Android (Java)

Gradleimplementation 'com.google.mediapipe:tasks-retrieval:latest.release'Sample code·Setup guide

import com.google.mediapipe.tasks.components.containers.EmbeddingResult;
import com.google.mediapipe.tasks.core.BaseOptions;
import com.google.mediapipe.tasks.core.Delegate;
import com.google.mediapipe.tasks.retrieval.universalembedder.UniversalEmbedder;
import com.google.mediapipe.tasks.retrieval.universalembedder.UniversalEmbedderOptions;

// 1. Create the task
BaseOptions baseOptions = BaseOptions.builder()
    .setModelAssetPath("embeddinggemma-2-text-vision-440m.litertlm")
    .build();
UniversalEmbedderOptions options = UniversalEmbedderOptions.builder()
    .setBaseOptions(baseOptions)
    .setTextDelegate(Delegate.GPU)
    .setVisionDelegate(Delegate.GPU)
    .setL2Normalize(true)
    .build();

UniversalEmbedder universalEmbedder = UniversalEmbedder.createFromOptions(context, options);

// 2. Run inference across modalities and compare cosine similarity
EmbeddingResult textResult = universalEmbedder.embedText("A photo of a monument");
EmbeddingResult imageResult = universalEmbedder.embedImage(mpImage);
double similarity = UniversalEmbedder.cosineSimilarity(
    textResult.embeddings().get(0),
    imageResult.embeddings().get(0)
);

iOS

SwiftPMhttps://github.com/google-ai-edge/mediapipe → MediaPipeTasksRetrievalPackage.swift·Sample code·Setup guide

import Foundation
import MediaPipeTasksRetrieval

// 1. Create the task
guard let modelPath = Bundle.main.path(
    forResource: "embeddinggemma-2-text-vision-440m",
    ofType: "litertlm"
) else { return }

let options = UniversalEmbedderOptions()
options.baseOptions.modelAssetPath = modelPath
options.l2Normalize = true

let universalEmbedder = try UniversalEmbedder(options: options)

// 2. Run inference across modalities and compare cosine similarity
let textResult = try universalEmbedder.embed(text: "A photo of a monument")
let imageResult = try universalEmbedder.embed(image: mpImage)
let similarity = try UniversalEmbedder.cosineSimilarity(
    embedding1: textResult.embeddings[0],
    embedding2: imageResult.embeddings[0]
)

Python

pippip install mediapipeSample code·Setup guide

import mediapipe as mp

BaseOptions = mp.tasks.BaseOptions
UniversalEmbedder = mp.tasks.retrieval.UniversalEmbedder
UniversalEmbedderOptions = mp.tasks.retrieval.UniversalEmbedderOptions

# 1. Create the task
options = UniversalEmbedderOptions(
    base_options=BaseOptions(model_asset_path="embeddinggemma-2-text-vision-440m.litertlm"),
    l2_normalize=True,
)

# 2. Run inference across modalities and compare cosine similarity
with UniversalEmbedder.create_from_options(options) as universal_embedder:
    text_result = universal_embedder.embed_text("A photo of a monument")
    image_result = universal_embedder.embed_image(mp.Image.create_from_file("monument.jpg"))
    similarity = UniversalEmbedder.cosine_similarity(
        text_result.embeddings[0], image_result.embeddings[0]
    )

Task details

This section describes the capabilities, inputs, outputs, and configuration options of this task.

Features

Unified multimodal embedding API - Extract embeddings from text strings, raw or compressed images, and audio data within a single task lifecycle.
On-device execution - Runs completely on-device on the LiteRT-LM runtime with a shared embedding engine for a small memory footprint.
Similarity comparison utilities - Built-in static utility functions to compute the cosine similarity between raw float arrays or embedding results.
Semantic memory bridge - Integrates directly with SemanticRetriever by exposing an EmbeddingProvider implementation.

Inputs

The Universal Embedder accepts inputs of various modalities, either individually or as a mixed list:

Text (String)
Image (MPImage / UIImage / Compressed bytes — see Prepare vision input data)
Audio (AudioData / Float samples — see Prepare audio input data)
Composite content (A mixed list of any of the preceding objects)

Outputs

The Universal Embedder outputs an EmbeddingResult containing:

Embedding: The high-dimensional feature vector, either as raw floating-point numbers or scalar-quantized.

Configuration options

This task has the following configuration options:

Web

Option Name	Description	Value Range	Default Value
`baseOptions.modelAssetPath`	Path, URL, or buffer for the `.litertlm` multimodal embedding model.	`String`	Required
`baseOptions.delegate`	Default hardware acceleration delegate for model execution.	{`CPU`, `GPU`}	`CPU`
`l2Normalize`	Whether to normalize the returned embedding feature vector using L2 normalization.	`Boolean`	`true`
`textDelegate / visionDelegate / audioDelegate`	Optional modality-specific hardware acceleration delegates (Android and iOS). When unset, defaults to `baseOptions.delegate`.	{`CPU`, `GPU`}	`baseOptions.delegate`
`activationDataType`	Precision type used for activations during inference.	{`FLOAT32`, `FLOAT16`, `INT16`, `INT8`}	Model default
`cacheDir`	Writable directory for storing compiled model cache artifacts.	`String`	Not set
`maxInputLength`	Maximum input signature sequence length to load from the multisignature model.	`Integer`	Not set
`visionTokensPerImage`	Number of vision soft tokens to generate per image.	`Integer`	Not set

Android

Option Name	Description	Value Range	Default Value
`baseOptions.modelAssetPath`	Path, URL, or buffer for the `.litertlm` multimodal embedding model.	`String`	Required
`baseOptions.delegate`	Default hardware acceleration delegate for model execution.	{`CPU`, `GPU`}	`CPU`
`l2Normalize`	Whether to normalize the returned embedding feature vector using L2 normalization.	`Boolean`	`true`
`textDelegate / visionDelegate / audioDelegate`	Optional modality-specific hardware acceleration delegates (Android and iOS). When unset, defaults to `baseOptions.delegate`.	{`CPU`, `GPU`}	`baseOptions.delegate`
`activationDataType`	Precision type used for activations during inference.	{`FLOAT32`, `FLOAT16`, `INT16`, `INT8`}	Model default
`cacheDir`	Writable directory for storing compiled model cache artifacts.	`String`	Not set
`maxInputLength`	Maximum input signature sequence length to load from the multisignature model.	`Integer`	Not set
`visionTokensPerImage`	Number of vision soft tokens to generate per image.	`Integer`	Not set

iOS

Option Name	Description	Value Range	Default Value
`baseOptions.modelAssetPath`	Path, URL, or buffer for the `.litertlm` multimodal embedding model.	`String`	Required
`baseOptions.delegate`	Default hardware acceleration delegate for model execution.	{`CPU`, `GPU`}	`CPU`
`l2Normalize`	Whether to normalize the returned embedding feature vector using L2 normalization.	`Boolean`	`true`
`textDelegate / visionDelegate / audioDelegate`	Optional modality-specific hardware acceleration delegates (Android and iOS). When unset, defaults to `baseOptions.delegate`.	{`CPU`, `GPU`}	`baseOptions.delegate`
`activationDataType`	Precision type used for activations during inference.	{`FLOAT32`, `FLOAT16`, `INT16`, `INT8`}	Model default
`cacheDirectory`	Writable directory for storing compiled model cache artifacts.	`String`	Not set
`maxInputLength`	Maximum input signature sequence length to load from the multisignature model.	`Integer`	Not set
`visionTokensPerImage`	Number of vision soft tokens to generate per image.	`Integer`	Not set

Python

Option Name	Description	Value Range	Default Value
`base_options.model_asset_path`	Path, URL, or buffer for the `.litertlm` multimodal embedding model.	`String`	Required
`base_options.delegate`	Default hardware acceleration delegate for model execution.	{`CPU`, `GPU`}	`CPU`
`l2_normalize`	Whether to normalize the returned embedding feature vector using L2 normalization.	`Boolean`	`True`
`text_delegate / vision_delegate / audio_delegate`	Optional modality-specific hardware acceleration delegates (Android and iOS). When unset, defaults to `baseOptions.delegate`.	{`CPU`, `GPU`}	`baseOptions.delegate`
`activation_data_type`	Precision type used for activations during inference.	{`FLOAT32`, `FLOAT16`, `INT16`, `INT8`}	Model default
`cache_dir`	Writable directory for storing compiled model cache artifacts.	`String`	Not set
`max_input_length`	Maximum input signature sequence length to load from the multisignature model.	`Integer`	Not set
`vision_tokens_per_image`	Number of vision soft tokens to generate per image.	`Integer`	Not set

Task output

Each UniversalEmbedderResult (EmbeddingResult) returns the projected feature vector in the shared multimodal embedding space, where text and image vectors can be compared directly via cosineSimilarity:

Field	Type / Range	Description
`floatEmbedding`	`Float32Array` / `float[]`	768-dimensional L2-normalized float vector in the shared text-and-image space (`embedding` in Python).
`headIndex`	`int`	Embedding head index (`0`).

Example `UniversalEmbedderResult` payload

The following example shows a 768-dimensional L2-normalized vector returned by embedText or embedImage:

{
  embeddings: [
    {
      headIndex: 0,
      floatEmbedding: [-0.0341, 0.0518, 0.0192, -0.0284 /* ... 768 L2-normalized floats */]
    }
  ]
}

Models

The Universal Embedder supports the EmbeddingGemma V2 model, which is available under the Gemma license.

EmbeddingGemma V2

EmbeddingGemma V2 is a multimodal embedding model capable of projecting text, images, and audio into a single shared vector space. This unified representation is what allows for accurate cross-modal capabilities, such as semantic image search based on natural language concepts.

You can download the .litertlm model bundles for each modality from the LiteRT community on Hugging Face:

Modality	Parameters	Model
Text	270M	`embeddinggemma-2-text-270m.litertlm`
Text / Vision	440M	`embeddinggemma-2-text-vision-440m.litertlm`
Text / Vision / Audio	740M	`embeddinggemma-2-740m.litertlm`

Universal Embedder Stay organized with collections Save and categorize content based on your preferences.

Quickstart

Web

Android (Kotlin)

Android (Java)

iOS

Python

Task details

Features

Configuration options

Web

Android

iOS

Python

Task output

Example UniversalEmbedderResult payload

Models

EmbeddingGemma V2

Universal Embedder

Example `UniversalEmbedderResult` payload