Semantic search

This guide walks you through how to implement on-device semantic search, a search technique that finds relevant content based on query meaning, rather than purely by keywords. Semantic search removes the need to manually build and maintain custom synonym tables or tagging taxonomies. To implement semantic search on-device, use the following libraries:

  • ML Kit SDK, specifically the Embedding API, to generate vector embeddings
  • AndroidX AppSearch to store the vector embeddings, and search structured data using cosine distance metrics to find content by their semantics

Prerequisites

To use on-device semantic search, you must meet the following device and account requirements:

  • Your device supports Android AICore. Most AICore-supported devices use NPU-accelerated models, while the remaining supported devices fall back to a CPU model (with XNNPACK acceleration).
  • The Google account on your device is part of the AICore Developer Preview program.
  • Your device has the latest version of the AICore Developer Preview APK: On your device, open the Android AICore Play Store page and tap Update to install the latest version of the APK. Alternatively go to your device Settings, search for System services or Google services, tap Android AICore (Beta), and tap Update if an update is available.
  • Your device has the latest version of the Private Compute Services app: On your device, open the Private Compute Services Play Store page and tap Update to install the latest version of the app. Alternatively go to Play Store > Manage apps & device > Manage > Private Compute Services and tap Update if an update is available.

Supported models and configuration

The ML Kit Embedding API uses Google's EmbeddingGemma model family, managed by Android AICore. Specifically, the Embedding API uses the EmbeddingGemma 2 (embedding_gemma_v2) for generating embeddings for text and image inputs.

Here are the input specifications and limits:

Input property EmbeddingGemma 2 Usage notes
Supported modalities Text and image (Modality.TEXT, Modality.IMAGE) To confirm that image input is supported, check at runtime that model.getMetadata()?.supportedModalities.contains(Modality.IMAGE) returns true.
Token limit 256 tokens Inspect using the model.getTokenLimit() method. Each image is encoded into 70 tokens, leaving 186 tokens for accompanying text.
Image resolution Up to 768 px on the shorter side The constructor for the ImagePart class accepts Bitmap, Uri, or ByteArray argument types. Images with a shorter dimension greater than 768 px are automatically downscaled by the SDK while preserving aspect ratio. For optimal performance and memory usage, avoid extreme aspect ratios.

The output vector dimension is 768 (a FloatArray object of 32-bit floats). You can inspect the output vector dimension at runtime using the model.getMetadata()?.dimension method.

Get started

To configure your project to use semantic search, add the following dependencies to your module-level build.gradle.kts (or build.gradle) file.

Kotlin

dependencies {
    // ...
    implementation("com.google.mlkit:genai-embedding:1.0.0-alpha1")
    implementation("androidx.appsearch:appsearch:1.2.0-alpha01")
    implementation("androidx.appsearch:appsearch-local-storage:1.2.0-alpha01")
    implementation("androidx.concurrent:concurrent-futures-ktx:1.3.0")
    // ...
}

Groovy

dependencies {
    // ...
    implementation 'com.google.mlkit:genai-embedding:1.0.0-alpha1'
    implementation 'androidx.appsearch:appsearch:1.2.0-alpha01'
    implementation 'androidx.appsearch:appsearch-local-storage:1.2.0-alpha01'
    implementation 'androidx.concurrent:concurrent-futures-ktx:1.3.0'
    // ...
}

To retrieve the available models, use the EmbeddingGeneration.getEmbeddingModels(EmbeddingOptions()) method.

Define an AppSearch schema

To enable semantic similarity searching, you must define an AppSearch schema where the vector property is explicitly configured for similarity indexing.

// Define the document schema
val documentSchema = AppSearchSchema.Builder("my_schema_type")
    .addProperty(
        AppSearchSchema.StringPropertyConfig.Builder("text")
            .setCardinality(AppSearchSchema.PropertyConfig.CARDINALITY_OPTIONAL)
            .setIndexingType(AppSearchSchema.StringPropertyConfig.INDEXING_TYPE_PREFIXES)
            .setTokenizerType(AppSearchSchema.StringPropertyConfig.TOKENIZER_TYPE_PLAIN)
            .build()
    )
    .addProperty(
        AppSearchSchema.StringPropertyConfig.Builder("imageUri")
            .setCardinality(AppSearchSchema.PropertyConfig.CARDINALITY_OPTIONAL)
            .setIndexingType(AppSearchSchema.StringPropertyConfig.INDEXING_TYPE_NONE)
            .setTokenizerType(AppSearchSchema.StringPropertyConfig.TOKENIZER_TYPE_NONE)
            .build()
    )
    .addProperty(
        AppSearchSchema.EmbeddingPropertyConfig.Builder("embedding")
            .setCardinality(AppSearchSchema.PropertyConfig.CARDINALITY_OPTIONAL)
            .setIndexingType(
                AppSearchSchema.EmbeddingPropertyConfig.INDEXING_TYPE_SIMILARITY)
            .setQuantizationType(
                AppSearchSchema.EmbeddingPropertyConfig.QUANTIZATION_TYPE_8_BIT)
            .build()
    )
    .build()

Index document content

To index document content, you must generate a vector embedding for each Content object, wrap it in a GenericDocument instance, and save it to your AppSearch database.

When constructing an EmbeddingVector object, you must also pass the active modelSignature value. AppSearch uses this modelSignature string to prevent incompatible search queries.

suspend fun syncDatabase(documents: List<Pair<String, Uri?>>) {

    val modelSignature = model?.getModelSignature() ?: return
    val docsWithEmbedding = mutableListOf<GenericDocument>()

    for ((index, item) in documents.withIndex()) {
        val (content, imageUri) = item
        // 1. Generate embedding (text and/or image)
        val parts = mutableListOf<Part>()
        if (content.isNotBlank()) {
            parts.add(TextPart(content))
        }
        var imagePart: ImagePart? = null
        val vector = try {
            if (imageUri != null && isImageSupported) {
                imagePart = ImagePart(imageUri)
                parts.add(imagePart)
            }
            if (parts.isEmpty()) continue
            val request = EmbeddingRequest(
                Content(parts)
            )
            model?.generateEmbedding(request)?.embedding?.vector
                ?: continue
        } finally {
            imagePart?.close()
        }

        // 2. Build the AppSearch document
        val docBuilder = GenericDocument.Builder<GenericDocument.Builder<*>>(
            "my_namespace",
            "doc_$index",
            "my_schema_type"
        )
        .setPropertyString("text", content)
        // Associate the vector with the signature that generated it
        .setPropertyEmbedding("embedding", EmbeddingVector(vector, modelSignature))

        if (imageUri != null) {
            docBuilder.setPropertyString("imageUri", imageUri.toString())
        }

        docsWithEmbedding.add(docBuilder.build())
    }

    // 3. Clear the old database by force-overriding the schema
    val clearRequest = SetSchemaRequest.Builder().setForceOverride(true).build()
    session?.setSchemaAsync(clearRequest)?.await()

    val schemaRequest = SetSchemaRequest.Builder().addSchemas(documentSchema).build()
    session?.setSchemaAsync(schemaRequest)?.await()

    // 4. Index documents
    val putRequest = PutDocumentsRequest.Builder().addGenericDocuments(docsWithEmbedding).build()
    session?.putAsync(putRequest)?.await()
}

To perform a semantic query, convert the search query (text and/or image) into an embedding vector, configure the vector-search parameters, and query your AppSearch database.

Recommendations:

  • Set the default metric to the SearchSpec.EMBEDDING_SEARCH_METRIC_TYPE_COSINE constant.
  • Rank results by the "sum(this.matchedSemanticScores(getEmbeddingParameter(0)))" formula.
  • Use the logical "semanticSearch(getEmbeddingParameter(0), $threshold)" string to filter results where the similarity meets your threshold value (typically 0.5 or higher).
suspend fun search(
    queryText: String,
    imageUri: Uri? = null,
    threshold: Double = 0.5
): List<Triple<String, Uri?, Double>> {
    val modelSignature = model?.getModelSignature() ?: return emptyList()

    // 1. Generate query embedding (text and/or image)
    val parts = mutableListOf<Part>()
    if (queryText.isNotBlank()) {
        parts.add(TextPart(queryText))
    }
    var imagePart: ImagePart? = null
    val queryVector = try {
        if (imageUri != null && isImageSupported) {
            imagePart = ImagePart(imageUri)
            parts.add(imagePart)
        }
        if (parts.isEmpty()) return emptyList()
        val request = EmbeddingRequest(
            content = Content(parts)
        )
        model?.generateEmbedding(request)?.embedding?.vector
            ?: return emptyList()
    } finally {
        imagePart?.close()
    }

    // 2. Configure search specifications
    val searchSpec = SearchSpec.Builder()
        .addFilterSchemas("my_schema_type")
        .setDefaultEmbeddingSearchMetricType(SearchSpec.EMBEDDING_SEARCH_METRIC_TYPE_COSINE)
        .setRankingStrategy("sum(this.matchedSemanticScores(getEmbeddingParameter(0)))")
        .setListFilterQueryLanguageEnabled(true)
        .setResultCountPerPage(5)
        // Add the query vector as parameter 0
        .addEmbeddingParameters(EmbeddingVector(queryVector, modelSignature))
        .build()

    // 3. Execute search
    val queryExpression = "semanticSearch(getEmbeddingParameter(0), $threshold)"
    val results = session?.search(queryExpression, searchSpec) ?: return emptyList()

    return try {
        val page = results.nextPageAsync.await()
        page.map {
            val text = it.genericDocument.getPropertyString("text") ?: ""
            val matchedImageUri = it.genericDocument.getPropertyString("imageUri")?.let(Uri::parse)
            // rankingSignal contains the computed similarity score
            Triple(text, matchedImageUri, it.rankingSignal)
        }
    } finally {
        results.close()
    }
}

Use a hybrid search index

For the best user experience, we recommend using a hybrid strategy involving both semantic and keyword search. When the user is actively typing in a search bar, use keyword search to show instant type-ahead suggestions; when they enter the completed query, or even at word breaks, use semantic search to read, understand, and fetch the best conceptual matches.

Reindex your vector database

Embeddings generated by different model versions reside in incompatible vector spaces and can't be compared directly. You must reindex your AppSearch database whenever you update the embedding model.

We recommend to start reindexing the embeddings as soon as the model you're using enters its deprecation period. You can use the Embedding API to check a model's deprecation schedule. Call the model.getMetadata() method to retrieve the EmbeddingMetadata object and check these lifecycle date properties (returned as LocalDate objects in UTC):

  • The deprecationDate value marks the beginning of the model deprecation period. You should start reindexing while the model continues to work and finish reindexing before its official shutdown.
  • The shutdownDate value is when the model will be decommissioned and no longer served. After this date, the model is inaccessible for inference and existing embeddings of this model are forced to reindex with the latest available model.

We also communicate upcoming model deprecations, migration timelines, and target shutdown dates through the ML Kit release notes and developer documentation updates.

Best practices

For a good user experience, follow these practices when integrating vector embeddings into your Android app:

  • Schedule indexing workloads: Always schedule background indexing workloads using the WorkManager library. Configure your background tasks to run only when the device is charging and idle. ML Kit imposes quota limits for model usage, similar to LLM APIs, with more liberal limits when the device is charging. In Developer Preview mode, you can bypass quota limitations for testing purposes.
  • Storage and query optimization: Optimize your database footprint and query latency by using quantization and dimensionality truncation:
    • Configure your AppSearchSchema schema to use the QUANTIZATION_TYPE_8_BIT constant, as shown at Define an AppSearch schema. This reduces memory usage with minimal impact on quality by shrinking the precision of each coordinate to an 8-bit integer.
    • Use Matryoshka Representation Learning (MRL), supported by the EmbeddingGemma 2 model, to truncate the embedding vector. To learn more, see EmbeddingGemma 2 on HuggingFace.
  • Signature consistency: Don't query a vector space using a mismatched signature. If you upgrade the embedding model version, you must wipe the AppSearch database and regenerate embeddings for all documents using the new model signature.
  • Safety and content moderation: You are responsible for implementing content moderation and safety filtering in your app. Unlike generative text or image APIs, the ML Kit Embedding API doesn't include built-in safety filters. Because the model outputs numerical vector representations (FloatArray coordinates) of caller-provided input rather than generating new text or media, numerical vectors don't inherently present generative content safety risks. Omitting on-device safety filtering also preserves low-latency indexing and search performance by avoiding CPU contention during high-throughput document indexing and real-time queries. To improve safety, follow these guidelines:
    • Moderate untrusted inputs and indexed content: If your app indexes or searches untrusted user-generated content or external data, apply content moderation and policy filters to the source text or images before indexing them or displaying search results.
    • Safeguard downstream generative AI pipelines: If retrieved documents or passages are passed as context to a downstream generative AI model, verify that safety filters and content policies are enabled on that downstream model and its user-facing output.

Additional resources