LiteRT-LM is the **production-ready** orchestration layer to run LLMs with
LiteRT, engineered for **high-performance** , **cross-platform** execution.

- **Cross-Platform Support:** Run on Android, iOS, Web, Desktop, and IoT (e.g. Raspberry Pi).
- **Hardware Acceleration:** Get peak performance and system stability by leveraging GPU and NPU accelerators across diverse hardware.
- **Multi-Modality:** Build with LLMs and embedding models that have vision and audio support.
- **Multimodal Embeddings (New):** Generate on-device vector representations across text, images, video, and audio for semantic search and RAG. See [Embedding Models](https://developers.google.com/edge/litert-lm/embedding_models).
- **Tool Use:** Function calling support for agentic workflows with constrained decoding for improved accuracy.
- **Broad Model Support:** Run Gemma, EmbeddingGemma 2, Llama, Phi-4, Qwen and more.

## What's New

For the latest release details and updates, refer to the
[GitHub Release Notes](https://github.com/google-ai-edge/LiteRT-LM/releases).

### On-Device GenAI Showcase

![Google AI Edge Gallery Screenshot](https://developers.google.com/edge/litert-lm/images/gallery_icon.jpg)

The Google AI Edge Gallery is an experimental app designed to showcase on-device
Generative AI capabilities running entirely offline using LiteRT-LM.

- **[Google Play](https://play.google.com/store/apps/details?id=com.google.ai.edge.gallery)**: Use LLMs locally on supported Android devices.
- **[App Store](https://apps.apple.com/us/app/google-ai-edge-gallery/id6749645337)**: Experience on-device AI on your iOS device.
- **[GitHub Source](https://github.com/google-ai-edge/gallery)**: View the source code for the gallery app to learn how to integrate LiteRT-LM inside your own projects.

## Featured Model: EmbeddingGemma 2

- Model Size: 485 MB (740M Omnimodal), 388 MB (440M Text-Vision), 165 MB (270M Text)
- Try it in your browser with the [Multimodal Search Web Demo](https://google-ai-edge.github.io/LiteRT-LM/web_demos/embedding_search/index.html).
- Additional technical details and integration examples are in the
  [Embedding Models Guide](https://developers.google.com/edge/litert-lm/embedding_models) and the
  [HuggingFace model card](https://huggingface.co/litert-community/embeddinggemma-2-740m-litert-lm).

  | Platform (Device) | Backend | Text Latency (ms) | Text + Vision Latency (ms) | Text + Audio Latency (ms) | Text + Vision + Audio Latency (ms) |
  |---|---|---|---|---|---|
  | Android (Pixel 11 Pro) | TPU | 8.3 | 49 | 193.5 | 149 |
  | Android (S26 Ultra) | CPU | 27.1 | 175 | 305 | 322 |
  | Android (S26 Ultra) | GPU | 25.9 | 119 | 350 | 324 |
  | iOS (iPhone 18 Pro) | CPU | 41.8 | 191 | 445 | 400 |
  | iOS (iPhone 18 Pro) | GPU | 11.6 | 69.8 | 165 | 150 |
  | Linux (Arm 2.3 \& 2.8 GHz, NVIDIA GeForce RTX 4090) | CPU | 105 | 825 | 947 | 1251 |
  | Linux (Arm 2.3 \& 2.8 GHz, NVIDIA GeForce RTX 4090) | GPU | 7.6 | 23.9 | 169 | 112 |
  | macOS (MacBook Pro M5) | CPU | 31.3 | 151 | 314 | 432 |
  | macOS (MacBook Pro M5) | GPU | 9.5 | 37.3 | 195 | 131 |
  | Windows (Intel Core Ultra Series 3) | CPU | 71.7 | 338 | 813 | 708 |
  | Windows (Intel Core Ultra Series 3) | GPU | 19.2 | 62.5 | 284 | 205 |
  | Windows (Intel Core Ultra Series 3) | NPU | 13.3 | 49.8 | 124.3 | 108.6 |
  | Web (MacBook Pro M5) | WebGPU | 21.8 | 107 | 228 | 226 |
  | IoT (Raspberry Pi 5 16GB) | CPU | 161 | 1761 | 946 | 2304 |

## Featured Model: Gemma-4-E2B

- Model Size: 2.58 GB
- Additional technical details are in the
  [HuggingFace model card](https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm)

  | Platform (Device) | Backend | Prefill (tk/s) | Decode (tk/s) | Time to First Token (seconds) | Peak CPU Memory (MB) |
  |---|---|---|---|---|---|
  | Android (S26 Ultra) | CPU | 557 | 47 | 1.8 | 1733 |
  | Android (S26 Ultra) | GPU | 3808 | 52 | 0.3 | 676 |
  | iOS (iPhone 17 Pro) | CPU | 532 | 25 | 1.9 | 607 |
  | iOS (iPhone 17 Pro) | GPU | 2878 | 56 | 0.3 | 1450 |
  | Linux (Arm 2.3 \& 2.8 GHz, NVIDIA GeForce RTX 4090) | CPU | 260 | 35 | 4 | 1628 |
  | Linux (Arm 2.3 \& 2.8 GHz, NVIDIA GeForce RTX 4090) | GPU | 11234 | 143 | 0.1 | 913 |
  | macOS (MacBook Pro M4 Max) | CPU | 901 | 42 | 1.1 | 736 |
  | macOS (MacBook Pro M4 Max) | GPU | 7835 | 160 | 0.1 | 1623 |
  | Windows (Intel LunarLake) | CPU | 435 | 30 | 2.4 | 3505 |
  | Windows (Intel LunarLake) | GPU | 3751 | 48 | 0.3 | 3540 |
  | IoT (Raspberry Pi 5 16GB) | CPU | 133 | 8 | 7.8 | 1546 |

## Start Building

LiteRT-LM provides APIs for several programming languages and platforms to help
you build on-device AI applications quickly. Select a guide below to get
started:

| Language | Status | Best For... | Documentation |
|---|---|---|---|
| **CLI** | ✅ Stable | Getting started with LiteRT-LM in less than 1 min. | [CLI Guide](https://developers.google.com/edge/litert-lm/cli) |
| **Python** | ✅ Stable | Rapid prototyping, development, on desktop, Android \& Raspberry Pi. | [Python Guide](https://developers.google.com/edge/litert-lm/python) |
| **Kotlin** | ✅ Stable | Native Android apps and JVM-based desktop tools. Optimized for Coroutines. | [Kotlin Guide](https://developers.google.com/edge/litert-lm/android) |
| **Swift** | 🚀 Early Preview | Native iOS and macOS integration with specialized Metal support. | [Swift Guide](https://developers.google.com/edge/litert-lm/swift) |
| **JavaScript (web)** | 🚀 Early Preview | Deploy models directly in web browsers with high performance. | [JavaScript Guide](https://developers.google.com/edge/litert-lm/js) |
| **Flutter** | 🚀 Community | Cross-platform Flutter apps using community `flutter_gemma`. | [Flutter Guide](https://developers.google.com/edge/litert-lm/flutter) |
| **C++** | ✅ Stable | High-performance, cross-platform core logic and embedded systems. | [C++ Guide](https://developers.google.com/edge/litert-lm/cpp) |

### Build from Source

If you want to customize LiteRT-LM or build it for a specific hardware
configuration, you can compile it directly from the source code. For
step-by-step instructions on how to set up your environment and build the
framework, refer to the
[LiteRT-LM Build and Run Guide](https://github.com/google-ai-edge/LiteRT-LM/blob/main/docs/getting-started/build-and-run.md)
on GitHub.

### Supported Backends \& Platforms

| Acceleration | Android | iOS | macOS | Windows | Linux | IoT |
|---|---|---|---|---|---|---|
| **CPU** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **GPU** | ✅ | ✅ | ✅ | ✅ | ✅ | - |
| **NPU** | ✅ | - | - | 🚀 | - | - |

### Supported Models

The following table lists a subset of models supported by LiteRT-LM.
For the latest models and performance data, visit the
[LiteRT Community on Hugging Face](https://huggingface.co/litert-community).

| Model | Type | Size (MB) | Details | Device | CPU Prefill (tk/s) | CPU Decode (tk/s) | GPU Prefill (tk/s) | GPU Decode (tk/s) |
|---|---|---|---|---|---|---|---|---|
| **Gemma4-E2B** | Chat | 2583 | [Model Card](https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm) | Samsung S26 Ultra | 557 | 47 | 3808 | 52 |
|   |   |   |   | iPhone 17 Pro | 532 | 25 | 2878 | 57 |
|   |   |   |   | MacBook Pro M4 Max | 901 | 42 | 7835 | 160 |
| **Gemma4-E4B** | Chat | 3654 | [Model Card](https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm) | Samsung S26 Ultra | 195 | 18 | 1293 | 22 |
|   |   |   |   | iPhone 17 Pro | 159 | 10 | 1189 | 25 |
|   |   |   |   | MacBook Pro M4 Max | 277 | 27 | 2560 | 101 |
| **Gemma-3n-E2B** | Chat | 2965 | [Model Card](https://huggingface.co/google/gemma-3n-E2B-it-litert-lm) | MacBook Pro M3 | 233 | 28 | - | - |
|   |   |   |   | Samsung S24 Ultra | 111 | 16 | 816 | 16 |
| **Gemma-3n-E4B** | Chat | 4235 | [Model Card](https://huggingface.co/google/gemma-3n-E4B-it-litert-lm) | MacBook Pro M3 | 170 | 20 | - | - |
|   |   |   |   | Samsung S24 Ultra | 74 | 9 | 548 | 9 |
| **Gemma3-1B** | Chat | 1005 | [Model Card](https://huggingface.co/litert-community/Gemma3-1B-IT) | Samsung S24 Ultra | 177 | 33 | 1191 | 24 |
| **FunctionGemma** | Base | 289 | [Model Card](https://huggingface.co/litert-community/functiongemma-270m-ft-mobile-actions) | Samsung S25 Ultra | 2238 | 154 | - | - |
| **phi-4-mini** | Chat | 3906 | [Model Card](https://huggingface.co/litert-community/Phi-4-mini-instruct) | Samsung S24 Ultra | 67 | 7 | 314 | 10 |
| **Qwen2.5-1.5B** | Chat | 1598 | [Model Card](https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct) | Samsung S25 Ultra | 298 | 34 | 1668 | 31 |
| **Qwen3-0.6B** | Chat | 586 | [Model Card](https://huggingface.co/litert-community/Qwen3-0.6B) | Vivo X300 Pro | 165 | 9 | 580 | 21 |
| **Qwen2.5-0.5B** | Chat | 521 | [Model Card](https://huggingface.co/litert-community/Qwen2.5-0.5B-Instruct) | Samsung S24 Ultra | 251 | 30 | - | - |

#### Embedding Models

| Model | Modalities | Size (MB) | Details |
|---|---|---|---|
| **EmbeddingGemma 2 (740M)** | Text, Images, Video, Audio | 485 | [Model Card](https://huggingface.co/litert-community/embeddinggemma-2-740m-litert-lm) |
| **EmbeddingGemma 2 Text-Vision (440M)** | Text, Images | 388 | [Model Card](https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm) |
| **EmbeddingGemma 2 Text (270M)** | Text | 165 | [Model Card](https://huggingface.co/litert-community/embeddinggemma-2-text-270m-litert-lm) |

## Report Issues

If you encounter a bug or have a feature request, report at
[LiteRT-LM GitHub Issues](https://github.com/google-ai-edge/LiteRT-LM/issues/).