You can use ML Kit to recognize entities in an image and label them. This API supports a wide range of custom image classification models. Refer to [Custom models with ML Kit](https://developers.google.com/ml-kit/custom-models.md.txt) for guidance on model compatibility requirements, where to find pre-trained models, and how to train your own models.
There are two ways to integrate image labeling with custom models: by bundling the pipeline as part of your app, or by using an unbundled pipeline that depends on Google Play services. If you select the unbundled pipeline, your app will be smaller. See the following table for details. | | Bundled | Unbundled | |---|---|---| | **Library name** | `com.google.mlkit:image-labeling-custom` | `com.google.android.gms:play-services-mlkit-image-labeling-custom` | | **Implementation** | Pipeline is statically linked to your app at build time. | Pipeline is dynamically downloaded using Google Play services. | | **App size** | About 3.8 MB size increase. | About 200 KB size increase. | | **Initialization time** | Pipeline is available immediately. | Might have to wait for pipeline to download before first use. | | **API lifecycle stage** | General Availability (GA) | Beta | > [!NOTE] > **Note:** The unbundled version of image labeling with custom models is offered in beta, which means it might change in backward-incompatible ways and is not subject to any SLA or deprecation policy. There are two ways to integrate a custom model: bundle the model by putting it inside your app's asset folder, or dynamically download it from Firebase. The following table compares these two options. | Bundled Model | Hosted Model | |---|---| | The model is part of your app's APK, which increases its size. | The model is not part your APK. It is hosted by uploading to Cloud Storage. We recommend using [Cloud Storage for Firebase](https://firebase.google.com/docs/storage). | | The model is available immediately, even when the Android device is offline | Your app must include code to download the model on demand | | No need for a Firebase project | Requires a Firebase project (if using Cloud Storage for Firebase). | | You must republish your app to update the model | Push model updates without republishing your app | | No built-in A/B testing | A/B testing with [Firebase Remote Config](https://firebase.google.com/docs/remote-config) | ## Try it out - See the [vision quickstart app](https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart) for an example usage of the bundled model and the [automl quickstart app](https://github.com/googlesamples/mlkit/tree/master/android/automl) for an example usage of the hosted model. ## Before you begin 1. In your project-level `build.gradle.kts` file, make sure to include Google's Maven repository in both your `buildscript` and `allprojects` sections. 2. Add the dependencies for the ML Kit Android libraries to your module's app-level gradle file, which is usually `app/build.gradle.kts`. Choose one of the following dependencies based on your needs: **For bundling the pipeline with your app:** ```kotlin dependencies { // ... // Use this dependency to bundle the pipeline with your app implementation("com.google.mlkit:image-labeling-custom:17.0.3") } ``` **For using the pipeline in Google Play services:** ```kotlin dependencies { // ... // Use this dependency to use the dynamically downloaded pipeline in Google Play services implementation("com.google.android.gms:play-services-mlkit-image-labeling-custom:16.0.0-beta5") } ``` 3. **If you choose to use the pipeline in Google Play services** , you can configure your app to automatically download the pipeline to the device after your app is installed from the Play Store. To do so, add the following declaration to your app's `AndroidManifest.xml` file: ```xml ... ``` You can also explicitly check the pipeline availability and request download through Google Play services [ModuleInstallClient API](https://developers.google.com/android/guides/module-install-apis.md.txt). If you don't enable install-time pipeline downloads or request explicit download, the pipeline is downloaded the first time you run the labeler. Requests you make before the download has completed produce no results. 4. **If you want to download a model using Cloud Storage for Firebase** , make sure you [add Firebase to your Android project](https://firebase.google.com/docs/android/setup), if you have not already done so. This is not required when you bundle the model. ## 1. Load the model You can load the model from a locally-bundled source or a remotely-hosted source. ### Configure a local model source To bundle the model with your app: 1. Copy the model file (usually ending in `.tflite` or `.lite`) to your app's `assets/` folder. (You might need to create the folder first by right-clicking the `app/` folder, then clicking **New \> Folder \> Assets Folder**.) 2. Create `LocalModel` object, specifying the path to the model file: ### Kotlin ```kotlin val localModel = LocalModel.Builder() .setAssetFilePath("model.tflite") // or .setAbsoluteFilePath(absolute path to model file) // or .setUri(URI to model file) .build() ``` ### Java ```java LocalModel localModel = new LocalModel.Builder() .setAssetFilePath("model.tflite") // or .setAbsoluteFilePath(absolute path to model file) // or .setUri(URI to model file) .build(); ``` > [!NOTE] > **Note:** apps target Android 11 (API level 30) can no longer access files from external storage because of [Storage updates in Android 11](https://developer.android.com/about/versions/11/privacy/storage). > [!NOTE] > **Note:** if you have a model that was trained with [AutoML Vision Edge in > Firebase](https://firebase.google.com/docs/ml/train-image-labeler) (not Google Cloud), the preceding instructions might not work. Follow the [migration guide](https://developers.google.com/ml-kit/vision/image-labeling/automl/migrate-automl.md.txt) for instructions. ### Configure a remotely-hosted model source To use the remotely-hosted model, you must download the model file to the device's local storage using your own app logic, and then load it as a local model. We recommend using Cloud Storage for Firebase to host a model. For implementation details, see the [Firebase ML to Cloud Storage migration guide](https://firebase.google.com/docs/ml/migrate-to-cloud-storage). ### Configure the image labeler After you configure your model sources, create an `ImageLabeler` object from one of them. The following options are available: | Options || |---|---| | `confidenceThreshold` | Minimum confidence score of detected labels. If not set, any classifier threshold specified by the model's metadata will be used. If the model does not contain any metadata or the metadata does not specify a classifier threshold, a default threshold of 0.0 will be used. | | `maxResultCount` | Maximum number of labels to return. If not set, the default value of 10 will be used. | If you only have a locally-bundled model, just create a labeler from your `LocalModel` object: ### Kotlin ```kotlin val customImageLabelerOptions = CustomImageLabelerOptions.Builder(localModel) .setConfidenceThreshold(0.5f) .setMaxResultCount(5) .build() val labeler = ImageLabeling.getClient(customImageLabelerOptions) ``` ### Java ```java CustomImageLabelerOptions customImageLabelerOptions = new CustomImageLabelerOptions.Builder(localModel) .setConfidenceThreshold(0.5f) .setMaxResultCount(5) .build(); ImageLabeler labeler = ImageLabeling.getClient(customImageLabelerOptions); ``` If you have a remotely-hosted model, you will have to check that it has been downloaded before you run it. Although you only have to confirm this before running the labeler, if you have both a remotely-hosted model and a locally-bundled model, it might make sense to perform this check when instantiating the image labeler: create a labeler from the remote model if it's been downloaded, and from the local model otherwise. ### Kotlin ```kotlin val modelFile = File(context.cacheDir, "my_downloaded_model.tflite") val model = if (modelFile.exists()) { // Use the downloaded model if available LocalModel.Builder().setAbsoluteFilePath(modelFile.absolutePath).build() } else { // Fall back to the bundled model LocalModel.Builder().setAssetFilePath("model.tflite").build() } val options = CustomImageLabelerOptions.Builder(model) .setConfidenceThreshold(0.5f) .setMaxResultCount(5) .build() val labeler = ImageLabeling.getClient(options) ``` ### Java ```java File modelFile = new File(context.getCacheDir(), "my_downloaded_model.tflite"); LocalModel model; if (modelFile.exists()) { // Use the downloaded model if available model = new LocalModel.Builder().setAbsoluteFilePath(modelFile.getAbsolutePath()).build(); } else { // Fall back to the bundled model model = new LocalModel.Builder().setAssetFilePath("model.tflite").build(); } CustomImageLabelerOptions options = new CustomImageLabelerOptions.Builder(model) .setConfidenceThreshold(0.5f) .setMaxResultCount(5) .build(); ImageLabeler labeler = ImageLabeling.getClient(options); ``` If you only have a remotely-hosted model, you should disable model-related functionality---for example, grey-out or hide part of your UI---until you confirm the model has been downloaded. ### Kotlin ```kotlin val localFile = File(context.cacheDir, "my_remote_model.tflite") if (localFile.exists()) { initializeLabeler(localFile) } else { showLoadingUI() val storage = Firebase.storage val modelRef = storage.getReferenceFromUrl("gs://YOUR_BUCKET/path/to/model.tflite") modelRef.getFile(localFile) .addOnSuccessListener { hideLoadingUI() initializeLabeler(localFile) } .addOnFailureListener { showErrorUI() } } private fun initializeLabeler(modelFile: File) { val localModel = LocalModel.Builder().setAbsoluteFilePath(modelFile.absolutePath).build() val options = CustomImageLabelerOptions.Builder(localModel).build() val labeler = ImageLabeling.getClient(options) enableMLFeatures(labeler) } ``` ### Java ```java File localFile = new File(context.getCacheDir(), "my_remote_model.tflite"); if (localFile.exists()) { initializeLabeler(localFile); } else { showLoadingUI(); FirebaseStorage storage = FirebaseStorage.getInstance(); StorageReference modelRef = storage.getReferenceFromUrl("gs://YOUR_BUCKET/path/to/model.tflite"); modelRef.getFile(localFile) .addOnSuccessListener(new OnSuccessListener() { @Override public void onSuccess(FileDownloadTask.TaskSnapshot taskSnapshot) { hideLoadingUI(); initializeLabeler(localFile); } }) .addOnFailureListener(new OnFailureListener() { @Override public void onFailure(@NonNull Exception exception) { showErrorUI(); } }); } private void initializeLabeler(File modelFile) { LocalModel localModel = new LocalModel.Builder().setAbsoluteFilePath(modelFile.getAbsolutePath()).build(); CustomImageLabelerOptions options = new CustomImageLabelerOptions.Builder(localModel).build(); ImageLabeler labeler = ImageLabeling.getClient(options); enableMLFeatures(labeler); } ``` ## 2. Prepare the input image Then, for each image you want to label, create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from your image. The image labeler runs fastest when you use a `Bitmap` or, if you use the camera2 API, a YUV_420_888 `media.Image`, which are recommended when possible.
You can create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from different sources, each is explained below. ### Using a `media.Image` To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a `media.Image` object, such as when you capture an image from a device's camera, pass the `media.Image` object and the image's rotation to `InputImage.fromMediaImage()`. If you use the [CameraX](https://developer.android.com/training/camerax) library, the `OnImageCapturedListener` and `ImageAnalysis.Analyzer` classes calculate the rotation value for you. ### Kotlin ```kotlin private class YourImageAnalyzer : ImageAnalysis.Analyzer { override fun analyze(imageProxy: ImageProxy) { val mediaImage = imageProxy.image if (mediaImage != null) { val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees) // Pass image to an ML Kit Vision API // ... } } } ``` ### Java ```java private class YourAnalyzer implements ImageAnalysis.Analyzer { @Override public void analyze(ImageProxy imageProxy) { Image mediaImage = imageProxy.getImage(); if (mediaImage != null) { InputImage image = InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees()); // Pass image to an ML Kit Vision API // ... } } } ``` If you don't use a camera library that gives you the image's rotation degree, you can calculate it from the device's rotation degree and the orientation of camera sensor in the device: ### Kotlin ```kotlin private val ORIENTATIONS = SparseIntArray() init { ORIENTATIONS.append(Surface.ROTATION_0, 0) ORIENTATIONS.append(Surface.ROTATION_90, 90) ORIENTATIONS.append(Surface.ROTATION_180, 180) ORIENTATIONS.append(Surface.ROTATION_270, 270) } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) @Throws(CameraAccessException::class) private fun getRotationCompensation(cameraId: String, activity: Activity, isFrontFacing: Boolean): Int { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. val deviceRotation = activity.windowManager.defaultDisplay.rotation var rotationCompensation = ORIENTATIONS.get(deviceRotation) // Get the device's sensor orientation. val cameraManager = activity.getSystemService(CAMERA_SERVICE) as CameraManager val sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION)!! if (isFrontFacing) { rotationCompensation = (sensorOrientation + rotationCompensation) % 360 } else { // back-facing rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360 } return rotationCompensation } ``` ### Java ```java private static final SparseIntArray ORIENTATIONS = new SparseIntArray(); static { ORIENTATIONS.append(Surface.ROTATION_0, 0); ORIENTATIONS.append(Surface.ROTATION_90, 90); ORIENTATIONS.append(Surface.ROTATION_180, 180); ORIENTATIONS.append(Surface.ROTATION_270, 270); } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) private int getRotationCompensation(String cameraId, Activity activity, boolean isFrontFacing) throws CameraAccessException { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation(); int rotationCompensation = ORIENTATIONS.get(deviceRotation); // Get the device's sensor orientation. CameraManager cameraManager = (CameraManager) activity.getSystemService(CAMERA_SERVICE); int sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION); if (isFrontFacing) { rotationCompensation = (sensorOrientation + rotationCompensation) % 360; } else { // back-facing rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360; } return rotationCompensation; } ``` Then, pass the `media.Image` object and the rotation degree value to `InputImage.fromMediaImage()`: ### Kotlin ```kotlin val image = InputImage.fromMediaImage(mediaImage, rotation) ``` ### Java ```java InputImage image = InputImage.fromMediaImage(mediaImage, rotation); ``` ### Using a file URI To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a file URI, pass the app context and file URI to `InputImage.fromFilePath()`. This is useful when you use an `ACTION_GET_CONTENT` intent to prompt the user to select an image from their gallery app. ### Kotlin ```kotlin val image: InputImage try { image = InputImage.fromFilePath(context, uri) } catch (e: IOException) { e.printStackTrace() } ``` ### Java ```java InputImage image; try { image = InputImage.fromFilePath(context, uri); } catch (IOException e) { e.printStackTrace(); } ``` ### Using a `ByteBuffer` or `ByteArray` To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a `ByteBuffer` or a `ByteArray`, first calculate the image rotation degree as previously described for `media.Image` input. Then, create the `InputImage` object with the buffer or array, together with image's height, width, color encoding format, and rotation degree: ### Kotlin ```kotlin val image = InputImage.fromByteBuffer( byteBuffer, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ) // Or: val image = InputImage.fromByteArray( byteArray, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ) ``` ### Java ```java InputImage image = InputImage.fromByteBuffer(byteBuffer, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ); // Or: InputImage image = InputImage.fromByteArray( byteArray, /* image width */480, /* image height */360, rotation, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 ); ``` ### Using a `Bitmap` To create an [`InputImage`](https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage.md.txt) object from a `Bitmap` object, make the following declaration: ### Kotlin ```kotlin val image = InputImage.fromBitmap(bitmap, 0) ``` ### Java ```java InputImage image = InputImage.fromBitmap(bitmap, rotationDegree); ``` The image is represented by a `Bitmap` object together with rotation degrees. ## 3. Run the image labeler To label objects in an image, pass the `image` object to the `ImageLabeler`'s `process()` method. ### Kotlin ```kotlin labeler.process(image) .addOnSuccessListener { labels -> // Task completed successfully // ... } .addOnFailureListener { e -> // Task failed with an exception // ... } ``` ### Java ```java labeler.process(image) .addOnSuccessListener(new OnSuccessListener>() { @Override public void onSuccess(List labels) { // Task completed successfully // ... } }) .addOnFailureListener(new OnFailureListener() { @Override public void onFailure(@NonNull Exception e) { // Task failed with an exception // ... } }); ``` Note: If you are using the [`CameraX`](https://developer.android.com/jetpack/androidx/releases/camera) API, make sure to close the `ImageProxy` when finish using it, e.g., by adding an `OnCompleteListener` to the `Task` returned from the `process` method. See the [`VisionProcessorBase`](https://github.com/googlesamples/mlkit/blob/e949654bef09e97b237c7692349c52ee06e9fd9e/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/java/VisionProcessorBase.java#L178-L181) class in the quickstart sample app for an example. ## 4. Get information about labeled entities If the image labeling operation succeeds, a list of [`ImageLabel`](https://developers.google.com/android/reference/com/google/mlkit/vision/label/ImageLabel.md.txt) objects is passed to the success listener. Each `ImageLabel` object represents something that was labeled in the image. You can get each label's text description (if available in the metadata of the LiteRT model file), confidence score, and index. For example:
### Kotlin ```kotlin for (label in labels) { val text = label.text val confidence = label.confidence val index = label.index } ``` ### Java ```java for (ImageLabel label : labels) { String text = label.getText(); float confidence = label.getConfidence(); int index = label.getIndex(); } ``` > [!NOTE] > **Note:** if you use a LiteRT model that is incompatible with ML Kit, you will get an `MlKitException` with error code `MlKitException#INVALID_ARGUMENT` and some details about why it is not compatible. See [Custom Model Compatibility Requirements](https://developers.google.com/ml-kit/custom-models.md.txt#model_compatibility_requirements) for details. ## Tips to improve real-time performance If you want to label images in a real-time application, follow these guidelines to achieve the best frame rates: - If you use the [`Camera`](https://developer.android.com/reference/android/hardware/Camera) or [`camera2`](https://developer.android.com/reference/android/hardware/camera2/package-summary) API, throttle calls to the image labeler. If a new video frame becomes available while the image labeler is running, drop the frame. See the [`VisionProcessorBase`](https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/kotlin/VisionProcessorBase.kt) class in the quickstart sample app for an example. - If you use the [`CameraX`](https://developer.android.com/jetpack/androidx/releases/camera) API, be sure that backpressure strategy is set to its default value [`ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST`](https://developer.android.com/reference/androidx/camera/core/ImageAnalysis#STRATEGY_KEEP_ONLY_LATEST). This guarantees only one image will be delivered for analysis at a time. If more images are produced when the analyzer is busy, they will be dropped automatically and not queued for delivery. Once the image being analyzed is closed by calling ImageProxy.close(), the next latest image will be delivered. - If you use the output of the image labeler to overlay graphics on the input image, first get the result from ML Kit, then render the image and overlay in a single step. This renders to the display surface only once for each input frame. See the [`CameraSourcePreview`](https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/CameraSourcePreview.java) and [`GraphicOverlay`](https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/GraphicOverlay.java) classes in the quickstart sample app for an example. - If you use the Camera2 API, capture images in `ImageFormat.YUV_420_888` format. If you use the older Camera API, capture images in `ImageFormat.NV21` format.