Build and deploy a custom object detection model with TensorFlow Lite (Android)

1. Before you begin

In this codelab, you'll learn how to train a custom object detection model using a set of training images with TFLite Model Maker, then deploy your model to an Android app using TFLite Task Library. You will:

• Build an Android app that detects ingredients in images of meals.
• Integrate a TFLite pre-trained object detection model and see the limit of what the model can detect.
• Train a custom object detection model to detect the ingredients/components of a meal using a custom dataset called salad and TFLite Model Maker.
• Deploy the custom model to the Android app using TFLite Task Library.

In the end, you'll create something similar to the image below:

Prerequisites

This codelab has been designed for experienced mobile developers who want to gain experience with Machine Learning. You should be familiar with:

• Android development using Kotlin and Android Studio
• Basic Python syntax

What you'll learn

• How to train a custom object detection model using TFLite Model Maker.
• How to deploy a TFLite object detection model using TFLite Task Library.

What you'll need

• A recent version of Android Studio (v4.2+)
• Android Studio Emulator or a physical Android device
• The sample code
• Basic knowledge of Android development in Kotlin

2. Object Detection

Object detection is a set of computer vision tasks that can detect and locate objects in a digital image. Given an image or a video stream, an object detection model can identify which of a known set of objects might be present, and provide information about their positions within the image.

TensorFlow provides pre-trained, mobile optimized models that can detect common objects, such as cars, oranges, etc. You can integrate these pre-trained models in your mobile app with just a few lines of code. However, you may want or need to detect objects in more distinctive or offbeat categories. That requires collecting your own training images, then training and deploying your own object detection model.

If you just want to do general object detection without training your own model, check out this codelab to learn how to use the ML Kit Object Detection and Tracking API to build an object detector using just a few lines of code.

TensorFlow Lite

TensorFlow Lite is a cross-platform machine learning library that is optimized for running machine learning models on edge devices, including Android and iOS mobile devices.

TensorFlow Lite is actually the core engine used inside ML Kit to run machine learning models. There are two components in the TensorFlow Lite ecosystem that make it easy to train and deploy machine learning models on mobile devices:

• Model Maker is a Python library that makes it easy to train TensorFlow Lite models using your own data with just a few lines of code, no machine learning expertise required.
• Task Library is a cross-platform library that makes it easy to deploy TensorFlow Lite models with just a few lines of code in your mobile apps.

This codelab focuses on TFLite. Concepts and code blocks that are not relevant to TFLite and object detection are not explained and are provided for you to simply copy and paste.

3. Get set up

Download the Code

Click the following link to download all the code for this codelab:

Download source code

Unpack the downloaded zip file. This will unpack a root folder (odml-pathways-main) with all of the resources you will need. For this codelab, you will only need the sources in the object-detection/codelab2/android subdirectory.

The android subdirectory in the object-detection/codelab2/android repository contains two directories:

• starter—Starting code that you build upon for this codelab.
• final—Completed code for the finished sample app.

Import the starter app

Let's start by importing the starter app into the Android Studio.

1. Open Android Studio and select Import Project (Gradle, Eclipse ADT, etc.)
2. Open the starter folder from the source code you downloaded earlier.

To be sure that all dependencies are available to your app, you should sync your project with gradle files when the import process has finished.

3. Select Sync Project with Gradle Files ( ) from the Android Studio toolbar. Import starter/app/build.gradle

If this button is disabled, make sure you import only starter/app/build.gradle and not the entire repository.

Run the starter app

Now that you have imported the project into Android Studio, you're ready to run the app for the first time.

Connect your Android device via USB to your computer or start the Android Studio emulator, and click Run ( ) in the Android Studio toolbar.

4. Understand the starter app

In order to keep this codelab simple and focused on the machine learning bits, the starter app contains some boilerplate code that do a few things for you:

• It can take photos using the device's camera.
• It contains some stock images for you to try out object detection on an Android emulator.
• It has a convenient method to draw the object detection result on the input bitmap.

You'll mostly interact with these methods in the app skeleton:

• fun runObjectDetection(bitmap: Bitmap) This method is called when you choose a preset image or take a photo. bitmap is the input image for object detection. Later in this codelab, you will add object detection code to this method.
• data class DetectionResult(val boundingBoxes: Rect, val text: String) This is a data class that represents an object detection result for visualization. boundingBoxes is the rectangle where the object locates, and text is the detection result string to display together with the object's bounding box.
• fun drawDetectionResult(bitmap: Bitmap, detectionResults: List<DetectionResult>): Bitmap This method draws the object detection results in detectionResults on the input bitmap and returns the modified copy of it.

Here is an example of an output of the drawDetectionResult utility method.

5. Add on-device object detection

Now you'll build a prototype by integrating a pre-trained TFLite model that can detect common objects into the starter app.

Download a pre-trained TFLite object detection model

There are several object detector models on TensorFlow Hub that you can use. For this codelab, you'll download the EfficientDet-Lite Object detection model, trained on the COCO 2017 dataset, optimized for TFLite, and designed for performance on mobile CPU, GPU, and EdgeTPU.

Download the model

Next, use the TFLite Task Library to integrate the pre-trained TFLite model into your starter app. The TFLite Task Library makes it easy to integrate mobile-optimized machine learning models into a mobile app. It supports many popular machine learning use cases, including object detection, image classification, and text classification. You can load the TFLite model and run it with just a few lines of code.

TFLite Task Library only supports TFLite models that contain valid metadata. You can find more supported object detection models from this TensorFlow Hub collection.

Add the model to the starter app

1. Copy the model that you have just downloaded to the assets folder of the starter app. You can find the folder in the Project navigation panel in Android Studio.

2. Name the file model.tflite.

Update the Gradle file Task Library dependencies

Go to the app/build.gradle file and add this line into the dependencies configuration:

implementation 'org.tensorflow:tensorflow-lite-task-vision:0.3.1'

Sync your project with gradle files

To be sure that all dependencies are available to your app, you should sync your project with gradle files at this point. Select Sync Project with Gradle Files ( ) from the Android Studio toolbar.

(If this button is disabled, make sure you import only starter/app/build.gradle, not the entire repository.)

Set up and run on-device object detection on an image

There are only 3 simple steps with 3 APIs to load and run an object detection model:

• prepare an image / a stream: TensorImage
• create a detector object: ObjectDetector
• connect the 2 objects above: detect(image)

You achieve these inside the function runObjectDetection(bitmap: Bitmap)in file MainActivity.kt.

/**
* TFLite Object Detection Function
*/
private fun runObjectDetection(bitmap: Bitmap) {
    //TODO: Add object detection code here
}

Right now the function is empty. Move on to the following steps to implement the TFLite object detector. Along the way, Android Studio will prompt you to add the necessary imports:

• org.tensorflow.lite.support.image.TensorImage
• org.tensorflow.lite.task.vision.detector.ObjectDetector

Create Image Object

The images you'll use for this codelab are going to come from either the on-device camera, or preset images that you select on the app's UI. The input image is decoded into the Bitmap format and passed to the runObjectDetection method.

TFLite provides a simple API to create a TensorImage from Bitmap. Add the code below to the top of runObjectDetection(bitmap:Bitmap):

// Step 1: create TFLite's TensorImage object
val image = TensorImage.fromBitmap(bitmap)

Create a Detector instance

TFLite Task Library follows the Builder Design Pattern. You pass the configuration to a builder, then acquire a detector from it. There are several options to configure, including those to adjust the sensitivity of the object detector:

• max result (the maximum number of objects that the model should detect)
• score threshold (how confidence the object detector should be to return a detected object)
• label allowlist/denylist (allow/deny the objects in a predefined list)

Initialize the object detector instance by specifying the TFLite model file name and the configuration options:

// Step 2: Initialize the detector object
val options = ObjectDetector.ObjectDetectorOptions.builder()
    .setMaxResults(5)
    .setScoreThreshold(0.5f)
    .build()
val detector = ObjectDetector.createFromFileAndOptions(
    this, // the application context
    "model.tflite", // must be same as the filename in assets folder
    options
)

Feed Image(s) to the detector

Add the following code to fun runObjectDetection(bitmap:Bitmap). This will feed your images to the detector.

// Step 3: feed given image to the model and print the detection result
val results = detector.detect(image)

Upon completion, the detector returns a list of Detection, each containing information about an object that the model has found in the image. Each object is described with:

• boundingBox: the rectangle declaring the presence of an object and its location within the image
• categories: what kind of object it is and how confident the model is with the detection result. The model returns multiple categories, and the most confident one is first.
• label: the name of the object category.
• classificationConfidence:a float between 0.0 to 1.0, with 1.0 representing 100%

Print the detection results

Add the following code to fun runObjectDetection(bitmap:Bitmap). This calls a method to print the object detection results to Logcat.

// Step 4: Parse the detection result and show it
debugPrint(results)

Then add this debugPrint()method to the MainActivity class:

private fun debugPrint(results : List<Detection>) {
    for ((i, obj) in results.withIndex()) {
        val box = obj.boundingBox

        Log.d(TAG, "Detected object: ${i} ")
        Log.d(TAG, "  boundingBox: (${box.left}, ${box.top}) - (${box.right},${box.bottom})")

        for ((j, category) in obj.categories.withIndex()) {
            Log.d(TAG, "    Label $j: ${category.label}")
            val confidence: Int = category.score.times(100).toInt()
            Log.d(TAG, "    Confidence: ${confidence}%")
        }
    }
} 

Now your object detector is ready! Compile and run the app by clicking Run ( ) in the Android Studio toolbar. Once the app has shown up on the device, tap on any of the preset images to start the object detector. Then look at the Logcat window*(* *)* inside your IDE, and you should see something similar to this:

D/TFLite-ODT: Detected object: 0 
D/TFLite-ODT:   boundingBox: (0.0, 15.0) - (2223.0,1645.0)
D/TFLite-ODT:     Label 0: dining table
D/TFLite-ODT:     Confidence: 77%
D/TFLite-ODT: Detected object: 1 
D/TFLite-ODT:   boundingBox: (702.0, 3.0) - (1234.0,797.0)
D/TFLite-ODT:     Label 0: cup
D/TFLite-ODT:     Confidence: 69%

This tells you that the detector saw 2 objects. The first one is:

• An object is inside rectangle of (0, 15) – (2223, 1645)
• Label is dining table
• The model is confident that the 1st is a dining table (77%)

Technically that is all that you need to get TFLite Task Library to work: you got it all at this moment! Congratulations!

However, on the UI side, you are still at the starting point. Now you have to make use of the detected results on the UI, by post-processing the detected results.

6. Draw the detection result on the input image

In previous steps, you printed the detection result into logcat: simple and fast. In this step, you'll make use of the utility method already implemented for you in the starter app, in order to:

• draw a bounding box on an image
• draw a category name and confidence percentage inside the bounding box

1. Replace the debugPrint(results) call with the following code snippet:

val resultToDisplay = results.map {
    // Get the top-1 category and craft the display text
    val category = it.categories.first()
    val text = "${category.label}, ${category.score.times(100).toInt()}%"

    // Create a data object to display the detection result
    DetectionResult(it.boundingBox, text)
}
// Draw the detection result on the bitmap and show it.
val imgWithResult = drawDetectionResult(bitmap, resultToDisplay)
runOnUiThread {
    inputImageView.setImageBitmap(imgWithResult)
}

Note: Because object detection is a computation intensive process and TFLite Task Library runs on the same thread where it is called, it's important to run it on a background thread to avoid blocking the app UI. You can see that we use Kotlin coroutine to call the runObjectDetection method

lifecycleScope.launch(Dispatchers.Default) { runObjectDetection(bitmap) }

2. Now click Run ( ) in the Android Studio toolbar.
3. Once the app loads, tap on one of the preset images to see the detection result.

Want to try with your own photo? Tap on the Take photo button and capture some pictures of objects around you.

7. Train a custom object detection model

In the previous step, you integrated a pre-trained TFLite object detection model to the Android app and saw for yourself that it can detect common objects, such as bowls or dining tables, in sample images. However, your goal is to detect ingredients of the dishes in the image, so general object detection doesn't suit your use case. You want to train a custom object detection model using a training dataset with the ingredients we want to detect.

Here is a dataset containing images and labels that you can use to practice training your own custom model. It was created using images from the Open Images Dataset V4.

Colaboratory

Next, let's go to Google Colab to train the custom model.

Go to Google Colab

If you're unfamiliar, Google Colaboratory, or Colab for short, is an interactive environment that lets you write and execute code. Colab is used extensively in the machine learning community, and with Colab you can import an image dataset, train an image classifier on it, and evaluate the model.

Colab notebooks execute code on Google's cloud servers, meaning you can leverage the power of Google hardware, including GPUs and TPUs, regardless of the power of your machine. All you need is a browser.

If you are unable to access Colab, you can use another environment and access the code via this GitHub link.

It will take about 30 minutes to train the custom model.

If you are in a rush, you can download a model that we have pre-trained for you on the provided dataset, and proceed to the next step.

Download the salad detection model

8. Integrate the custom TFLite model to the Android app

Now that you have trained a salad detection model, integrate it, and turn your app from a common object detector to, specifically, a salad detector.

1. Copy the salad TFLite model to the assets folder. Name the new model salad.tflite.

2. Open the MainActivity.kt file and find the ObjectDetector initialization code.
3. Replace the EfficientDet-Lite model (model.tflite) with the salad model (salad.tflite)

val detector = ObjectDetector.createFromFileAndOptions(
    this, // the application context
    "salad.tflite", // must be same as the filename in assets folder
    options
)

4. Click Run ( ) in the Android Studio toolbar to re-run the app with the new model. Voilà! The app can now recognize cheeses, salad, baked goods.

If you run the app in an Android emulator, you may find that it takes a few seconds for the object detection model to return its result. It is because TFLite quantized models are not optimized to run on an emulator. You'll find it run much faster on a physical Android device.

9. Congratulations!

You have used TFLite to train a custom model and add Object Detection capabilities to your app. That is all you need to get it up and running!

What we've covered

• How to find pre-trained TFLite object detection models on TensorFlow Hub
• How to integrate objection detection models to your Android app using TFLite Task Library
• How to train custom object detection model with TFLite Model Maker

Next Steps

• Use Firebase to enhance your TFLite model deployment
• Collect training data to train your own model
• Apply object detection in your own Android app

Learn More

• On-device Machine Learning with Google technologies