You can use ML Kit to detect and track objects in successive video frames.

<br />

When you pass an image to ML Kit, it detects up to five objects in the image
along with the position of each object in the image. When detecting objects in
video streams, each object has a unique ID that you can use to track the object
from frame to frame.

You can use a custom image classification model to classify the objects that are
detected. Refer to [Custom models with ML Kit](https://developers.google.com/ml-kit/custom-models.md.txt) for
guidance on model compatibility requirements, where to find pre-trained models,
and how to train your own models.

There are two ways to integrate a custom model. You can bundle the model by
putting it inside your app's asset folder, or you can dynamically download it
from Cloud Storage. The following table compares the two options.

| Bundled Model | Hosted Model |
|---|---|
| The model is part of your app's `.ipa` file, which increases its size. | The model is not part of your app's `.ipa` file. It is hosted by uploading to Cloud Storage. We recommend using [Cloud Storage for Firebase](https://firebase.google.com/docs/storage). |
| The model is available immediately, even when the Android device is offline | Your app must include code to download the model on demand |
| No need for a Firebase project | Requires a Firebase project (if using Cloud Storage for Firebase). |
| You must republish your app to update the model | Push model updates without republishing your app |
| No built-in A/B testing | A/B testing with [Firebase Remote Config](https://firebase.google.com/docs/remote-config) |

> [!NOTE]
> **Note:** ML Kit iOS APIs only run on 64-bit devices. If you build your app with 32-bit support, check the device's architecture before using this API.

## Try it out

- See the [vision quickstart app](https://github.com/googlesamples/mlkit/tree/master/ios/quickstarts/vision) for an example usage of the bundled model and the [automl quickstart app](https://github.com/googlesamples/mlkit/tree/master/ios/quickstarts/automl) for an example usage of the hosted model.
- See the [Material Design showcase
  app](https://github.com/googlesamples/mlkit/tree/master/ios/showcase/vision-showcase) for an end-to-end implementation of this API.

## Before you begin

1. Include the ML Kit libraries in your Podfile:

       pod 'GoogleMLKit/ObjectDetectionCustom', '8.0.0'

2. After you install or update your project's Pods, open your Xcode project
   using its `.xcworkspace`. ML Kit is supported in Xcode version 13.2.1
   or higher.

3. **If you want to download a model using Cloud Storage for Firebase** , make
   sure you
   [add Firebase to your iOS project](https://firebase.google.com/docs/ios/setup),
   if you have not already done so. This is not required when you bundle the
   model.

## 1. Load the model

### Configure a local model source

To bundle the model with your app:

1. Copy the model file (usually ending in `.tflite` or `.lite`) to your Xcode
   project, taking care to select `Copy bundle resources` when you do so. The
   model file will be included in the app bundle and available to ML Kit.

2. Create `LocalModel` object, specifying the path to the model file:

   ### Swift

   ```swift
   let localModel = LocalModel(path: localModelFilePath)
   ```

   ### Objective-C

   ```objective-c
   MLKLocalModel *localModel =
       [[MLKLocalModel alloc] initWithPath:localModelFilePath];
   ```

> [!NOTE]
> **Note:** if you have a model that was trained with [AutoML Vision Edge in
> Firebase](https://firebase.google.com/docs/ml/train-image-labeler) (not Google Cloud), this may not work. Follow the [migration guide](https://developers.google.com/ml-kit/vision/image-labeling/automl/migrate) for instructions.

### Configure a remotely-hosted model source

To use the remotely-hosted model, you must download the model file to the
device's local storage using your own app logic, and then load it as a local
model. We recommend using Cloud Storage for Firebase to host a model. For
implementation details, see the
[Firebase ML to Cloud Storage migration guide](https://firebase.google.com/docs/ml/migrate-to-cloud-storage).

## 2. Configure the object detector

After you configure your model sources, configure the object detector for your
use case with a `CustomObjectDetectorOptions` object. You can change the
following settings:

| Object Detector Settings ||
|---|---|
| Detection mode | `STREAM_MODE` (default) \| `SINGLE_IMAGE_MODE` In `STREAM_MODE` (default), the object detector runs with low latency, but might produce incomplete results (such as unspecified bounding boxes or category labels) on the first few invocations of the detector. Also, in `STREAM_MODE`, the detector assigns tracking IDs to objects, which you can use to track objects across frames. Use this mode when you want to track objects, or when low latency is important, such as when processing video streams in real time. In `SINGLE_IMAGE_MODE`, the object detector returns the result after the object's bounding box is determined. If you also enable classification it returns the result after the bounding box and category label are both available. As a consequence, detection latency is potentially higher. Also, in `SINGLE_IMAGE_MODE`, tracking IDs are not assigned. Use this mode if latency isn't critical and you don't want to deal with partial results. |
| Detect and track multiple objects | `false` (default) \| `true` Whether to detect and track up to five objects or only the most prominent object (default). |
| Classify objects | `false` (default) \| `true` Whether or not to classify detected objects by using the provided custom classifier model. To use your custom classification model, you need to set this to `true`. |
| Classification confidence threshold | Minimum confidence score of detected labels. If not set, any classifier threshold specified by the model's metadata will be used. If the model does not contain any metadata or the metadata does not specify a classifier threshold, a default threshold of 0.0 will be used. |
| Maximum labels per object | Maximum number of labels per object that the detector will return. If not set, the default value of 10 will be used. |

If you only have a locally-bundled model, just create an object detector from
your `LocalModel` object:

### Swift

```swift
let options = CustomObjectDetectorOptions(localModel: localModel)
options.detectorMode = .singleImage
options.shouldEnableClassification = true
options.shouldEnableMultipleObjects = true
options.classificationConfidenceThreshold = NSNumber(value: 0.5)
options.maxPerObjectLabelCount = 3
```

### Objective-C

```objective-c
MLKCustomObjectDetectorOptions *options =
    [[MLKCustomObjectDetectorOptions alloc] initWithLocalModel:localModel];
options.detectorMode = MLKObjectDetectorModeSingleImage;
options.shouldEnableClassification = YES;
options.shouldEnableMultipleObjects = YES;
options.classificationConfidenceThreshold = @(0.5);
options.maxPerObjectLabelCount = 3;
```

If you have a remotely-hosted model, you will have to check that it has been
downloaded before you run it.

Although you only have to confirm this before running the object detector, if
you have both a remotely-hosted model and a locally-bundled model, it might make
sense to perform this check when instantiating the `ObjectDetector`: create a
detector from the remote model if it's been downloaded, and from the local model
otherwise.

### Swift

```swift
// Path where your download logic saves the model
let documentDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let localModelURL = documentDirectory.appendingPathComponent("my_remote_model.tflite")

let model: LocalModel
if FileManager.default.fileExists(atPath: localModelURL.path) {
  // Use the downloaded model
  model = LocalModel(path: localModelURL.path)
} else {
  // Fall back to bundled model
  guard let bundledModelPath = Bundle.main.path(forResource: "model", ofType: "tflite") else { return }
  model = LocalModel(path: bundledModelPath)
}

let options = CustomObjectDetectorOptions(localModel: model)
options.detectorMode = .singleImage
options.shouldEnableClassification = true
options.shouldEnableMultipleObjects = true
options.classificationConfidenceThreshold = NSNumber(value: 0.5)
options.maxPerObjectLabelCount = 3
let objectDetector = ObjectDetector.objectDetector(options: options)
```

### Objective-C

```objective-c
NSString *documentsDirectory = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) firstObject];
NSString *localModelPath = [documentsDirectory stringByAppendingPathComponent:@"my_remote_model.tflite"];

MLKLocalModel *model;
if ([NSFileManager.defaultManager fileExistsAtPath:localModelPath]) {
  // Use the downloaded model
  model = [[MLKLocalModel alloc] initWithPath:localModelPath];
} else {
  // Fall back to bundled model
  NSString *bundledModelPath = [NSBundle.mainBundle pathForResource:@"model" ofType:@"tflite"];
  model = [[MLKLocalModel alloc] initWithPath:bundledModelPath];
}

MLKCustomObjectDetectorOptions *options = [[MLKCustomObjectDetectorOptions alloc] initWithLocalModel:model];
options.detectorMode = MLKObjectDetectorModeSingleImage;
options.shouldEnableClassification = YES;
options.shouldEnableMultipleObjects = YES;
options.classificationConfidenceThreshold = @(0.5);
options.maxPerObjectLabelCount = 3;
MLKObjectDetector *objectDetector = [MLKObjectDetector objectDetectorWithOptions:options];
```

If you only have a remotely-hosted model, you should disable model-related
functionality---for example, gray-out or hide part of your UI---until
you confirm the model has been downloaded.

### Swift

```swift
let documentDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let localModelURL = documentDirectory.appendingPathComponent("my_remote_model.tflite")
if FileManager.default.fileExists(atPath: localModelURL.path) {
  // Model is already cached, initialize immediately
  self.initializeDetector(with: localModelURL)
} else {
  // Model is not yet available, show loading UI and start download
  self.showLoadingUI()
  let storage = Storage.storage()
  let modelRef = storage.reference(forURL: "gs://YOUR_BUCKET/path/to/model.tflite")
  modelRef.write(toFile: localModelURL) { url, error in
    self.hideLoadingUI()
    if let error = error {
      // Handle download error
      self.showErrorUI()
    } else if let modelURL = url {
      // Download success, initialize detector
      self.initializeDetector(with: modelURL)
    }
  }
}

func initializeDetector(with modelURL: URL) {
  let localModel = LocalModel(path: modelURL.path)
  let options = CustomObjectDetectorOptions(localModel: localModel)
  options.detectorMode = .singleImage
  options.shouldEnableClassification = true
  options.shouldEnableMultipleObjects = true
  self.objectDetector = ObjectDetector.objectDetector(options: options)
  // Enable ML features in UI
  self.enableMLFeatures()
}
```

### Objective-C

```objective-c
NSString *documentsDirectory = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) firstObject];
NSString *localModelPath = [documentsDirectory stringByAppendingPathComponent:@"my_remote_model.tflite"];
NSURL *localModelURL = [NSURL fileURLWithPath:localModelPath];

if ([NSFileManager.defaultManager fileExistsAtPath:localModelPath]) {
  // Model is already cached, initialize immediately
  [self initializeDetectorWithURL:localModelURL];
} else {
  // Model is not yet available, show loading UI and start download
  [self showLoadingUI];

  FIRStorage *storage = [FIRStorage storage];
  FIRStorageReference *modelRef = [storage referenceForURL:@"gs://YOUR_BUCKET/path/to/model.tflite"];

  [modelRef writeToFile:localModelURL
             completion:^(NSURL * _Nullable URL, NSError * _Nullable error) {
               [self hideLoadingUI];
               if (error != nil) {
                 // Handle download error
                 [self showErrorUI];
               } else {
                 // Download success, initialize detector
                 [self initializeDetectorWithURL:URL];
               }
             }];
}

- (void)initializeDetectorWithURL:(NSURL *)modelURL {
  MLKLocalModel *localModel = [[MLKLocalModel alloc] initWithPath:modelURL.path];
  MLKCustomObjectDetectorOptions *options = [[MLKCustomObjectDetectorOptions alloc] initWithLocalModel:localModel];
  options.detectorMode = MLKObjectDetectorModeSingleImage;
  options.shouldEnableClassification = YES;
  options.shouldEnableMultipleObjects = YES;
  self.objectDetector = [MLKObjectDetector objectDetectorWithOptions:options];

  // Enable ML features in UI
  [self enableMLFeatures];
}
```

The object detection and tracking API is optimized for these two core use
cases:

- Live detection and tracking of the most prominent object in the camera viewfinder.
- The detection of multiple objects from a static image.

To configure the API for these use cases:

### Swift

```swift
// Live detection and tracking
let options = CustomObjectDetectorOptions(localModel: localModel)
options.shouldEnableClassification = true
options.maxPerObjectLabelCount = 3

// Multiple object detection in static images
let options = CustomObjectDetectorOptions(localModel: localModel)
options.detectorMode = .singleImage
options.shouldEnableMultipleObjects = true
options.shouldEnableClassification = true
options.maxPerObjectLabelCount = 3
```

### Objective-C

```objective-c
// Live detection and tracking
MLKCustomObjectDetectorOptions *options =
    [[MLKCustomObjectDetectorOptions alloc] initWithLocalModel:localModel];
options.shouldEnableClassification = YES;
options.maxPerObjectLabelCount = 3;

// Multiple object detection in static images
MLKCustomObjectDetectorOptions *options =
    [[MLKCustomObjectDetectorOptions alloc] initWithLocalModel:localModel];
options.detectorMode = MLKObjectDetectorModeSingleImage;
options.shouldEnableMultipleObjects = YES;
options.shouldEnableClassification = YES;
options.maxPerObjectLabelCount = 3;
```

## 3. Prepare the input image

Create a [`VisionImage`](https://developers.google.com/ml-kit/reference/swift/mlkitvision/api/reference/Classes/VisionImage.md.txt) object using a `UIImage` or a
`CMSampleBuffer`.

If you use a `UIImage`, follow these steps:

- Create a [`VisionImage`](https://developers.google.com/ml-kit/reference/swift/mlkitvision/api/reference/Classes/VisionImage.md.txt) object with the `UIImage`. Make sure to specify the correct `.orientation`.

  ### Swift

  ```
  let image = VisionImage(image: UIImage)
  visionImage.orientation = image.imageOrientation
  ```

  ### Objective-C

  ```
  MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:image];
  visionImage.orientation = image.imageOrientation;
  ```

If you use a `CMSampleBuffer`, follow these steps:

- Specify the orientation of the image data contained in the
  `CMSampleBuffer`.

  To get the image orientation:

  ### Swift

  ```
  func imageOrientation(
    deviceOrientation: UIDeviceOrientation,
    cameraPosition: AVCaptureDevice.Position
  ) -> UIImage.Orientation {
    switch deviceOrientation {
    case .portrait:
      return cameraPosition == .front ? .leftMirrored : .right
    case .landscapeLeft:
      return cameraPosition == .front ? .downMirrored : .up
    case .portraitUpsideDown:
      return cameraPosition == .front ? .rightMirrored : .left
    case .landscapeRight:
      return cameraPosition == .front ? .upMirrored : .down
    case .faceDown, .faceUp, .unknown:
      return .up
    }
  }
        
  ```

  ### Objective-C

  ```
  - (UIImageOrientation)
    imageOrientationFromDeviceOrientation:(UIDeviceOrientation)deviceOrientation
                           cameraPosition:(AVCaptureDevicePosition)cameraPosition {
    switch (deviceOrientation) {
      case UIDeviceOrientationPortrait:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationLeftMirrored
                                                              : UIImageOrientationRight;

      case UIDeviceOrientationLandscapeLeft:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationDownMirrored
                                                              : UIImageOrientationUp;
      case UIDeviceOrientationPortraitUpsideDown:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationRightMirrored
                                                              : UIImageOrientationLeft;
      case UIDeviceOrientationLandscapeRight:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationUpMirrored
                                                              : UIImageOrientationDown;
      case UIDeviceOrientationUnknown:
      case UIDeviceOrientationFaceUp:
      case UIDeviceOrientationFaceDown:
        return UIImageOrientationUp;
    }
  }
        
  ```
- Create a [`VisionImage`](https://developers.google.com/ml-kit/reference/swift/mlkitvision/api/reference/Classes/VisionImage.md.txt) object using the `CMSampleBuffer` object and orientation:

  ### Swift

  ```
  let image = VisionImage(buffer: sampleBuffer)
  image.orientation = imageOrientation(
    deviceOrientation: UIDevice.current.orientation,
    cameraPosition: cameraPosition)
  ```

  ### Objective-C

  ```
   MLKVisionImage *image = [[MLKVisionImage alloc] initWithBuffer:sampleBuffer];
   image.orientation =
     [self imageOrientationFromDeviceOrientation:UIDevice.currentDevice.orientation
                                  cameraPosition:cameraPosition];
  ```

## 4. Create and run the object detector

1. Create a new object detector:

   ### Swift

   ```swift
   let objectDetector = ObjectDetector.objectDetector(options: options)
   ```

   ### Objective-C

   ```objective-c
   MLKObjectDetector *objectDetector = [MLKObjectDetector objectDetectorWithOptions:options];
   ```
2. Then, use the detector:

   Asynchronously:

   ### Swift

   ```swift
   objectDetector.process(image) { objects, error in
       guard error == nil, let objects = objects, !objects.isEmpty else {
           // Handle the error.
           return
       }
       // Show results.
   }
   ```

   ### Objective-C

   ```objective-c
   [objectDetector
       processImage:image
         completion:^(NSArray *_Nullable objects,
                      NSError *_Nullable error) {
           if (objects.count == 0) {
               // Handle the error.
               return;
           }
           // Show results.
        }];
   ```

   Synchronously:

   ### Swift

   ```swift
   var objects: [Object]
   do {
       objects = try objectDetector.results(in: image)
   } catch let error {
       // Handle the error.
       return
   }
   // Show results.
   ```

   ### Objective-C

   ```objective-c
   NSError *error;
   NSArray *objects =
       [objectDetector resultsInImage:image error:&error];
   // Show results or handle the error.
   ```

## 5. Get information about labeled objects

If the call to the image processor succeeds, it either passes a list of
`Object`s to the completion handler or returns the list, depending on
whether you called the asynchronous or synchronous method.

> [!NOTE]
> **Note:** In streaming mode, the object detector might need to process 30 or more frames, depending on device performance, before it detects the first object.

Each `Object` contains the following properties:
`frame` A `CGRect` indicating the position of the object in the image. `trackingID` An integer that identifies the object across images, or \`nil\` in SINGLE_IMAGE_MODE. `labels` `label.text` The label's text description. Only returned if the LiteRT model's metadata contains label descriptions. `label.index` The label's index among all the labels supported by the classifier. `label.confidence` The confidence value of the object classification.

### Swift

```swift
// objects contains one item if multiple object detection wasn't enabled.
for object in objects {
  let frame = object.frame
  let trackingID = object.trackingID
  let description = object.labels.enumerated().map { (index, label) in
    "Label \(index): \(label.text), \(label.confidence), \(label.index)"
  }.joined(separator: "\n")
}
```

### Objective-C

```objective-c
// The list of detected objects contains one item if multiple object detection
// wasn't enabled.
for (MLKObject *object in objects) {
  CGRect frame = object.frame;
  NSNumber *trackingID = object.trackingID;
  for (MLKObjectLabel *label in object.labels) {
    NSString *labelString =
        [NSString stringWithFormat:@"%@, %f, %lu",
                                   label.text,
                                   label.confidence,
                                   (unsigned long)label.index];
  }
}
```

## Ensuring a great user experience

For the best user experience, follow these guidelines in your app:

- Successful object detection depends on the object's visual complexity. In order to be detected, objects with a small number of visual features might need to take up a larger part of the image. You should provide users with guidance on capturing input that works well with the kind of objects you want to detect.
- When you use classification, if you want to detect objects that don't fall cleanly into the supported categories, implement special handling for unknown objects.

Also, check out the
[ML Kit Material Design showcase app](https://github.com/googlesamples/mlkit/tree/master/ios/showcase/vision-showcase) and the
Material Design
[Patterns for machine learning-powered features](https://material.io/collections/machine-learning/) collection.

## Improving performance

If you want to use object detection in a real-time application, follow these guidelines to achieve the best framerates:

<br />

- When you use streaming mode in a real-time application, don't use multiple
  object detection, as most devices won't be able to produce adequate framerates.

- For processing video frames, use the `results(in:)` synchronous API of the detector. Call this method from the [`AVCaptureVideoDataOutputSampleBufferDelegate`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate)'s [`captureOutput(_, didOutput:from:)`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate/1385775-captureoutput) function to synchronously get results from the given video frame. Keep [`AVCaptureVideoDataOutput`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutput)'s [`alwaysDiscardsLateVideoFrames`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutput/1385780-alwaysdiscardslatevideoframes) as `true` to throttle calls to the detector. If a new video frame becomes available while the detector is running, it will be dropped.
- If you use the output of the detector to overlay graphics on the input image, first get the result from ML Kit, then render the image and overlay in a single step. By doing so, you render to the display surface only once for each processed input frame. See the [updatePreviewOverlayViewWithLastFrame](https://github.com/googlesamples/mlkit/blob/master/ios/quickstarts/vision/VisionExample/CameraViewController.swift) in the ML Kit quickstart sample for an example.