You can use ML Kit to label objects recognized in an image. The default model provided with
ML Kit supports 400+ different labels.

<br />

> [!NOTE]
> **Note:** ML Kit iOS APIs only run on 64-bit devices. If you build your app with 32-bit support, check the device's architecture before using this API.

## Try it out

- Play around with [the sample app](https://github.com/googlesamples/mlkit/tree/master/ios/quickstarts/vision) to see an example usage of this API.

## Before you begin

1. Include the following ML Kit pods in your Podfile:

   ```
   pod 'GoogleMLKit/ImageLabeling', '8.0.0'
   ```
2. After you install or update your project's Pods, open your Xcode project using its `.xcworkspace`. ML Kit is supported in Xcode version 12.4 or greater.

Now you are ready to label images.

## 1. Prepare the input image

Create a [`VisionImage`](https://developers.google.com/ml-kit/reference/swift/mlkitvision/api/reference/Classes/VisionImage.md.txt) object using a `UIImage` or a
`CMSampleBuffer`.

If you use a `UIImage`, follow these steps:

- Create a [`VisionImage`](https://developers.google.com/ml-kit/reference/swift/mlkitvision/api/reference/Classes/VisionImage.md.txt) object with the `UIImage`. Make sure to specify the correct `.orientation`.

  ### Swift

  ```
  let image = VisionImage(image: UIImage)
  visionImage.orientation = image.imageOrientation
  ```

  ### Objective-C

  ```
  MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:image];
  visionImage.orientation = image.imageOrientation;
  ```

If you use a `CMSampleBuffer`, follow these steps:

- Specify the orientation of the image data contained in the
  `CMSampleBuffer`.

  To get the image orientation:

  ### Swift

  ```
  func imageOrientation(
    deviceOrientation: UIDeviceOrientation,
    cameraPosition: AVCaptureDevice.Position
  ) -> UIImage.Orientation {
    switch deviceOrientation {
    case .portrait:
      return cameraPosition == .front ? .leftMirrored : .right
    case .landscapeLeft:
      return cameraPosition == .front ? .downMirrored : .up
    case .portraitUpsideDown:
      return cameraPosition == .front ? .rightMirrored : .left
    case .landscapeRight:
      return cameraPosition == .front ? .upMirrored : .down
    case .faceDown, .faceUp, .unknown:
      return .up
    }
  }
        
  ```

  ### Objective-C

  ```
  - (UIImageOrientation)
    imageOrientationFromDeviceOrientation:(UIDeviceOrientation)deviceOrientation
                           cameraPosition:(AVCaptureDevicePosition)cameraPosition {
    switch (deviceOrientation) {
      case UIDeviceOrientationPortrait:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationLeftMirrored
                                                              : UIImageOrientationRight;

      case UIDeviceOrientationLandscapeLeft:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationDownMirrored
                                                              : UIImageOrientationUp;
      case UIDeviceOrientationPortraitUpsideDown:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationRightMirrored
                                                              : UIImageOrientationLeft;
      case UIDeviceOrientationLandscapeRight:
        return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationUpMirrored
                                                              : UIImageOrientationDown;
      case UIDeviceOrientationUnknown:
      case UIDeviceOrientationFaceUp:
      case UIDeviceOrientationFaceDown:
        return UIImageOrientationUp;
    }
  }
        
  ```
- Create a [`VisionImage`](https://developers.google.com/ml-kit/reference/swift/mlkitvision/api/reference/Classes/VisionImage.md.txt) object using the `CMSampleBuffer` object and orientation:

  ### Swift

  ```
  let image = VisionImage(buffer: sampleBuffer)
  image.orientation = imageOrientation(
    deviceOrientation: UIDevice.current.orientation,
    cameraPosition: cameraPosition)
  ```

  ### Objective-C

  ```
   MLKVisionImage *image = [[MLKVisionImage alloc] initWithBuffer:sampleBuffer];
   image.orientation =
     [self imageOrientationFromDeviceOrientation:UIDevice.currentDevice.orientation
                                  cameraPosition:cameraPosition];
  ```

## 2. Configure and run the image labeler

To label objects in an image, pass the `VisionImage` object to the `ImageLabeler`'s `processImage()` method.

<br />

1. First, get an instance of `ImageLabeler`.

### Swift

```swift
let labeler = ImageLabeler.imageLabeler()

// Or, to set the minimum confidence required:
// let options = ImageLabelerOptions()
// options.confidenceThreshold = 0.7
// let labeler = ImageLabeler.imageLabeler(options: options)
```

### Objective-C

```objective-c
MLKImageLabeler *labeler = [MLKImageLabeler imageLabeler];

// Or, to set the minimum confidence required:
// MLKImageLabelerOptions *options =
//         [[MLKImageLabelerOptions alloc] init];
// options.confidenceThreshold = 0.7;
// MLKImageLabeler *labeler =
//         [MLKImageLabeler imageLabelerWithOptions:options];
```

1. Then, pass the image to the `processImage()` method:

### Swift

```swift
labeler.process(image) { labels, error in
    guard error == nil, let labels = labels else { return }

    // Task succeeded.
    // ...
}
```

### Objective-C

```objective-c
[labeler processImage:image
completion:^(NSArray *_Nullable labels,
            NSError *_Nullable error) {
   if (error != nil) { return; }

   // Task succeeded.
   // ...
}];
```

## 3. Get information about labeled objects

If image labeling succeeds, the completion handler receives an array of
`ImageLabel` objects. Each `ImageLabel` object represents something that was
labeled in the image. The base model supports [400+ different labels](https://developers.google.com/ml-kit/vision/image-labeling/label-map.md.txt).
You can get each label's text description, index among all labels supported by
the model, and the confidence score of the match. For example:

### Swift

```swift
for label in labels {
    let labelText = label.text
    let confidence = label.confidence
    let index = label.index
}
```

### Objective-C

```objective-c
for (MLKImageLabel *label in labels) {
   NSString *labelText = label.text;
   float confidence = label.confidence;
   NSInteger index = label.index;
}
```

## Tips to improve real-time performance

If you want to label images in a real-time application, follow these
guidelines to achieve the best framerates:

- For processing video frames, use the `results(in:)` synchronous API of the image labeler. Call this method from the [`AVCaptureVideoDataOutputSampleBufferDelegate`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate)'s [`captureOutput(_, didOutput:from:)`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutputsamplebufferdelegate/1385775-captureoutput) function to synchronously get results from the given video frame. Keep [`AVCaptureVideoDataOutput`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutput)'s [`alwaysDiscardsLateVideoFrames`](https://developer.apple.com/documentation/avfoundation/avcapturevideodataoutput/1385780-alwaysdiscardslatevideoframes) as `true` to throttle calls to the image labeler. If a new video frame becomes available while the image labeler is running, it will be dropped.
- If you use the output of the image labeler to overlay graphics on the input image, first get the result from ML Kit, then render the image and overlay in a single step. By doing so, you render to the display surface only once for each processed input frame. See the [updatePreviewOverlayViewWithLastFrame](https://github.com/googlesamples/mlkit/blob/master/ios/quickstarts/vision/VisionExample/CameraViewController.swift) in the ML Kit quickstart sample for an example.