Learn how to use the [Scene Semantics API](https://developers.google.com/ar/develop/scene-semantics) in your own apps.

The Scene Semantics API enables developers to understand the scene surrounding the user, by providing ML model-based, real-time semantic information. Given an image of an outdoor scene, the API returns a label for each pixel across a set of useful semantic classes, such a sky, building, tree, road, sidewalk, vehicle, person, and more. In addition to pixel labels, the Scene Semantics API also offers confidence values for each pixel label and an easy-to-use way to query the prevalence of a given label in an outdoor scene.

Your browser does not support the video tag.

From left to right, examples of an input image, the semantic image of pixel labels, and the corresponding confidence image:

![Example of input image, semantic image, and semantic confidence image.](https://developers.google.com/static/codelabs/arcore-scene-semantics-geospatial-depth/images/semantics-images.png)

## Prerequisites

Make sure that you understand [fundamental AR concepts](https://developers.google.com/ar/develop/fundamentals)
and how to [configure an ARCore session](https://developers.google.com/ar/develop/unity-arf/session-config) before proceeding.

## Enable Scene Semantics

In [a new ARCore session](https://developers.google.com/ar/develop/unity-arf/session-config), check whether a user's device supports the Scene Semantics API. Not all ARCore-compatible devices support the Scene Semantics API due to processing power constraints.

To save resources, Scene Semantics is disabled by default on ARCore. Enable semantic mode to have your app use the Scene Semantics API.

In your **ARCoreExtensionsConfig**, set the Semantics Mode to Enabled.

![Semantics mode set to Enabled.](https://developers.google.com/static/ar/develop/scene-semantics/images/unity-enable-semantics-mode.png)

If using iOS, Semantics must also be enabled in the project settings:

1. Navigate to **Edit** \> **Project Settings** \> **XR Plug-In Management** \> **ARCore Extensions**.
2. Under **Optional Features** , select **Semantics on iOS**.

![Semantics on iOS Enabled in Optional Features.](https://developers.google.com/static/ar/develop/scene-semantics/images/unity-enable-semantics-mode-ios.png)

## Obtain the semantic image

Once Scene Semantics is enabled, the semantic image can be retrieved. The semantic image is a [`TextureFormat.R8`](https://docs.unity3d.com/ScriptReference/TextureFormat.R8.html) image, where each pixel corresponds to a semantic label defined by [`SemanticLabel`](https://developers.google.com/ar/reference/unity-arf/namespace/Google/XR/ARCoreExtensions#semanticlabel).

Use [`ArSemanticManager.TryGetSemanticTexture()`](https://developers.google.com/ar/reference/unity-arf/class/Google/XR/ARCoreExtensions/ARSemanticManager#trygetsemantictexture) to acquire the semantic image:

    if (semanticManager.TryGetSemanticTexture(out Texture2D semanticImage))
    {
        using (semanticImage)
        {
            // Use the semantic image here.
        }
    }

Output semantic images should be available after about 1-3 frames from the start of the session, depending on the device.

## Obtain the confidence image

In addition to the semantic image, which provides a label for each pixel, the API also provides a confidence image of corresponding pixel confidence values. The confidence image is a [`TextureFormat.Alpha8`](https://docs.unity3d.com/ScriptReference/TextureFormat.Alpha8.html) image, where each pixel corresponds to a value in the range `[0, 255]`, corresponding to the probability associated with the semantic label for each pixel.

Use [`ArSemanticManager.TryGetSemanticConfidenceTexture()`](https://developers.google.com/ar/reference/unity-arf/class/Google/XR/ARCoreExtensions/ARSemanticManager#trygetsemanticconfidencetexture) to acquire the semantic confidence image:

    if (semanticManager.TryGetSemanticConfidenceTexture(out Texture2D semanticConfidenceImage))
    {
        using (semanticConfidenceImage)
        {
            // Use the semantic confidence image here.
        }
    }

Output confidence images should be available after about 1-3 frames from the start of the session, depending on the device.

## Query the fraction of pixels for a semantic label

You can also query the fraction of pixels in the current frame that belong to a particular class, such as sky. This query is more efficient than returning the semantic image and performing a pixel-wise search for a specific label. The returned fraction is a float value in the range `[0.0, 1.0]`.

Use [`ArSemanticManager.GetSemanticLabelFraction()`](https://developers.google.com/ar/reference/unity-arf/class/Google/XR/ARCoreExtensions/ARSemanticManager#getsemanticlabelfraction) to acquire the fraction for a given label:

    var fraction = semanticManager.GetSemanticLabelFraction(SemanticLabel.SKY);