Introducing Google AI Edge Portal: Benchmark Edge AI at scale. Sign-up to request access during private preview.

Holistic landmark detection guide for Android

The MediaPipe Holistic Landmarker task lets you combine components of the face, hand, and pose landmarkers to detect human body landmarks in images or video. This task outputs holistic landmarks in normalized image coordinates and 3D world coordinates.

The code sample described in these instructions is available on GitHub. For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The MediaPipe Tasks example code is a simple implementation of a Holistic Landmarker app for Android. The example uses the camera on a physical Android device to detect holistic landmarks in a continuous video stream. The app can also detect landmarks in images from the device gallery.

You can use the app as a starting point for your own Android app, or refer to it when modifying an existing app. The Holistic Landmarker example code is hosted on GitHub.

Download the code

The following instructions show you how to create a local copy of the example code using the git command line tool.

To download the example code:

Clone the git repository using the following command:

git clone https://github.com/google-ai-edge/mediapipe-samples

Optionally, configure your git instance to use sparse checkout, so you have only the files for the Holistic Landmarker example app:
```
cd mediapipe-samples
git sparse-checkout init --cone
git sparse-checkout set examples/holistic_landmarker/android
```

Key components

The following files contain the crucial code for this holistic landmarking example application:

HolisticLandmarkerHelper.kt - Initializes the holistic landmarker and handles the model and delegate selection.
CameraFragment.kt - Handles the device camera and processes the image and video input data.
GalleryFragment.kt - Interacts with OverlayView to display the output image or video.
OverlayView.kt - Implements the display for the detected landmarks.

Setup

This section describes key steps for setting up your development environment and code projects specifically to use Holistic Landmarker. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for Android.

Dependencies

The Holistic Landmarker task uses the com.google.mediapipe:tasks-vision library. Add this dependency to the build.gradle file of your Android app:

dependencies {
    implementation 'com.google.mediapipe:tasks-vision:latest.release'
}

Model

The MediaPipe Holistic Landmarker task requires a trained model bundle that is compatible with this task. For more information on available trained models for Holistic Landmarker, see the task overview Models section.

Select and download the model, and store it within your project directory:

<dev-project-root>/src/main/assets

Specify the path of the model within the ModelAssetPath parameter:

val modelName = "holistic_landmarker.task"
baseOptionsBuilder.setModelAssetPath(modelName)

Create the task

The MediaPipe Holistic Landmarker task uses the createFromOptions() function to set up the task. The createFromOptions() function accepts values for the configuration options. For more information on configuration options, see Configuration options.

The Holistic Landmarker supports the following input data types: still images, video files, and live video streams. You need to specify the running mode corresponding to your input data type when creating the task. Choose the tab for your input data type to see how to create the task.

Image

val baseOptionsBuilder = BaseOptions.builder().setModelAssetPath(modelName)
val optionsBuilder = 
    HolisticLandmarkerOptions.builder()
        .setBaseOptions(baseOptionsBuilder.build())
        .setMinFaceDetectionConfidence(0.5f)
        .setMinPoseDetectionConfidence(0.5f)
        .setMinHandLandmarksConfidence(0.5f)
        .setRunningMode(RunningMode.IMAGE)

val options = optionsBuilder.build()
val holisticLandmarker = HolisticLandmarker.createFromOptions(context, options)

Video

val baseOptionsBuilder = BaseOptions.builder().setModelAssetPath(modelName)
val optionsBuilder = 
    HolisticLandmarkerOptions.builder()
        .setBaseOptions(baseOptionsBuilder.build())
        .setMinFaceDetectionConfidence(0.5f)
        .setMinPoseDetectionConfidence(0.5f)
        .setMinHandLandmarksConfidence(0.5f)
        .setRunningMode(RunningMode.VIDEO)

val options = optionsBuilder.build()
val holisticLandmarker = HolisticLandmarker.createFromOptions(context, options)

Live stream

val baseOptionsBuilder = BaseOptions.builder().setModelAssetPath(modelName)
val optionsBuilder = 
    HolisticLandmarkerOptions.builder()
        .setBaseOptions(baseOptionsBuilder.build())
        .setMinFaceDetectionConfidence(0.5f)
        .setMinPoseDetectionConfidence(0.5f)
        .setMinHandLandmarksConfidence(0.5f)
        .setRunningMode(RunningMode.LIVE_STREAM)
        .setResultListener(resultListener)
        .setErrorListener(errorListener)

val options = optionsBuilder.build()
val holisticLandmarker = HolisticLandmarker.createFromOptions(context, options)

Configuration options

This task has the following configuration options for Android applications:

Option Name	Description	Value Range	Default Value
`runningMode`	Sets the running mode for the task. There are three modes: IMAGE: The mode for single image inputs. VIDEO: The mode for decoded frames of a video. LIVE_STREAM: The mode for a livestream of input data, such as from a camera. In this mode, a result listener must be called to set up a listener to receive results asynchronously.	{`IMAGE, VIDEO, LIVE_STREAM`}	`IMAGE`
`minFaceDetectionConfidence`	The minimum confidence score for the face detection to be considered successful.	`Float [0.0, 1.0]`	`0.5`
`minFaceSuppressionThreshold`	The minimum non-maximum-suppression threshold for face detection to be considered overlapped.	`Float [0.0, 1.0]`	`0.3`
`minFacePresenceConfidence`	The minimum confidence score of face presence score in the face landmarks detection.	`Float [0.0, 1.0]`	`0.5`
`minPoseDetectionConfidence`	The minimum confidence score for the pose detection to be considered successful.	`Float [0.0, 1.0]`	`0.5`
`minPoseSuppressionThreshold`	The minimum non-maximum-suppression threshold for pose detection to be considered overlapped.	`Float [0.0, 1.0]`	`0.3`
`minPosePresenceConfidence`	The minimum confidence score of pose presence score in the pose landmarks detection.	`Float [0.0, 1.0]`	`0.5`
`minHandLandmarksConfidence`	The minimum confidence score of hand presence score in the hand landmarks detection.	`Float [0.0, 1.0]`	`0.5`
`outputFaceBlendshapes`	Whether to output face blendshapes classification. Face blendshapes are used for rendering the 3D face model.	`Boolean`	`false`
`outputPoseSegmentationMasks`	Whether to output segmentation masks for the human pose.	`Boolean`	`false`

Prepare data

Holistic Landmarker can detect holistic landmarks in images in any format supported by the host platform. The task also handles data input preprocessing, including resizing, rotation and value normalization. To detect landmarks in videos, you can use the API to quickly process one frame at a time, using the timestamp of the frame to determine when the landmarks occur within the video.

Run the task

Use the detect, detectForVideo, or detectAsync methods depending on your running mode to execute the task. The task returns the HolisticLandmarkerResult containing face, pose, left hand, and right hand landmarks.

The following code demonstrates how to run the task:

// Run holistic landmarker on a single image
val image = MPImage(bitmap)
val result = holisticLandmarker.detect(image)

Holistic landmark detection guide for Android Stay organized with collections Save and categorize content based on your preferences.