Pose Classification Tips

With the ML Kit Pose Detection API, you can derive meaningful interpretations of a pose by checking the relative positions of various body parts. This page demonstrates two examples.

Recognizing gestures

When two or more landmarks are close to each other, you can recognize gestures. For example, when the landmark for one or more fingers on a hand is close to the landmark for the nose, you can infer the user is most likely touching their face.

Figure 1.Interpreting a pose

Recognizing a yoga pose

You can identify a yoga pose by computing the angles of various joints. For example, Figure 2 below shows the Warrior II yoga pose. The approximate angles that identify this pose are written in:

Figure 2.Breaking a pose into angles

This pose can be described as the following combination of approximate body part angles:

  • 90 degree angle at both shoulders
  • 180 degrees at both elbows
  • 90 degree angle at the front leg and waist
  • 180 degree angle at the back knee
  • 135 degree angle at the waist

You can use the pose landmarks to compute these angles. For example, the angle at the right front leg and waist is the angle between the line from the right shoulder to the right hip, and the line from the right hip to the right knee.

Once you've computed all the angles needed to identify the pose, you can check to see if there's a match, in which case you've recognized the pose.

The code snippet below demonstrates how to use the X and Y coordinates to calculate the angle between two body parts. This approach to classification has some limitations. By only checking X and Y, the calculated angles vary according to the angle between the subject and the camera. You'll get the best results with a level, straight forward, head-on image. You could also try extending this algorithm by making use of the Z coordinate and see if it performs better for your use case.

Computing landmark angles on Android

The following method computes the angle between any three landmarks. It ensures the angle returned is between 0 and 180 degrees.

Kotlin

fun getAngle(firstPoint: PoseLandmark, midPoint: PoseLandmark, lastPoint: PoseLandmark): Double {
        var result = Math.toDegrees(atan2(lastPoint.getPosition().y - midPoint.getPosition().y,
                lastPoint.getPosition().x - midPoint.getPosition().x)
                - atan2(firstPoint.getPosition().y - midPoint.getPosition().y,
                firstPoint.getPosition().x - midPoint.getPosition().x))
        result = Math.abs(result) // Angle should never be negative
        if (result > 180) {
            result = 360.0 - result // Always get the acute representation of the angle
        }
        return result
    }

Java

static double getAngle(PoseLandmark firstPoint, PoseLandmark midPoint, PoseLandmark lastPoint) {
  double result =
        Math.toDegrees(
            atan2(lastPoint.getPosition().y - midPoint.getPosition().y,
                      lastPoint.getPosition().x - midPoint.getPosition().x)
                - atan2(firstPoint.getPosition().y - midPoint.getPosition().y,
                      firstPoint.getPosition().x - midPoint.getPosition().x));
  result = Math.abs(result); // Angle should never be negative
  if (result > 180) {
      result = (360.0 - result); // Always get the acute representation of the angle
  }
  return result;
}

Here's how to compute the angle at the right hip:

Kotlin

val rightHipAngle = getAngle(
                pose.getPoseLandmark(PoseLandmark.Type.RIGHT_SHOULDER),
                pose.getPoseLandmark(PoseLandmark.Type.RIGHT_HIP),
                pose.getPoseLandmark(PoseLandmark.Type.RIGHT_KNEE))

Java

double rightHipAngle = getAngle(
                pose.getPoseLandmark(PoseLandmark.Type.RIGHT_SHOULDER),
                pose.getPoseLandmark(PoseLandmark.Type.RIGHT_HIP),
                pose.getPoseLandmark(PoseLandmark.Type.RIGHT_KNEE));

Computing landmark angles on iOS

The following method computes the angle between any three landmarks. It ensures the angle returned is between 0 and 180 degrees.

Swift

func angle(
      firstLandmark: PoseLandmark,
      midLandmark: PoseLandmark,
      lastLandmark: PoseLandmark
  ) -> CGFloat {
      let radians: CGFloat =
          atan2(lastLandmark.position.y - midLandmark.position.y,
                    lastLandmark.position.x - midLandmark.position.x) -
            atan2(firstLandmark.position.y - midLandmark.position.y,
                    firstLandmark.position.x - midLandmark.position.x)
      var degrees = radians * 180.0 / .pi
      degrees = abs(degrees) // Angle should never be negative
      if degrees > 180.0 {
          degrees = 360.0 - degrees // Always get the acute representation of the angle
      }
      return degrees
  }

Objective-C

(CGFloat)angleFromFirstLandmark:(MLKPoseLandmark *)firstLandmark
                      midLandmark:(MLKPoseLandmark *)midLandmark
                     lastLandmark:(MLKPoseLandmark *)lastLandmark {
    CGFloat radians = atan2(lastLandmark.position.y - midLandmark.position.y,
                            lastLandmark.position.x - midLandmark.position.x) -
                      atan2(firstLandmark.position.y - midLandmark.position.y,
                            firstLandmark.position.x - midLandmark.position.x);
    CGFloat degrees = radians * 180.0 / M_PI;
    degrees = fabs(degrees); // Angle should never be negative
    if (degrees > 180.0) {
        degrees = 360.0 - degrees; // Always get the acute representation of the angle
    }
    return degrees;
}

Here's how to compute the angle at the right hip:

Swift

let rightHipAngle = angle(
      firstLandmark: pose.landmark(ofType: .rightShoulder),
      midLandmark: pose.landmark(ofType: .rightHip),
      lastLandmark: pose.landmark(ofType: .rightKnee))

Objective-C

CGFloat rightHipAngle =
    [self angleFromFirstLandmark:[pose landmarkOfType:MLKPoseLandmarkTypeRightShoulder]
                     midLandmark:[pose landmarkOfType:MLKPoseLandmarkTypeRightHip]
                    lastLandmark:[pose landmarkOfType:MLKPoseLandmarkTypeRightKnee]];