MediaPipe Pose with 3D landmarks, and more Python & JavaScript Solution APIs

• MediaPipe Pose
  • New pose landmark model with Z coordinates (BlazePose GHUM 3D), generated from synthetic data
  • Added tutorials and Colabs for Pose Classification
• MediaPipe Objectron
  • Added Python Solution API
• MediaPipe Face Detection
  • Added Python Solution API and JavaScript Solution API
• MediaPipe Face Mesh
  • Face Geometry Module (code) now supports face-detection input (to generate a face geometry from a detection), in addition to the existing support for face-landmark input
• Image: a new multi-backend image data container:
  • Covering both CPU and GPU storage
  • Currently used in ImageToTensorCalculator and Face Detection Module
  • Companion utility ToImageCalculator and FromImageCalculator

Experimental new framework APIs

• Released experimental new framework APIs
• Updated MediaPipe Objectron to support simultaneous detection of multiple objects
• Bug fixes
  • Fixed a crash in the MediaPipe Iris iOS example app
  • Fixed memory leak in Android example apps (in ExternalTextureConverter)
  • Fixed a file read mode issue in pose_landmark_model_loader.pbtxt.

MediaPipe Holistic, and MediaPipe in Python and JavaScript

• MediaPipe Holistic for real-time simultaneous tracking of human pose, face and hand landmarks
  • Google AI blog post
• MediaPipe in Python
  • Ready-to-use Python Solution APIs for MediaPipe Face Mesh, Hands, Pose and Holistic
  • Try it in MediaPipe on Google Colab
• MediaPipe in JavaScript
  • Ready-to-use JavaScript Solution APIs for MediaPipe Face Mesh, Hands, Pose and Holistic
  • Try it in MediaPipe on CodePen
• Switch build from C++14 to C++17

Python Solution APIs

• MediaPipe in Python
  • Released Python Solution API for MediaPipe Face Mesh and MediaPipe Hands
  • Updated Python Solution API for MediaPipe Pose
  • Also released usage examples as Google Colabs
• MediaPipe Objectron
  • Released a faster two-stage pipeline
  • Added support for more object classes: shoe, chair, cup and camera
  • Released training dataset, (to be) announced in a Google AI Blog post
• New Tensor class
  • Released as a multi-dimensional tensor data container, supporting multiple backends like CPU, Metal buffer, GL buffer and GL texture 2D
  • Added new Tensor-based pre-processing, inference and post-processing calculators in mediapipe/calculators/tensor, branched from existing calculators in mediapipe/calculators/tflite
  • Most of the (sub-)graph in mediapipe/modules have been updated to use Tensor and the associated calculators. The plan is to fully migrate all in the repo (and deprecate mediapipe/calculators/tflite by end of 2020.
• MediaPipe Hands
  • Refactored graphs to depend on the new palm_detection and hand_landmark module.
  • Improved model speed for both palm detection and hand landmark.
  • Extended the main hand tracking example apps to support multiple hands, to replace the separate multi-hand tracking examples.
• MediaPipe Face Detection
  • Refactored graphs to depend on the face_detection module.

3D Face Transform in MediaPipe Face Mesh

• Added 3D Face Transform to MediaPipe Face Mesh
  • Enables Face Effect Rendering
  • Demonstrated with an Face Effect Example App
  • More info in Google Developers Blog post
• Added MediaPipe Models and Model Cards doc

MediaPipe Instant Motion Tracking

• MediaPipe Instant Motion Tracking that performs AR tracking without initialization or calibration.
  • Google Developers Blog post
• MediaPipe BlazePose in Python now uses pip install mediapipe instead of building from source. Also published a Colab example.
• MediaPipe Iris updated to output a set of 478 3D landmarks, including 468 face landmarks from MediaPipe Face Mesh, with those around the eyes further refined, and 10 additional iris landmarks appended at the end.

MediaPipe Pose

• MediaPipe Pose for upper-body pose tracking
  • Google AI blog post for full and upper body Blazepose
• Preview of Python support that runs MediaPipe Pose in Python interpreter

MediaPipe Iris - Iris tracking and depth estimation

• MediaPipe Iris for iris tracking and single-image depth-from-iris
• Fixed mirrored text rendering in iOS example apps
• Google AI blog post on MediaPipe Iris
• Iris tracking web demo
• Depth estimation from Iris web demo

Automatic provisioning for iOS examples

• Added support for automatic provisioning for building iOS examples
• Fixed sample trace file (#849)

Changes in Z coordinate normalization in 3D landmarks

• TfLiteTensorsToLandmarksCalculator
  • It now always normalizes Z coordinates by image width (as for X) when producing NORM_LANDMARKS output, assuming a weak perspective projection camera model.
  • The normalize_z option can be used to further normalize Z coordinates by an additional factor. For instance, normalize_z: 0.4 is now used in hand tracking to better account for the Z coordinate distribution in the training data.
• Bug fixes
  • Fixes issues on Ubuntu 20.04, resolving #820.
• Documentation update
  • Added instructions for building for Nvidia Jetson and Raspberry Pi devices with ARM Ubuntu.