Media Pipe Pose Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. Install and Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install mediapipe opencv-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This line installs the required packages for the tutorial, specifically mediapipe (which is a library for hand, pose, and face detection) and opencv-python (a library for computer vision tasks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import mediapipe as mp\n",
    "import numpy as np\n",
    "mp_drawing = mp.solutions.drawing_utils\n",
    "mp_pose = mp.solutions.pose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the necessary libraries are imported.\n",
    "cv2: This is the OpenCV library which is used for various computer vision tasks.\n",
    "mediapipe as mp: Mediapipe is imported with the alias mp.\n",
    "numpy as np: Numpy is imported with the alias np.\n",
    "mp_drawing: This is a utility from Mediapipe to help with drawing on images.\n",
    "mp_pose: This helps in detecting human poses using Mediapipe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# VIDEO FEED\n",
    "cap = cv2.VideoCapture(0)\n",
    "while cap.isOpened():\n",
    "    ret, frame = cap.read()\n",
    "    cv2.imshow('Mediapipe Feed 1', frame)\n",
    "    \n",
    "    if cv2.waitKey(10) & 0xFF == ord('q'):\n",
    "        break\n",
    "        \n",
    "cap.release()\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This block of code sets up a video feed using OpenCV.\n",
    "cv2.VideoCapture(0): This initializes the video capture with the primary webcam of the computer.\n",
    "while cap.isOpened(): Continuously capture frames as long as the webcam is active.\n",
    "ret, frame = cap.read(): Read a frame from the webcam. ret is a boolean that indicates if the frame was successfully grabbed.\n",
    "cv2.imshow('Mediapipe Feed', frame): This displays the captured frame in a window named 'Mediapipe Feed'.\n",
    "if cv2.waitKey(10) & 0xFF == ord('q'): This checks if the 'q' key is pressed. If pressed, it breaks out of the loop to stop capturing.\n",
    "cap.release(): Releases the video capture object.\n",
    "cv2.destroyAllWindows(): Closes all OpenCV windows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Make Detections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cap = cv2.VideoCapture(0)\n",
    "## Setup mediapipe instance\n",
    "with mp_pose.Pose(min_detection_confidence=0.1, min_tracking_confidence=0.1) as pose:\n",
    "    while cap.isOpened():\n",
    "        ret, frame = cap.read()\n",
    "        \n",
    "        # Recolor image to RGB\n",
    "        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "        image.flags.writeable = False\n",
    "      \n",
    "        # Make detection\n",
    "        results = pose.process(image)\n",
    "    \n",
    "        # Recolor back to BGR\n",
    "        image.flags.writeable = True\n",
    "        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n",
    "        \n",
    "        # Render detections\n",
    "        mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS,\n",
    "                                mp_drawing.DrawingSpec(color=(245,117,66), thickness=2, circle_radius=2), \n",
    "                                mp_drawing.DrawingSpec(color=(245,66,230), thickness=2, circle_radius=2) \n",
    "                                 )               \n",
    "        \n",
    "        cv2.imshow('Mediapipe Feed', image)\n",
    "\n",
    "        if cv2.waitKey(10) & 0xFF == ord('q'):\n",
    "            break\n",
    "\n",
    "    cap.release()\n",
    "    cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This block of code captures video from the webcam and processes each frame using Mediapipe to detect human poses.\n",
    "The code inside the with statement initializes the pose detection from Mediapipe.\n",
    "Each frame from the webcam is converted from BGR (Blue, Green, Red - the default format of OpenCV) to RGB (Red, Green, Blue - the format Mediapipe uses).\n",
    "pose.process(image): Detects the pose in the image.\n",
    "The detected landmarks (keypoints of the pose) are then drawn on the image using mp_drawing.draw_landmarks(...).\n",
    "The processed image is then displayed using cv2.imshow(...).\n",
    "The loop continues until the 'q' key is pressed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mp_drawing.DrawingSpec??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "mp_drawing.DrawingSpec??\n",
    "This line is likely an interactive command to inspect the documentation or source code of the DrawingSpec class from Mediapipe. In Jupyter notebooks, appending ?? to a function or class name displays its source code and docstring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Determining Joints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.imgur.com/3j8BPdc.png\" style=\"height:300px\" >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cap = cv2.VideoCapture(0)\n",
    "## Setup mediapipe instance\n",
    "with mp_pose.Pose(min_detection_confidence=0.1, min_tracking_confidence=0.1) as pose:\n",
    "    while cap.isOpened():\n",
    "        ret, frame = cap.read()\n",
    "        \n",
    "        # Recolor image to RGB\n",
    "        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "        image.flags.writeable = False\n",
    "      \n",
    "        # Make detection\n",
    "        results = pose.process(image)\n",
    "    \n",
    "        # Recolor back to BGR\n",
    "        image.flags.writeable = True\n",
    "        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n",
    "        \n",
    "        # Extract landmarks\n",
    "        try:\n",
    "            landmarks = results.pose_landmarks.landmark\n",
    "            print(landmarks)\n",
    "        except:\n",
    "            pass\n",
    "        \n",
    "        \n",
    "        # Render detections\n",
    "        mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS,\n",
    "                                mp_drawing.DrawingSpec(color=(245,117,66), thickness=2, circle_radius=2), \n",
    "                                mp_drawing.DrawingSpec(color=(245,66,230), thickness=2, circle_radius=2) \n",
    "                                 )               \n",
    "        \n",
    "        cv2.imshow('Mediapipe Feed', image)\n",
    "\n",
    "        if cv2.waitKey(10) & 0xFF == ord('q'):\n",
    "            break\n",
    "\n",
    "    cap.release()\n",
    "    cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This block of code is very similar to the one we discussed earlier. The main addition is the extraction of landmarks.\n",
    "The line landmarks = results.pose_landmarks.landmark extracts the pose landmarks (keypoints) detected by Mediapipe. These landmarks represent different parts of the body, like the nose, eyes, shoulders, elbows, etc.\n",
    "The extracted landmarks are then printed to the console.\n",
    "The try and except block is used to handle any potential errors that might arise when extracting landmarks. If there's an error (e.g., no landmarks detected), the code inside the except block is executed, which in this case, does nothing (using pass)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Pose Landmarks in Mediapipe:\n",
    "Mediapipe's Pose solution provides 33 landmarks across the body, from the nose tip to the feet. Each landmark has a unique ID and corresponds to a specific part of the body. For instance:\n",
    "\n",
    "mp_pose.PoseLandmark.LEFT_SHOULDER.value gives the ID for the left shoulder landmark.\n",
    "mp_pose.PoseLandmark.LEFT_ELBOW.value provides the ID for the left elbow landmark.\n",
    "\n",
    "2. Landmark Object:\n",
    "Each landmark object (like the ones in the landmarks list) typically contains:\n",
    "\n",
    "x, y, z: The normalized coordinates of the landmark in the frame. \"Normalized\" means the values are between 0 and 1, with (0, 0) being the top-left corner and (1, 1) being the bottom-right corner of the frame.\n",
    "visibility: A score between 0 and 1 representing the probability that the landmark is visible and not occluded. A higher score indicates higher confidence in visibility.\n",
    "\n",
    "3. Working with Landmarks:\n",
    "To access specific landmarks, you can index into the landmarks list with the appropriate landmark ID. For example:\n",
    "\n",
    "To get the x-coordinate of the left shoulder: landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x\n",
    "To get the visibility score of the left elbow: landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].visibility\n",
    "\n",
    "4. Error Handling with Landmarks:\n",
    "In the code, there's a try and except block when extracting landmarks:\n",
    "\n",
    "try:\n",
    "    landmarks = results.pose_landmarks.landmark\n",
    "    print(landmarks)\n",
    "except:\n",
    "    pass\n",
    "\n",
    "This is important because sometimes Mediapipe might not detect any poses in a frame (maybe due to poor lighting, occlusion, or the person being out of the frame). In such cases, results.pose_landmarks would be None, and trying to access landmark on it would raise an error. The try and except block gracefully handles such scenarios by skipping the processing for that particular frame.\n",
    "\n",
    "5. Visualization of Landmarks:\n",
    "The mp_drawing.draw_landmarks() function is used to draw the detected landmarks and their connections on the image. The function is provided with:\n",
    "\n",
    "The image to draw on.\n",
    "The landmarks to draw.\n",
    "The connections between landmarks (defined by mp_pose.POSE_CONNECTIONS).\n",
    "Drawing specifications, such as color, line thickness, and circle radius.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(landmarks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "len(landmarks)\n",
    "\n",
    "This line simply gets and displays the number of landmarks detected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for lndmrk in mp_pose.PoseLandmark:\n",
    "    print(lndmrk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This block of code prints out all the pose landmarks that Mediapipe can detect. \n",
    " \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].visibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "mp_pose.PoseLandmark is an enumeration of all possible pose landmarks.\n",
    "\n",
    "landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].visibility\n",
    "\n",
    "This line gets the visibility score of the left shoulder landmark.\n",
    "The visibility score indicates the probability of the landmark being visible in the image (not occluded)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value]\n",
    "This line retrieves details about the left elbow landmark, such as its coordinates and visibility score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This line retrieves details about the left elbow landmark, such as its coordinates and visibility score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Calculate Angles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_angle(a,b,c):\n",
    "    a = np.array(a) # First\n",
    "    b = np.array(b) # Mid\n",
    "    c = np.array(c) # End\n",
    "    \n",
    "    radians = np.arctan2(c[1]-b[1], c[0]-b[0]) - np.arctan2(a[1]-b[1], a[0]-b[0])\n",
    "    angle = np.abs(radians*180.0/np.pi)\n",
    "    \n",
    "    if angle >180.0:\n",
    "        angle = 360-angle\n",
    "        \n",
    "    return angle "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]\n",
    "elbow = [landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].x,landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].y]\n",
    "wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shoulder, elbow, wrist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calculate_angle(shoulder, elbow, wrist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuple(np.multiply(elbow, [640, 480]).astype(int))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cap = cv2.VideoCapture(0)\n",
    "## Setup mediapipe instance\n",
    "with mp_pose.Pose(min_detection_confidence=0.5, min_tracking_confidence=0.5) as pose:\n",
    "    while cap.isOpened():\n",
    "        ret, frame = cap.read()\n",
    "        \n",
    "        # Recolor image to RGB\n",
    "        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "        image.flags.writeable = False\n",
    "      \n",
    "        # Make detection\n",
    "        results = pose.process(image)\n",
    "    \n",
    "        # Recolor back to BGR\n",
    "        image.flags.writeable = True\n",
    "        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n",
    "        \n",
    "        # Extract landmarks\n",
    "        try:\n",
    "            landmarks = results.pose_landmarks.landmark\n",
    "            \n",
    "            # Get coordinates\n",
    "            shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]\n",
    "            elbow = [landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].x,landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].y]\n",
    "            wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]\n",
    "            \n",
    "            # Calculate angle\n",
    "            angle = calculate_angle(shoulder, elbow, wrist)\n",
    "            \n",
    "            # Visualize angle\n",
    "            cv2.putText(image, str(angle), \n",
    "                           tuple(np.multiply(elbow, [640, 480]).astype(int)), \n",
    "                           cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2, cv2.LINE_AA\n",
    "                                )\n",
    "                       \n",
    "        except:\n",
    "            pass\n",
    "        \n",
    "        \n",
    "        # Render detections\n",
    "        mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS,\n",
    "                                mp_drawing.DrawingSpec(color=(245,117,66), thickness=2, circle_radius=2), \n",
    "                                mp_drawing.DrawingSpec(color=(245,66,230), thickness=2, circle_radius=2) \n",
    "                                 )               \n",
    "        \n",
    "        cv2.imshow('Mediapipe Feed', image)\n",
    "\n",
    "        if cv2.waitKey(10) & 0xFF == ord('q'):\n",
    "            break\n",
    "\n",
    "    cap.release()\n",
    "    cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Curl Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cap = cv2.VideoCapture(0)\n",
    "\n",
    "# Curl counter variables\n",
    "counter = 0 \n",
    "stage = None\n",
    "\n",
    "## Setup mediapipe instance\n",
    "with mp_pose.Pose(min_detection_confidence=0.5, min_tracking_confidence=0.5) as pose:\n",
    "    while cap.isOpened():\n",
    "        ret, frame = cap.read()\n",
    "        \n",
    "        # Recolor image to RGB\n",
    "        image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "        image.flags.writeable = False\n",
    "      \n",
    "        # Make detection\n",
    "        results = pose.process(image)\n",
    "    \n",
    "        # Recolor back to BGR\n",
    "        image.flags.writeable = True\n",
    "        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n",
    "        \n",
    "        # Extract landmarks\n",
    "        try:\n",
    "            landmarks = results.pose_landmarks.landmark\n",
    "            \n",
    "            # Get coordinates\n",
    "            shoulder = [landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].x,landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value].y]\n",
    "            elbow = [landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].x,landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value].y]\n",
    "            wrist = [landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].x,landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value].y]\n",
    "            \n",
    "            # Calculate angle\n",
    "            angle = calculate_angle(shoulder, elbow, wrist)\n",
    "            \n",
    "            # Visualize angle\n",
    "            cv2.putText(image, str(angle), \n",
    "                           tuple(np.multiply(elbow, [640, 480]).astype(int)), \n",
    "                           cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2, cv2.LINE_AA\n",
    "                                )\n",
    "            \n",
    "            # Curl counter logic\n",
    "            if angle > 160:\n",
    "                stage = \"down\"\n",
    "            if angle < 30 and stage =='down':\n",
    "                stage=\"up\"\n",
    "                counter +=1\n",
    "                print(counter)\n",
    "                       \n",
    "        except:\n",
    "            pass\n",
    "        \n",
    "        # Render curl counter\n",
    "        # Setup status box\n",
    "        cv2.rectangle(image, (0,0), (225,73), (245,117,16), -1)\n",
    "        \n",
    "        # Rep data\n",
    "        cv2.putText(image, 'REPS', (15,12), \n",
    "                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,0,0), 1, cv2.LINE_AA)\n",
    "        cv2.putText(image, str(counter), \n",
    "                    (10,60), \n",
    "                    cv2.FONT_HERSHEY_SIMPLEX, 2, (255,255,255), 2, cv2.LINE_AA)\n",
    "        \n",
    "        # Stage data\n",
    "        cv2.putText(image, 'STAGE', (65,12), \n",
    "                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,0,0), 1, cv2.LINE_AA)\n",
    "        cv2.putText(image, stage, \n",
    "                    (60,60), \n",
    "                    cv2.FONT_HERSHEY_SIMPLEX, 2, (255,255,255), 2, cv2.LINE_AA)\n",
    "        \n",
    "        \n",
    "        # Render detections\n",
    "        mp_drawing.draw_landmarks(image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS,\n",
    "                                mp_drawing.DrawingSpec(color=(245,117,66), thickness=2, circle_radius=2), \n",
    "                                mp_drawing.DrawingSpec(color=(245,66,230), thickness=2, circle_radius=2) \n",
    "                                 )               \n",
    "        \n",
    "        cv2.imshow('Mediapipe Feed', image)\n",
    "\n",
    "        if cv2.waitKey(10) & 0xFF == ord('q'):\n",
    "            break\n",
    "\n",
    "    cap.release()\n",
    "    cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}