Working on Stretch3 AI, implementing state-of-the-art RL methods such as Diffusion Policy, VQ-BeT, Pi_0, and Deep Perception (closed-loop and open-loop). The focus is on robot teleoperation, policy learning, and perception models.
- Collecting Demonstration Data for various tasks: push button, press elevator button, open the door, fill the water bottle.
- Training the robot using Diffusion Policy
The dataset is uploaded to 🤗 Stretch3 Demodata.
- Familiarized with Stretch3’s ROS2 interface (testing/debugging tools).
- Implemented:
- Keyboard teleoperation
- Perception (point-cloud from 2D images)
- Depth-map generation (gripper camera & sensors)
- Debugged StretchAI installation (NVIDIA drivers, CUDA, dependencies).
- Set up StretchAI on GPU and robot.
- Verified deep perception models (face/object detection).
- Started working on
ai_pickup(language-directed pick & place feature).
- Implemented custom pickup functions for identifying and navigating to a Red Button.
- Explored how StretchAI agent generates robot instructions from natural language.
- Began debugging button press navigation errors.
- Addressed object detection issues with Detic + CLIP.
- Investigated grasping methods:
- AnyGrasp (IP protected)
- Opensource alternatives: GraspNet, Graspness
- Developed two methods for pressing the Red Button:
- Open-loop approach: Fixed execution steps.
- Closed-loop approach: Uses Detic + CLIP for adaptive behavior.
- Started working on Dexterous Teleop Kit for training RL policies.
- Built Dexterous Teleop Kit for RL policy training.
- Studied the Diffusion Policy paper.
- Finished building teleop kit.
- Started analyzing Diffusion Policy code:
- Available for LeRobot, but not for Stretch3.
- Began adapting LeRobot’s implementation for Stretch3.
- Set up Dex-teleop on PC & robot.
- Migrated from Docker to Mamba virtual environment.
- Understood Diffusion Policy’s noising-denoising process.
- Began robot teleoperation data collection.
- Calibrated Logitech-Webcam-C930e for Dex Teleop.
- Finished studying Diffusion Policy (presented findings to Prof. Zhi).
- Started implementing Diffusion Policy for Stretch3.
- Collected human demonstration data for "Press Button" task.
- Collecting 30+ teleop demonstrations for policy training.
- Implementing Diffusion Policy adaptation for Stretch3 based on LeRobot’s implementation.
- Diffusion Policy: Implementing & training visuomotor policies.
- VQ-BeT: Exploring state-of-the-art RL methods.
- Dexterous Teleoperation: Data collection & policy training.
- Deep Perception: Improving robot awareness in both open-loop and closed-loop systems.
This research aims to advance robot manipulation and learning by integrating cutting-edge reinforcement learning techniques with natural language-driven control.