A ROS package that implements EMG-controlled Franka robot grasping for object manipulation to be presented at automatica 2025.

Designed for and tested on Ubuntu 22.04 LTS, ROS (2) Humble with Python 3.10.

Note: The package is in an initial development phase. It is unstable and may significantly change in concept and implementation.

Author: Ahmed Abdelrahman

Contents

• ➤ Overview
• ➤ Main Components
  • EMG Classifier
  • Robot Motion
  • Prensilia Hand Control
• ➤ Installation
• ➤ Usage
• ➤ Directory Structure
• ➤ Dependencies
• ➤ Future Plans

Overview

franka-emg-grasping implements the functionalities required to deploy an EMG-based motion intent classifier to drive a Franka Panda's grasping actions in simple manipulation tasks. It primarily consists of:

• a Python package containing the EMG classification and other tools
• a ROS package containing a motion interface for the robot and other periphery control and communication nodes

We currently target object pick-and-place and cloth folding tasks and include support for either i) a standard Franka Hand or ii) an anthropomorphic Prensilia IH2 Azzurra hand as end-effectors.

Note: The main motion node expects EMG classification results through UDP messages. This is currently necessary because data collection from the EMG sensors that we use (Delsys) must be done on a (separate) Windows machine.

Example Applications: Pick-and-Place and Cloth Folding

Main Components

EMG Classifier

The emg_classification Python module implements an EMGClassifier which loads pre-trained pattern recognition models of several types that are trained on EMG features to classify live signals into a discrete set of classes.

Currently supported supervised machine learning models include:

• K-neareast neighbours (KNN)
• Random Forests (RF)
• Multi-layer Perceptrons (MLPs)

The input to the models is a set of standard time-based and frequency-based features that are extracted from a fixed window of a 4-channel EMG signal, which include:

• Mean Absolute Value (MAV)
• Zero Crossings (ZC))
• Waveform Length (WL)
• Root Mean Square (RMS)
• Skewness
• Mean Frequency (MNF)
• Median Frequency (MDF)
• Mean Power (MNP)
• Peak Frequency (PKF)
• Variance of Central Frequency (VCF)

At the moment, we train models to recognize 2 or 3 hand actions:

• 2-class: Grasping, Relaxing/Opening
• 3-class: Grasping, Relaxing, Opening

Robot Motion

The motion generation node implements the key functionalities of:

• sending joint space motion commands to the robot, using Moveit's MoveGroup interface
• grasping and opening (i.e. ungrasping), using a Franka or Prensilia hand
• implementing sequences of fixed motions
• awaiting for signals to grasp/open for multiple sources: EMG pattern recognition model, a gamepad 'joy' command, or a terminal command.

The node currently implements the following motion sequences:

1. Franka hand object pick-and-place
2. Franka hand cloth folding
3. Prensilia hand object pick-and-place
4. Prensilia hand cloth folding
5. Prensilia hand handkerchief folding

UDP Interface

In its current implementation, the node receives classification results through UDP messages that are sent by a UDP listener node, which in turn listens to these results from a separate component that runs the data collection and EMG classifier. This component is based on a fork of the Delsys EMG sensors' API, which is not included in this package.

Prensilia Hand Control

To enable the control and integration of the Prensilia hand, the package contains a control module and a ROS node. Together, these enable sending commands to e.g. fully open, reset, or execute several grasps, including variants of tri-grasps and cylindrical grasps.

Installation

Set Up Dependencies

franka-emg-grasping depends on a custom version of the franka_ros2 repository (hosted on GitLab and not open-source at the time of writing), which differs from the upstream version in:

• being modified to support the older Franka Panda robot (not just the FR3)
• including a MuJoCo simulation (for testing)
• adding a Moveit interface for the simulated MuJoCo robot (for testing)

If this package is made available to the public, its installation instructions will be added in the future.

In turn, that package's Moveit interface (which is used here to execute robot motions) requires the humble versions of the following moveit packages to be built from source in the same ROS2 workspace: moveit2_tutorials, moveit_task_constructor, py_binding_tools.

Assuming you are in a ROS2 workspace's src folder, clone each repository and check out the appropriate branch as follows:

git clone https://github.com/moveit/moveit2_tutorials && cd moveit2_tutorials && git checkout humble && cd ..
git clone https://github.com/moveit/moveit_task_constructor.git && cd moveit_task_constructor && git checkout humble && cd ..
git clone https://github.com/moveit/py_binding_tools.git && cd py_binding_tools && git checkout ros2 && cd ..

Build Package

After cloning this repository in your workspace, build the package using:

colcon build --packages-select franka_emg_grasping

Usage

Ensure that the Franka robot bring-up has been launched, then run the main node using:

ros2 launch franka_emg_grasping moveit_panda_motion_generator.launch.py velocity_factor:=0.8 acceleration_factor:=0.9 sim:=False auto_grasp:=True

Note on arguments:

• sim configures parameters for a simulated robot when set to True
• If autograsp is set to True, the robot automatically performs grasping actions during motion sequences. Otherwise, it awaits a grasping signal (from a trigger topic, EMG signal, or joy command).

Start the Prensilia control node:

ros2 launch franka_emg_grasping prensilia_control.launch.py serial_port:=/dev/ttyUSB0

Start the UDP listener:

ros2 launch franka_emg_grasping udp_listener_py.launch.py

To start a motion sequence, publish a message that specifies a sequence ID:

ros2 topic pub --once /panda_motion_generator/motion_trigger std_msgs/msg/Int8 "{'data': 1}"

Optional: joy Commands

To enable joy commands from a gamepad (e.g. Xbox controller), also run the joy node:

ros2 run joy joy_node

The following commands can be sent using the respective `joy` button combinations:

RT + Right Button    : Start Prensilia Pick-and-Place Sequence
RT + Left Button     : Start Prensilia Cloth Folding Sequence
RT + Up Button       : Start Prensilia Handkerchief Folding Sequence
LT + X               : Grasp (Prensilia)
LT + B               : Open (Prensilia)
LT + A               : Grasp (Franka Hand)
LT + Y               : Open (Franka Hand)
LT + Xbox Button     : Go Home

Optional: Terminal Commands

To execute a grasp with the Franka hand:

ros2 topic pub --once /panda_motion_generator/grasp_trigger std_msgs/msg/Float32 "{'data': 0.05}"

To open the Franka hand:

ros2 topic pub --once /panda_motion_generator/open_trigger std_msgs/msg/Bool "{'data': 1}"

To execute a tri-grasp with the Prensilia hand:

ros2 topic pub --once /prensilia_control_node/action_command std_msgs/msg/String "{'data': 'tri_grasp'}"

To open the Prensilia hand:

ros2 topic pub --once /prensilia_control_node/action_command std_msgs/msg/String "{'data': 'open'}"

To execute a more gradual opening of the Prensilia hand from a tri-grasp position (for more delicate object release):

ros2 topic pub --once /prensilia_control_node/action_command std_msgs/msg/String "{'data': gradual_open_tri_grasp_objects}"

Directory Structure

Package Files

franka-emg-grasping
│
├── franka_emg_grasping
│   ├── __init__.py
│   ├── dataset.py
│   ├── emg_classification.py
│   ├── feature_extraction.py
│   ├── prensilia_hand_control.py
│   └── visualization.py
│
├── ros
│   └── src
│   |   └── moveit_panda_motion_generator_node.cpp
│   └── scripts
│   |   ├── prensilia_control_node.py
│   |   └── udp_listener_node_py.py
│   └── launch
│       ├── moveit_panda_motion_generator.launch.py
│       ├── prensilia_control.launch.py
│       └── udp_listener_py.launch.py
│
├── models/
├── scripts/
├── media/
├── dev/
├── setup.py
├── CMakeLists.txt
├── package.xml
├── README.md
└── LICENSE

Dependencies

External repositories:

• This fork of Delsys Python API

Python:

• matplotlib
• numpy
• pandas
• scipy
• scikit-learn

ROS:

• rclpy
• rclcpp
• ament_cmake
• ament_cmake_python
• geometry_msgs
• std_msgs
• sensor_msgs
• franka_msgs
• moveit
• moveit_ros_planning_interface
• moveit_ros_move_group

Future Plans

• Add support for Franka FR3 robot