LFD Apples - Gripper Control

Things You'll Need

• Robot: Franka Research
• Gripper: micro-ROS enabled
• ROS2
• Moveit2

---

GRIPPER

lfd_gripper.launch.py

This launch file performs the following actions:

1. Turns your laptop into a Wi-Fi hotspot.
2. Runs microROS agent to handle the communication with the ESP32 on the gripper's side.
3. Runs a ROS2 node to control the air valve, and fingers. It reduces the demonstrator's overhead by automatically switching the air ON/OFF, and CLOSE/OPEN the fingers.
4. Runs a node to publish the In-Hand camera image.
5. Runs a node to publish the Fixed camera image

ros2 launch lfd_apples lfd_gripper.launch.py ssid:=my_hotspot password:=mypassword palm_camera_device_num:=4 fixed_camera_device_num:=5

Notes:

• Replace wlp108s0f0 in the launch file with your wireless interface name. You can check this with nmcli device status or ip link.
• Use the ssid / password that were previously uploaded to the ESP32 board. By default these values are alejos / harvesting.
• Check the camera device number with v4l2-ctl --list-devices.

ARM

1. Recording Demonstrations

This step is meant for recording the trajectory of the arm as the human drives it. It requires the following nodes to be running:

listen_franka.py node

This node subscribes to the arm's and gripper's ros2 topics and saves them in a bagfile

ros2 run lfd_apples listen_franka

Free drive node

I use this example to simply free drive the arm while doing the demo.

ros2 launch franka_bringup example.launch.py controller_name:=move_to_start_example_controller

2. Replay Demonstrations

This step is to make the robot replay the demonstration without human intervention. Hence, wrench topic is cleaner and free from human introduced forces while driving the arm. All topics are saved in a new bagfile, which is now a 'pristine' demonstration.

Move it node

This node is to use move it as the robot controller

ros2 launch franka_fr3_moveit_config moveit.launch.py robot_ip:=192.168.1.11

lfd_replay.py node

This node is reads the joint positions from the human demonstration, sends it to the arm controller.

ros2 run lfd_apples lfd_replay

listen_franka.py node

This node subscribes to the arm's and gripper's ros2 topics and saves them in a bagfile

ros2 run lfd_apples listen_franka

TROUBLESHOOTING

Franka Arm

Failed to lock the realtime publisher issue
solution 1: edit franka_robot_broadcaster.hpp by adding the following right before including the realtime_publisher (near line 24):

#define NON_POLLING TRUE

Controllers

By default, moveit controllers do not have gravity_compensation_example_controller.

To fix this simply replace the file fr3_ros_controllers.yaml with the one in this repo at config/fr3_ros_controllers.yaml.

FILES DESCRIPTION

I. Scripts for Data Collection

• lfd_automatic_gripper.py
• lfd_data_collection.py
• lfd_inhand_camera.py

II. Scripts for Data Pre-processing

• lfd_data_preprocessing.py
• lfd_vision.py

III. Scripts for Learning

• lfd_learning.py
• lfd_lstm.py

IV. Scripts for Implementation

• lfd_implementation.py
• yolo_latent_node.py

V. Scripts for Data Analysis

• lfd_metadata_analysis.py
• lfd_data_visualization.py
• lfd_figures_for_papers.py