PND adam teleop

pnd-retarget This framework is used for developing teleoperation of Adam. It currently supports Noitom motion capture and Meta Quest 3S (coming soon) as data input sources.

🤖Retarget demonstration

🎨Architecture

The following is the overall architecture diagram of pnd-retarget.

The Noitom application currently only supports running on Windows computers.
VR (coming soon)
pnd-retarget and pnd-control run on the NUC built into the Adam robot.

teleop diagram

🖥️Environment

The computer built into the Adam robot already has the following environment pre-installed. If you need to run pnd-retarget on another computer, please configure the environment as follows.

Ubuntu 22
Ros2 humble
casadi 3.6.7 (3.6.7, install from source, binary is not compatible with ros2 humble)

sudo apt-get update && sudo apt-get install -y \
    build-essential \
    coinor-libipopt-dev \
    gfortran \
    liblapack-dev \
    pkg-config \
    swig \
    git \
    cmake \
    python3 \
    python3-pip \
    --install-recommends

cd /tmp && \
git clone https://github.com/casadi/casadi.git casadi && \
cd casadi && \
git checkout 3.6.7 && \
mkdir build && \
cd build && \
cmake -DWITH_PYTHON=ON -DWITH_IPOPT=ON -DWITH_OPENMP=ON -DWITH_THREAD=ON .. && \
make -j$(nproc) && \
sudo make install && \
sudo ldconfig

💃Motion Capture

Noitom Motion Capture

Open Axis Studio, connect the sensors, and calibrate the pose. Use the BVH data broadcasting function to send data to the target machine's IP via UDP protocol. For the specific usage tutorial of Noitom, please refer to the video.

The BVH data broadcasting configuration is as follows:

Where Local Address is set to the IP address of the computer running the Axis software, and Target Address is set to the IP address of the Adam robot's computer. The two computers must be on the same local area network. axis broadcast bvh config

VR

Coming soon...

🎯pnd-retarget

Log in to the Adam robot:

ssh pnd-humanoid@xxx.xxx.xxx.xxx

Build

cd ~/Documents
wget https://pndwiki.oss-cn-beijing.aliyuncs.com/sdk_adlskjfas412838_sakjfhrjdsaljf_skfj3jskdjfd32-38439/pnd-retarget-0.5.0.tar.gz
tar -zxf pnd-retarget-0.5.0.tar.gz
cd pnd-retarget-0.5.0
source /opt/ros/humble/setup.bash
colcon build

Run

sudo su
./run.sh [adam_sp|adam_u] # default adam_sp

Preview

Open another terminal on Adam and navigate to the root directory of pnd-retarget:

cd ~/Documents/pnd-retarget-0.5.0
sudo su
./preview.sh

Start Foxglove Studio on your PC and register/login.

Click open connection. Use Foxglove WebSocket to connect to the robot. In ws://localhost:8765, replace localhost with the corresponding robot IP.

Click on the 3D panel, and the panel configuration will appear on the left. Import the 3D panel configuration: copy the following content into the configuration and click Apply. After successful configuration, you will see the effect as shown in the figure below, where the robot follows the motion capture.

三维面板配置

{
  "layers": {
    "845139cb-26bc-40b3-8161-8ab60af4baf5": {
      "visible": true,
      "frameLocked": true,
      "label": "Grid",
      "instanceId": "845139cb-26bc-40b3-8161-8ab60af4baf5",
      "layerId": "foxglove.Grid",
      "size": 10,
      "divisions": 10,
      "lineWidth": 1,
      "color": "#248eff",
      "position": [
        0,
        0,
        0
      ],
      "rotation": [
        0,
        0,
        0
      ],
      "order": 1
    }
  },
  "cameraState": {
    "perspective": true,
    "distance": 8.362406704376102,
    "phi": 78.01200929751727,
    "thetaOffset": -81.68452995867766,
    "targetOffset": [
      -4.304686610834797,
      0.46408895757669916,
      -2.8097074529725844e-16
    ],
    "target": [
      0,
      0,
      0
    ],
    "targetOrientation": [
      0,
      0,
      0,
      1
    ],
    "fovy": 45,
    "near": 0.5,
    "far": 5000
  },
  "followMode": "follow-pose",
  "followTf": "world",
  "scene": {
    "meshUpAxis": "z_up",
    "transforms": {
      "showLabel": false,
      "axisScale": 0
    },
    "ignoreColladaUpAxis": true
  },
  "transforms": {},
  "topics": {
    "/robot_description": {
      "visible": true
    }
  },
  "publish": {
    "type": "point",
    "poseTopic": "/move_base_simple/goal",
    "pointTopic": "/clicked_point",
    "poseEstimateTopic": "/initialpose",
    "poseEstimateXDeviation": 0.5,
    "poseEstimateYDeviation": 0.5,
    "poseEstimateThetaDeviation": 0.26179939
  },
  "imageMode": {}
}

Physical Machine Operation

Physical machine teleoperation is currently developed based on adam_demo, and joystick operations are consistent with adam_demo. In principle, adam_demo functions are fully compatible with teleop_bin (untested).

Special Note

Physical machine teleoperation carries certain risks. Ensure that the area around the robot is clear within 3 meters before connecting. The teleop_bin can only access retarget data when the above pnd-retarget is running correctly and the preview effect is normal. When accessing data, ensure that the joint positions of adam_sp in the preview are roughly the same as those of the physical machine. Do not switch to other adam_demo modes (such as walking, running, action demonstration) during teleoperation.

In case of abnormal robot behavior, immediately perform the following joystick operations: 1.LT+B: Enter stop mode 2.LT+RT: Exit the control program 3.LB+RB: Power off the actuators

Start teleop_bin: sh run.sh
Zeroing (press A)
Stand up (press X)
Press Right to access retarget data while standing
Press Left to disconnect retarget data while standing

xbox

Open a new terminal on Adam:

cd ~/Documents
wget https://pndwiki.oss-cn-beijing.aliyuncs.com/sdk_adlskjfas412838_sakjfhrjdsaljf_skfj3jskdjfd32-38439/teleop_bin_20250424.tar.gz
tar -zxf teleop_bin_20250424.tar.gz
cd teleop_bin
source /opt/ros/humble/setup.bash
sh run.sh

Releases

v0.5.0

Fixed:

adam_u bug fixed.

v0.4.0

Features:

Code optimization.
Performance optimization.

v0.3.0

Features:

Support for adam_u.

v0.2.0

Features:

Added Foxglove support.

v0.1.1

Features:

Initial commit.

FAQ

Foxglove Runs Abnormally in Browser

When opening Foxglove in a browser and encountering the following abnormality, please click Load unsafe scripts.

ssl异常