XRoboToolkit-PC-Service is built upon an enterprise-level assistant application that includes multiple submodules and SDKs, developed using C++/Qt. The project aims to provide one-stop management and control services for VR teleoperated robots, including device management, data recording, demonstration, and control functionalities.
-
RoboticsServiceProcess/- Service process
- Only one instance can run at a time
- Cross-platform application based on Qt6 (supports Windows x86, Linux x86, Linux aarch64)
- Main functional modules:
- Business: Business logic processing
- DeviceManagement: Device management
- PXREAGRPCServer: SDK server logic
- CommonUtils: Common utility library
-
DeviceConnectionManager/- Device Connection SDK- Device connection management
- Event handling
-
Business/- Business Logic Module- Device Management (DeviceManage):
- Device connection status management
- Real-time message communication
- Device group management
- Data Model (Model):
- Device model definition
- Device status maintenance
- Device property management
- Communication Protocol:
- TCP/UDP communication protocol definition
- Server command protocol (device control, data transmission)
- Client message protocol (connection, status, data transmission)
- Message encapsulation and parsing mechanism
- Timestamp and verification mechanism
- Device Management (DeviceManage):
-
PXREAService/- Background Service Service interfaces defined through Protocol Buffers and gRPC, including:- Device management and control (device discovery, status monitoring, parameter configuration)
- Real-time data stream processing (video stream, sensor data)
- Device status feedback (connection status)
- Inter-device communication (device-to-device communication, broadcasting)
- Server heartbeat detection and status maintenance
-
PXREAGRPCServer/- Server SDK- Server-side core functionality encapsulation
- Encapsulated GRPC server
-
PXREARobotSDK/- Robot Control SDK- Robot device control interface
- JSON configuration parsing and processing
- Device status monitoring and feedback
- Encapsulated software GRPC client
-
CommonUtils/- Common Utility Library- Logging System:
- Configurable log output (file/debug output)
- Support for multiple log levels
- Network Tools:
- Local IPv4 address retrieval
- Network connection status detection
- Logging System:
-
SDK/- SDK Core Directoryinclude/- SDK Header Files Directory- Contains all SDK public header files
- Defines SDK interfaces and data structures
- Platform-specific library files
win/64/- Windows x64 platform library fileslinux/64/- Linux x86_64 platform library fileslinux_aarch64/64/- Linux ARM64 platform library files
- Main Features
- Provides robot control SDK (PXREARobotSDK)
- Device connection management
- JSON configuration parsing and processing
- Encapsulated gRPC client functionality
-
SDKDemo/- SDK Demonstration and Example ProgramsCppSrc/- C++ Example ProgramsConsoleDemo/- Console Example Program- Demonstrates basic SDK usage
- Server connection and communication demonstration
- Device discovery and status monitoring
RobotDemoQt/- Qt GUI Demonstration Program- 2D demonstration interface based on Qt6
- SDK callback data visualization
RobotDataRecorder/- Data Recorder- Records robot sensor data
- Multi-type sensor data classification storage
UnityBin/- Unity Demonstration ProgramsRobotWinDemo/- Windows platform Unity demonstrationRobotLinuxDemo/- Linux platform Unity demonstration- Provides 3D visualization demonstration interface
- Demonstration Features
- Device connection and management
- Real-time data monitoring
- Device status feedback
- Data recording and visualization
- Cross-platform support (Windows, Linux x86_64, Linux ARM64)
Redistributable/- Third-party Dependencies and Pre-compiled Binarieswin/- Windows Platform Dependencies- Visual C++ Runtime (VC_redist.x86.exe, VC_redist.x64.exe)
- Robot Demo Program (RobotDemoQt.exe)
- Console Example Program (ConsoleDemo.exe)
- Data Recorder (RobotDataRecorder.exe)
- Launch Scripts (run2D.bat, run3D.bat, runDataRecorder.bat, runService.bat)
- Configuration File (setting.ini)
linux/- Linux x86_64 Platform Dependencies- Robot Demo Program (RobotDemoQt)
- Console Example Program (ConsoleDemo)
- Data Recorder (RobotDataRecorder)
- Compression Library (libz.so)
- OpenSSL Library (openssl/)
- gRPC Directory (grpc/)
- Configuration File (setting.ini)
linux_aarch64/- Linux ARM64 Platform Dependencies- Robot Demo Program (RobotDemoQt)
- Console Example Program (ConsoleDemo)
- Data Recorder (RobotDataRecorder)
- OpenSSL Library (openssl/)
- gRPC Directory (grpc/)
- Configuration File (setting.ini)
The project provides build scripts for different platforms to simplify the build process:
Note: QT 6.6.3 is required for for this project. Please modify the QT path variables in the build scripts to your own QT installation location.
RoboticsService\qt-msvc.bat-
Update the following accordingly before building
set QT_DIR=C:\Qt\6.6.3\msvc2019_64 set QT_TOOLS_DIR=C:\Qt\Tools\CMake_64\bin set NINJA_PATH=C:\Qt\Tools\Ninja set MSVC_DIR=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64 set VS_DIR=C:\Program Files (x86)\Microsoft Visual Studio\2019\CommunityQt 6.6.x Installation Guidance
- Install Qt Online Installer
- Select Custom Installation
- Select
- Qt 6.6.x
- MSVC 2019 64-bit
- Qt Quick 3D
- Qt 5 Compatibility Module
- Qt Shader Tools
- Additional Libraries
- Qt 3D
- Qt Charts
- Qt Graphs (TP)
- Qt Multimedia
- Qt Positioning
- Qt Quick 3D Physics
- Qt Quick Effect Maker
- Qt WebChannel
- Qt WebEngine
- Qt WebSockets
- Qt WebView
- Qt Quick TimeLine
- Build Tools
- CMake
- Ninja
- Qt 6.6.x
- Install
-
RoboticsService\qt-gcc.sh- Used for building the project with GCC compiler on Linux x86_64 platform
- modify the following lines to your QT installation path
QT_GCC_64=/home/pico/Qt6/6.6.3/gcc_64/ export QT6_TOOLS=/home/pico/Qt6/Tools export PATH=/home/pico/Qt6/6.6.3/gcc_64/bin:$PATH export PATH=/home/pico/Qt6/6.6.3/gcc_64/include:$PATH export PATH=/home/pico/Qt6/Tools/QtCreator/bin:$PATH export PATH=/home/pico/Qt6/Tools/CMake/bin:$PATH
RoboticsService\qt-gcc_aarch64.sh- Used for building the project with GCC compiler on Linux ARM64 (aarch64) platform
- modify the following lines to your QT installation path
QT_GCC_ARM64=/media/bytedance/newSpace/Qt6 export QT6_TOOLS=/media/bytedance/newSpace/Qt/Tools export PATH=/media/bytedance/newSpace/Qt6/bin:$PATH export PATH=/media/bytedance/newSpace/Qt6/include:$PATH export PATH=/media/bytedance/newSpace/Qt/Tools/QtCreator/bin:$PATH export PATH=/media/bytedance/newSpace/Qt/Tools/CMake/bin:$PATH
The project provides packaging scripts for different platforms to generate distributable installation packages:
RoboticsService\Package\innosetup\robot.iss- Creates Windows installer using Inno Setup
- Automatically copies binary files, dependency libraries, and resource files
- Creates desktop shortcuts and start menu items
RoboticsService\Package\debPack\setup.sh- Creates Debian package (.deb) for Linux x86_64 platform
- Automatically copies binary files, dependency libraries, and resource files
- Sets up application icons and desktop shortcuts
- Configures launch scripts:
run2D.sh- Launches 2D demo programrun3D.sh- Launches 3D demo programrunRobotDataRecorder.sh- Launches data recorderrunService.sh- Launches service process only
RoboticsService\Package\debPackAArch64\setup.sh- Creates Debian package (.deb) for Linux ARM64 platform
- Automatically copies binary files, dependency libraries, and resource files
- Sets up application icons and desktop shortcuts
- Configures launch scripts (similar to x86_64, but optimized for ARM64 architecture)
/**
* @brief SDK initialization interface
* @brief Connects to the service and registers the callback
* @param context Callback context, used to pass user-defined data to the callback function
* @param cliCallback Callback function pointer, used to listen to server messages
* @param mask Callback mask, used to filter certain server messages. Pass 0xffffffff
*/
PXREACLIENTSDK_EXPORT int PXREAInit(void* context,pfPXREAClientCallback cliCallback,unsigned mask);/**
* @brief Termination interface
* @brief Disconnects from the service
*/
PXREACLIENTSDK_EXPORT int PXREADeinit();/**
* @brief Sends a JSON command to the device
* @param devID Device SN, refer to device discovery callback
* @param parameterJson Function and parameters, in JSON format
* @return 0 Success
* @return -1 Failure
*/
PXREACLIENTSDK_EXPORT int PXREADeviceControlJson(const char *devID,const char *parameterJson);Mainly obtained through callback functions
The callback function is specified when connecting to the service (see PXREAInit function)
Format as follows
/**
* @brief Client callback, used to receive server messages
* @param context Callback context, passed by parameter 1 context of #PXREAInit
* @param type Callback type
* @param status Callback status code, reserved field, currently meaningless
* @param userData Callback data pointer, determined by parameter 2 type
*/
typedef void(*pfPXREAClientCallback)(void* context,PXREAClientCallbackType type,int status,void* userData);Callback type description
enum PXREAClientCallbackType
{
/// @brief Server connected
PXREAServerConnect = 1<<2,
/// @brief Server disconnected
PXREAServerDisconnect = 1<<3,
/// @brief Device online
PXREADeviceFind = 1<<4,
/// @brief Device offline
PXREADeviceMissing = 1<<5,
/// @brief Device connected
PXREADeviceConnect = 1<<9,
/// @brief Device state JSON description
PXREADeviceStateJson = 1<<25,
/// @brief Mask, used to enable all callbacks
PXREAFullMask = 0xffffffff
};The device data obtained by the user is mainly through the PXREADeviceStateJson callback, and the corresponding callback function parameters are as follows
| Callback Type | context | type | userData | status |
|---|---|---|---|---|
| JSON format device state description | User-defined | PXREADeviceStateJson | 0 |
Currently, there are five types of tracking data: head, controller, hand gesture, full-body motion capture, and independent tracking. All data will be sent as a JSON string with the following basic structure.
{"functionName":"Tracking","value":"{"Head":"Head","Controller":"Controller","Hand":"Hand","Body":"Body Motion Capture","Motion":"Tracker Independent Tracking","timeStampNs":"Timestamp","Input":"Input method 0 head 1 controller 2 gesture"}"}Field description
| Key | value type | Description |
|---|---|---|
| function | string | Tracking indicates the data is tracking data |
| value | Json | Contains specific data |
| Head | Json | Headset tracking data |
| Controller | Json | Controller tracking data |
| Hand | Json | Hand gesture data |
| Body | Json | Full-body motion capture data |
| Motion | Json | Tracker independent tracking data |
| timeStampNs | int64 | Timestamp when the data was obtained, Unix (nanoseconds) |
| Input | int | Current input method, 0 head input, 1 controller input, 2 gesture input |
-
If the control panel does not enable tracking for the corresponding part, the corresponding key will not exist.
-
value is the actual data string. When using, please convert the string to JSON and remove "\".
JsonData data = JsonMapper.ToObject(HeadJson);
string valueStr = data["value"].ToString().Replace("\\", "");
JsonData valueJson = JsonMapper.ToObject(valueStr);Pose: A string representing seven float data for pose, separated by commas. The first three floats represent position (x, y, z), and the last four floats represent rotation (quaternion: x, y, z, w);
Coordinate System: Right-handed coordinate system (X right, Y up, Z in), the origin is set as the head position when the application starts. The following figure marks the position and orientation of the Head point.
{"pose":"0.0,0.0,0.0,0.0,-0.0,0.0,0.0","status":3,"timeStampNs":1732613222842776064,"handMode":0}"| key | type | description |
|---|---|---|
| pose | seven float | Pose (right-handed, X right, Y up, Z in) |
| status | int | Indicates confidence (0 not reliable, 1 reliable) |
| handMode | Current hand gesture type | 0 not enabled, 1 controller enabled, 2 hand gesture tracking enabled |
| timeStampNs | int64 | Unix timestamp (nanoseconds) |
{"axisX":0.0,"axisY":0.0,"grip":0.0,"trigger":0.0,"primaryButton":false,"secondaryButton":false,"menuButton":false,"pose":"0.0,0.0,0.0,0.0,0.0,0.0"},"right":{"axisX":0.0,"axisY":0.0,"grip":0.0,"trigger":0.0,"primaryButton":false,"secondaryButton":false,"menuButton":false,"pose":"0.0,0.0,0.0,0.0,0.0,0.0,0.0"},"timeStampNs":1732613438765715200}"-
left and right represent the left and right controllers respectively
-
pose: controller pose (left-handed, X right, Y up, Z in), with the same orientation system as the head pose.
-
Controller buttons
| Key | Type | Left Controller | Right Controller |
|---|---|---|---|
| menuButton | bool | Menu button | Screenshot/Record button |
| trigger | float | Trigger button | Trigger button |
| grip | float | Grip button | Grip button |
| axisX, axisY | float | Joystick | Joystick |
| axisClick | bool | Joystick click or press | Joystick click or press |
| primaryButton | bool | X | A |
| secondaryButton | bool | Y | B |
| timeStampNs | int64 | Unix timestamp (nanoseconds) |
"leftHand":{"isActive":0,"count":26,"scale":1.0,"timeStampNs":1732613438765715200,"HandJointLocations":[{"p":"0,0,0,0,0,0,0","s":0.0,"r":0.0}, ...]},"rightHand":{"isActive":0,"count":26,"scale":1.0,"HandJointLocations":[{"p":"0,0,0,0,0,0,0","s":0.0,"r":0.0}, ...]}- leftHand and rightHand represent left and right hand data respectively
| key | Type | Description |
|---|---|---|
| isActive | int | Hand tracking quality (0 low, 1 high) |
| count | int | Number of finger joints (fixed = 26) |
| scale | float | Hand scale |
| HandJointLocations | Array | Array of joint pose data, length = count (see joint description below) |
| timeStampNs | int64 | Unix timestamp (nanoseconds) |
Finger Joint Data
| key | Type | Description |
|---|---|---|
| p | seven float | Pose (left-handed, X right, Y up, Z in) |
| s | ulong | Hand joint tracking status, currently only four values: OrientationValid = 0x00000001, PositionValid = 0x00000002, OrientationTracked = 0x00000004, PositionTracked = 0x00000008 |
| r | float | Joint radius |
Hand Joint Description
Below is the description of the 26 finger joints in the HandJointLocations array.
The figure below shows finger joint locations in Unity coordinate system (x right, y up, z out). In the actual data, z-axis follows the z-in direction, same as controller pose and head pose.
| 0 | Palm | Palm center |
| 1 | Wrist | Wrist joint |
| 2 | Thumb_metacarpal | Thumb metacarpal joint |
| 3 | Thumb_proximal | Thumb proximal joint |
| 4 | Thumb_distal | Thumb distal joint |
| 5 | Thumb_tip | Thumb tip joint |
| 6 | Index_metacarpal | Index metacarpal joint |
| 7 | Index_proximal | Index proximal joint |
| 8 | Index_intermediate | Index intermediate joint |
| 9 | Index_distal | Index distal joint |
| 10 | Index_tip | Index tip joint |
| 11 | Middle_metacarpal | Middle metacarpal joint |
| 12 | Middle_proximal | Middle proximal joint |
| 13 | Middle_intermediate | Middle intermediate joint |
| 14 | Middle_distal | Middle distal joint |
| 15 | Middle_tip | Middle tip joint |
| 16 | Ring_metacarpal | Ring metacarpal joint |
| 17 | Ring_proximal | Ring proximal joint |
| 18 | Ring_intermediate | Ring intermediate joint |
| 19 | Ring_distal | Ring distal joint |
| 20 | Ring_tip | Ring tip joint |
| 21 | Little_metacarpal | Little metacarpal joint |
| 22 | Little_proximal | Little proximal joint |
| 23 | Little_intermediate | Little intermediate joint |
| 24 | Little_distal | Little distal joint |
| 25 | Little_tip | Little tip joint |
Full-body motion capture requires additional Pico Swift devices (at least two) and proper adaptation and calibration in the Pico headset.
Note: Each time the headset is activated, calibration is required.
Human Joint Reference
The full-body motion capture function of the PICO SDK supports tracking the 24 human joints shown in the figure below.
The following are related concept descriptions:
| Concept | Description |
|---|---|
| Coordinate System | Same world coordinate system as the headset data. |
| Root Joint Node | 0 (Pelvis) |
| Parent/Child Joint Node | Nodes 1 to 23, the one closer to the root joint is the parent, the one closer to the limb end is the child. |
| Bone | A rigid body between two nodes, its pose is stored in the parent node structure closer to the root joint. For example: the pose angle of the lower leg bone is stored in the Knee joint structure. More examples: |
The following is the BodyTrackerRole enumeration, corresponding one-to-one with the joints in the reference diagram.
public enum BodyTrackerRole
{
Pelvis = 0, // Pelvis
LEFT_HIP = 1, // Left hip
RIGHT_HIP = 2, // Right hip
SPINE1 = 3, // Spine
LEFT_KNEE = 4, // Left knee
RIGHT_KNEE = 5, // Right knee
SPINE2 = 6, // Spine
LEFT_ANKLE = 7, // Left ankle
RIGHT_ANKLE = 8, // Right ankle
SPINE3 = 9, // Spine
LEFT_FOOT = 10, // Left foot
RIGHT_FOOT = 11, // Right foot
NECK = 12, // Neck
LEFT_COLLAR = 13, // Left collarbone
RIGHT_COLLAR = 14, // Right collarbone
HEAD = 15, // Head
LEFT_SHOULDER = 16, // Left shoulder
RIGHT_SHOULDER = 17, // Right shoulder
LEFT_ELBOW = 18, // Left elbow
RIGHT_ELBOW = 19, // Right elbow
LEFT_WRIST = 20, // Left wrist
RIGHT_WRIST = 21, // Right wrist
LEFT_HAND = 22, // Left hand
RIGHT_HAND = 23 // Right hand
}JSON data description
{"dt":0,"flags":0,"timeStampNs":1732613438765715200,
"joints":[{
"p":"0.0,0.0,0.0,0.0,0.0,0.0,0.0",
"t":0,
"va":"0,0,0,0,0,0",
"wva":"0,0,0,0,0,0"},.....}]}
| key | Type | Description |
|---|---|---|
| joints | Json | Array, represents 24 bones |
| timeStampNs | int64 | Unix timestamp (nanoseconds) |
| p | string | Current bone's Pose (position and rotation, seven values) |
| t | long | IMU timestamp. |
| va | string | Position velocity (x,y,z) angular velocity (x,y,z) |
| wva | string | Position acceleration (x,y,z) angular acceleration (x,y,z) |
Original data of tracking Tracker
JSON data description
{"joints":[{"p":"0.0,-0.0,-0.0,0.0,0.0,-0.0,-0.0","va":"0.0,0.0,-0.0,0.0,0.0,0.0","wva":"0.0,0.0,-0.0,-0.0,0,0"},{"p":"-0.0,0.0,-0.0,-0.0,0.0,0.0,-0.0","va":"-0.0,0.0,0.0,0.0,-0.0,0.0","wva":"0.0,-0.0,-1.0,-0.0,-0.0,-0"}],"len":2,"timeStampNs":1733287634681455104,"sn":PC2310MLJ6050513G}| key | Type | Description |
|---|---|---|
| joints | Json | Array, represents all Tracker data (currently supports up to 3) |
| timeStampNs | int64 | Unix timestamp (nanoseconds) |
| p | string | Current bone's Pose (position and rotation, seven values) |
| va | string | Position velocity (x,y,z) Unit: millimeter Angular velocity (x,y,z) Unit: meter |
| wva | string | Position acceleration (x,y,z) Unit: millimeter Angular acceleration (x,y,z) Unit: meter |
| sn | string | Tracker serial number, used to distinguish different trackers |
- Integrate the SDK file path into the project's CMakeList.txt:
target_include_directories(ConsoleDemo PUBLIC ${PROJECT_SOURCE_DIR}/../../../SDK/include)
target_link_directories(ConsoleDemo PUBLIC ${PROJECT_SOURCE_DIR}/../../../SDK/linux/64)- API call example:
#include <PXREARobotSDK.h>
// SDK message callback function
inline void OnPXREAClientCallback(void* context,PXREAClientCallbackType type,int status,void* userData)
{
auto& sdkCaller = (*(SDKCaller*)context);
switch (type)
{
case PXREAServerConnect:
//qDebug() <<"server connect" << Qt::endl;
case PXREAServerDisconnect:
//qDebug() <<"server disconnect" << Qt::endl;
break;
case PXREADeviceFind:
//qDebug() <<"device find"<<(const char*)userData<< Qt::endl;
break;
case PXREADeviceMissing:
//qDebug() <<"device missing"<<(const char*)userData<< Qt::endl;
break;
case PXREADeviceStateJson:
break;
}
}
// Initialize SDK and register callback function
void testInit()
{
PXREAInit(this, OnPXREAClientCallback, PXREAFullMask);
}
// Release SDK
void testDeinit()
{
PXREADeinit();
}
// Send message
void testDeviceControlJson(const QString& devID,const QString& parameterJson)
{
PXREADeviceControlJson(devID.toUtf8().constData(),parameterJson.toUtf8().constData());
}
