Incorrect Color Map and Segmentation IDs not applied correctly for meshes via `simSetSegmentationObjectID`

[ssaigarimella]

1. What happened (actual behavior)

• I call simSetSegmentationObjectID(name, 200, false) on each of ~19 BP_SplineHuman_* objects.
• The Python client prints OK for every call and “seg_id=200, color=RGB(200,0,0)”.
• When I fetch the segmentation image (1920×1080), every human remains a neutral gray (≈84,84,84).
• Querying back with simGetSegmentationObjectID(name, false) shows each human still has various old IDs (e.g. 95, 127, 191, 255), not 200.
• Attempting a full regex-based set (simSetSegmentationObjectID(".*BP_SplineHuman.*",200,true)) either crashes Unreal or is ignored.

2. What I expected (expected behavior)

• After assigning ID=200 to all BP_SplineHuman instances, every human should render in pure red (200,0,0) in the segmentation image.
• simGetSegmentationObjectID(name, false) should return 200 for each name immediately after setting.
• No crash when using the regex-based call.

3. Steps to reproduce

1. Launch UE5 with the Cosys-AirSim plugin.
2. Verify your settings.json enables segmentation with fixed IDs (see below).
3. Run this client script immediately after connection:

import setup_path                  
import cosysairsim as airsim
import numpy as np
import csv
import cv2
import re

# ---- user params ----
PORT          = 41451
CAMERA_NAME   = "front_center"
VEHICLE_NAME  = ""  # empty for default / single-vehicle setups

# where to dump / load the mesh-color CSV
CSV_FILENAME      = "instance_segmentation_colormap.csv"
# where your UE label vs. mesh-name CSV lives
UE_LABEL_CSV_PATH = "ue_label_vs_name.csv"

# if your raw segmentation image appears upside-down, set to True
FLIP_VERTICAL = False

# keywords to look for in the human-readable labels
KEYWORDS = ("human", "car", "truck", "sedan", "suv", "vehicle")

# regex for your spline-humans
REGEX = ".*BP_SplineHuman.*"

# --------------------------------

def seg_id_to_rgb(seg_id):
    # AirSim maps seg_id→(R,G,B) as (seg_id,0,0)
    return (seg_id, 0, 0)

# 1. connect
client = airsim.MultirotorClient(port=PORT)
client.confirmConnection()

# 2. list & filter scene objects
all_objs = client.simListSceneObjects()
pattern  = re.compile(REGEX)
matches  = [n for n in all_objs if pattern.match(n)]

print(f"\nFound {len(matches)} objects matching '{REGEX}':")
for n in matches:
    print(" ", n)

# 3. assign a shared seg-ID to all of them
SEG_ID = 200
print(f"\nSetting segmentation ID = {SEG_ID} on each human:")
for name in matches:
    ok = client.simSetSegmentationObjectID(name, SEG_ID, False)
    color = seg_id_to_rgb(SEG_ID)
    status = "OK" if ok else "FAIL"
    print(f" {status}: '{name}' → seg_id={SEG_ID}, color=RGB{color}")

# 4. flush the Python-side colormap cache in cosys-airsim
for attr in ("_segmentation_colormap",
             "segmentation_colormap",
             "_seg_colormap",
             "seg_colormap"):
    if hasattr(client, attr):
        delattr(client, attr)

# 5. now fetch a fresh colormap and print each spline-human’s actual ID
objects   = client.simListInstanceSegmentationObjects()
color_map = client.simGetSegmentationColorMap()  # Nx3 array

print("\n=== Current Segmentation IDs for BP_SplineHuman objects ===")
for name in matches:
    if name in objects:
        idx    = objects.index(name)
        seg_id = int(color_map[idx][0])  # R channel = seg_id
        print(f" '{name}' -> seg_id={seg_id}")
    else:
        print(f" '{name}' not found in simListInstanceSegmentationObjects()")

# 6. grab a full-HD segmentation image (1920×1080)
resp = client.simGetImages([
    airsim.ImageRequest(CAMERA_NAME, airsim.ImageType.Segmentation, False, False)
], vehicle_name=VEHICLE_NAME)[0]

if resp.width == 0 or resp.height == 0:
    raise RuntimeError("Empty segmentation image.")

print(f"\nSeg image resolution: {resp.width}×{resp.height}")

# 7. decode to RGB numpy array
img1d   = np.frombuffer(resp.image_data_uint8, dtype=np.uint8)
img_rgb = img1d.reshape(resp.height, resp.width, 3)
if FLIP_VERTICAL:
    img_rgb = np.flipud(img_rgb)

# 8. convert to BGR for OpenCV and display
img_bgr     = cv2.cvtColor(img_rgb, cv2.COLOR_RGB2BGR)
window_name = f"Segmentation (ID={SEG_ID})"
cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)
cv2.resizeWindow(window_name, resp.width // 2, resp.height // 2)
cv2.imshow(window_name, img_bgr)
print("\nPress any key in the image window to exit.")
cv2.waitKey(0)
cv2.destroyAllWindows()

Output:

Connected!
Client Ver:3 (Min Req: 3), Server Ver:3 (Min Req: 3)


Found 19 objects matching '.*BP_SplineHuman.*':
  BP_SplineHuman_Type14_C_UAID_E08F4CF5208A407A02_1421465596
  BP_SplineHuman_Type11_C_UAID_E08F4CF5208A487A02_1157467017
  BP_SplineHuman_Type5_C_UAID_E08F4CF5208A477A02_1587259839
  BP_SplineHuman_Type3_C_UAID_E08F4CF5208A3F7A02_1918728418
  BP_SplineHuman_Girl2_C_UAID_E08F4CF5208A457A02_1135924483
  BP_SplineHuman_Type13_C_UAID_E08F4CF5208A497A02_1081776195
  BP_SplineHuman_Type7_C_UAID_E08F4CF5208A417A02_1772760774
  BP_SplineHuman_Mannequin1_C_UAID_E08F4CF5208A457A02_1512040484
  BP_SplineHuman_Type10_C_UAID_E08F4CF5208A437A02_1596611129
  BP_SplineHuman_Type2_C_UAID_E08F4CF5208A417A02_1509275773
  BP_SplineHuman_Type6_C_UAID_E08F4CF5208A407A02_1088594595
  BP_SplineHuman_Mannequin2_C_UAID_E08F4CF5208A457A02_1886793485
  BP_SplineHuman_Type4_C_UAID_E08F4CF5208A467A02_1775944662
  BP_SplineHuman_Type9_C_UAID_E08F4CF5208A477A02_1940754840
  BP_SplineHuman_Type8_C_UAID_E08F4CF5208A437A02_1199966128
  BP_SplineHuman_Type12_C_UAID_E08F4CF5208A487A02_1781865018
  BP_SplineHuman_C_UAID_E08F4CF5208ADB7502_2044960591
  BP_SplineHuman_Mannequin2_C_UAID_E08F4CF5208A427A02_1146249951
  BP_SplineHuman_Girl1_C_UAID_E08F4CF5208A3F7A02_1511822415

Setting segmentation ID = 200 on each human:
 OK: 'BP_SplineHuman_Type14_C_UAID_E08F4CF5208A407A02_1421465596' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type11_C_UAID_E08F4CF5208A487A02_1157467017' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type5_C_UAID_E08F4CF5208A477A02_1587259839' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type3_C_UAID_E08F4CF5208A3F7A02_1918728418' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Girl2_C_UAID_E08F4CF5208A457A02_1135924483' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type13_C_UAID_E08F4CF5208A497A02_1081776195' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type7_C_UAID_E08F4CF5208A417A02_1772760774' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Mannequin1_C_UAID_E08F4CF5208A457A02_1512040484' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type10_C_UAID_E08F4CF5208A437A02_1596611129' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type2_C_UAID_E08F4CF5208A417A02_1509275773' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type6_C_UAID_E08F4CF5208A407A02_1088594595' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Mannequin2_C_UAID_E08F4CF5208A457A02_1886793485' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type4_C_UAID_E08F4CF5208A467A02_1775944662' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type9_C_UAID_E08F4CF5208A477A02_1940754840' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type8_C_UAID_E08F4CF5208A437A02_1199966128' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Type12_C_UAID_E08F4CF5208A487A02_1781865018' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_C_UAID_E08F4CF5208ADB7502_2044960591' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Mannequin2_C_UAID_E08F4CF5208A427A02_1146249951' → seg_id=200, color=RGB(200, 0, 0)
 OK: 'BP_SplineHuman_Girl1_C_UAID_E08F4CF5208A3F7A02_1511822415' → seg_id=200, color=RGB(200, 0, 0)

=== Current Segmentation IDs for BP_SplineHuman objects ===
 'BP_SplineHuman_Type14_C_UAID_E08F4CF5208A407A02_1421465596' -> seg_id=127
 'BP_SplineHuman_Type11_C_UAID_E08F4CF5208A487A02_1157467017' -> seg_id=191
 'BP_SplineHuman_Type5_C_UAID_E08F4CF5208A477A02_1587259839' -> seg_id=95
 'BP_SplineHuman_Type3_C_UAID_E08F4CF5208A3F7A02_1918728418' -> seg_id=95
 'BP_SplineHuman_Girl2_C_UAID_E08F4CF5208A457A02_1135924483' -> seg_id=255
 'BP_SplineHuman_Type13_C_UAID_E08F4CF5208A497A02_1081776195' -> seg_id=255
 'BP_SplineHuman_Type7_C_UAID_E08F4CF5208A417A02_1772760774' -> seg_id=95
 'BP_SplineHuman_Mannequin1_C_UAID_E08F4CF5208A457A02_1512040484' -> seg_id=159
 'BP_SplineHuman_Type10_C_UAID_E08F4CF5208A437A02_1596611129' -> seg_id=191
 'BP_SplineHuman_Type2_C_UAID_E08F4CF5208A417A02_1509275773' -> seg_id=95
 'BP_SplineHuman_Type6_C_UAID_E08F4CF5208A407A02_1088594595' -> seg_id=255
 'BP_SplineHuman_Mannequin2_C_UAID_E08F4CF5208A457A02_1886793485' -> seg_id=159
 'BP_SplineHuman_Type4_C_UAID_E08F4CF5208A467A02_1775944662' -> seg_id=159
 'BP_SplineHuman_Type9_C_UAID_E08F4CF5208A477A02_1940754840' -> seg_id=127
 'BP_SplineHuman_Type8_C_UAID_E08F4CF5208A437A02_1199966128' -> seg_id=223
 'BP_SplineHuman_Type12_C_UAID_E08F4CF5208A487A02_1781865018' -> seg_id=223
 'BP_SplineHuman_C_UAID_E08F4CF5208ADB7502_2044960591' -> seg_id=191
 'BP_SplineHuman_Mannequin2_C_UAID_E08F4CF5208A427A02_1146249951' -> seg_id=95
 'BP_SplineHuman_Girl1_C_UAID_E08F4CF5208A3F7A02_1511822415' -> seg_id=255

Seg image resolution: 1920×1080

This is the settings.json:

{
  "SettingsVersion": 2.0,
  "SimMode": "Multirotor",
  "ClockSpeed": 1.0,
  "SegmentationSettings": {
    "InitMethod": "CommonObjectsRandomIDs",
    "MeshNamingMethod": "StaticMeshName",
    "OverrideExisting": true
  },
  "PawnPaths": {
    "CustomPawn": {
      "PawnBP": "Class'/AirSim/VehicleAdv/SUV/UGVPawn.UGVPawn_C'"
    },
    "SUVCar": {
      "PawnBP": "Class'/AirSim/VehicleAdv/SUV/SuvCarPawn.SuvCarPawn_C'"
    }
  },
  "Vehicles": {
    "Drone1": {
      "VehicleType": "SimpleFlight",
      "X": 0,
      "Y": 0,
      "Z": 0,
      "Yaw": -90,
      "Cameras": {
        "front_center": {
          "X": 0.35,
          "Y": 0,
          "Z": 0.1,
          "Pitch": -10.0,
          "Roll": 0.0,
          "Yaw": 0.0,
          "CaptureSettings": [
            {
              "ImageType": 0,
              "Width": 1920,
              "Height": 1080,
              "FOV_Degrees": 100
            },
            {
              "ImageType": 1,
              "Width": 1920,
              "Height": 1080,
              "FOV_Degrees": 100
            },
            {
              "ImageType": 5,
              "Width": 1920,
              "Height": 1080,
              "FOV_Degrees": 100
            }
          ]
        }
      },
      "Sensors": {
        "LidarSensor1": {
          "SensorType": 6,
          "Enabled": true,
          "NumberOfChannels": 300,
          "RotationsPerSecond": 10,
          "Range": 280,
          "MeasurementsPerCycle": 300,
          "X": 0,
          "Y": 0,
          "Z": -0.50,
          "Roll": 0,
          "Pitch": 0,
          "Yaw": 0,
          "VerticalFOVUpper": 10,
          "VerticalFOVLower": -30,
          "HorizontalFOVStart": -50,
          "HorizontalFOVEnd": 50,
          "DrawDebugPoints": false,
          "DrawSensor": false
        },
        "GpsSensor1": {
          "SensorType": 3,
          "Enabled": false
        }
      }
    }
  }
}

(I’ve also tried "InitMethod": "None" & empty MeshNamingMethod per issues #3423 and #1852, but segmentation IDs keep being randomized or ignored.)

I used the color map (obtained using code below) to do draw 2D bounding boxes to my custom UE5 objects, and it worked correctly the first time:

However, I closed the editor and reopened, and reran the same code to collect the color map (pasted below), now the BP_SplineHuman objects, among others, have different colors, but the color map data still shows the previous colors from the screenshot above which are incorrect:

Code I used to save the color map data:
PythonClient/segmentation/segmentation_generate_list.py

#!/usr/bin/env python

import setup_path
import cosysairsim as airsim
import csv
import random
import numpy as np
from PIL import Image
from datetime import datetime

if __name__ == '__main__':

    # Make connection to AirSim API
    # client = airsim.CarClient()
    client = airsim.MultirotorClient()
    client.confirmConnection()

    # Generate list of all colors available for segmentation
    print("Loading segmentation colormap...")
    colorMap = client.simGetSegmentationColorMap()
    print("Loaded segmentation colormap.")

    # Get names of all objects in simulation world and store in list together with the associated RGB value
    # In a dynamic world, this is never the same!!
    currentObjectList = client.simListInstanceSegmentationObjects()
    print("Generating list of all current objects...")
    with open('airsim_segmentation_colormap_list_' +  datetime.now().strftime("%Y_%m_%d_%H_%M_%S") + '.csv', 'w') as f:
        f.write("ObjectName,R,G,B\n")
        for index, item in enumerate(currentObjectList):
            f.write("%s,%s\n" % (item, ','.join([str(x) for x in colorMap[index,:]])))
    print("Generated list of all current objects with a total of " + str(len(currentObjectList)) + ' objects.')

Environment

OS: Ubuntu 22.04
UE5 version: 5.2.1
Cosys-AirSim branch: 5.2.1

This may be related to a stale/cached segmentation colormap in the plugin (IDs get overwritten on each play session), or to a bug in the client’s simSetSegmentationObjectID. I’ve referenced:

Issue #4608

Issue #3423 – wrong colormap vs. UE colors, looked at pull request to fix this, same error persists

Issue #1852 – random IDs each launch

[Kokika]

@ssaigarimella Does your human contains multiple mesh?

[ssaigarimella]

@ssaigarimella Does your human contains multiple mesh?
The human is actually just a skeletal mesh; yes it contains multiple meshes for the limbs torso etc

[Kokika]

@ssaigarimella Does your human contains multiple mesh?
The human is actually just a skeletal mesh; yes it contains multiple meshes for the limbs torso etc

Cosys Airsim currently only support one mesh for skeletal meshes. One object can be draw with multiple color, however when you ask down only last color available, because it can only store one color per skeletal meshes.