Two computer vision tasks combined

[JasonSloan]

Hi, there!
Can i combine the two tasks keypoints and object-detection bouding boxes labeling into one template?
Best wishes!

[lovelesh]

Yes. You can definitely do that.
Refer to SAM Integration guide

[heidi-humansignal]

Hello Jason,

Do you mean something like this:

<View> <Image name="image" value="$image"/> <RectangleLabels name="bbox" toName="image"> <Label value="Object" background="blue"/> </RectangleLabels> <KeyPointLabels name="keypoint" toName="image"> <Label value="Keypoint" background="red"/> </KeyPointLabels></View>

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run

[JasonSloan]

Hello Jason,

Do you mean something like this:

<View> <Image name="image" value="$image"/> <RectangleLabels name="bbox" toName="image"> <Label value="Object" background="blue"/> </RectangleLabels> <KeyPointLabels name="keypoint" toName="image"> <Label value="Keypoint" background="red"/> </KeyPointLabels></View>

Thank you, Abu

Comment by Abubakar Saad
Workflow Run

Yes, exactly!
Can you give me more detail?

Thanks a lot!

[heidi-humansignal]

Hello Jason!

Thanks for the follow-up. Here is a more detailed explanation of how to include both bounding boxes and keypoints within a single labeling configuration.

1. Create a common tag to define your image source.

2. Add a element for bounding box annotations, specifying a name (for example, "bbox").

3. Add a element for keypoint (pose) detection, specifying a name (for example, "kp").

4. Make sure both use the same toName reference ("image" in the example):

• is for bounding boxes, so you can annotate objects.

• allows you to mark specific keypoints (like body joints or reference points within your objects).

Make sure the from_name and toName attributes match up correctly in your labeling config.
This setup lets you annotate bounding boxes and keypoints in the same project and on the same image tasks.

Let us know if you have any other questions!

Comment by Oussama Assili
Workflow Run

[JasonSloan]

Hello Jason,

Do you mean something like this:

<View> <Image name="image" value="$image"/> <RectangleLabels name="bbox" toName="image"> <Label value="Object" background="blue"/> </RectangleLabels> <KeyPointLabels name="keypoint" toName="image"> <Label value="Keypoint" background="red"/> </KeyPointLabels></View>

Thank you, Abu

Comment by Abubakar Saad
Workflow Run

Hi there!
I had given your suggestion a shot, while it wouldnt work well.
It seemed the keypoints could be only added outside the bounding box rather than inside the bounding box.
I wanted to make it clear that i had integrated model predictions into the labeling task, maybe this influenced the behaviour of label-studio?

Looking forward to your help! Thanks!

[heidi-humansignal]

Hello Jason,

Based on the feedback you provided, here are a few points and additional insights that may help resolve the issue when combining ML-predicted bounding boxes with keypoint annotations:

1. Consistent toName References:

Make sure that both your and tags share the same toName attribute (e.g., "image"). This is essential so that Label Studio links annotations to the same image and can group them correctly.
2. Model Predictions and Keypoint Grouping:

When using an ML backend that provides both bounding box and keypoint predictions, keypoints are typically grouped with their corresponding object if the predictions include a shared region ID. If you integrate model predictions, ensure that your keypoint labeling element includes the parameter model_add_bboxes="true". This tells Label Studio to include (or group) the bounding boxes from keypoint detections together with the keypoint results. For example:

<KeyPointLabels name="kp" toName="image" model_add_bboxes="true" model_score_threshold="0.75" model_point_threshold="0.5" model_point_size="1" model_path="yolov8n-pose.pt"> <Label value="Nose" background="red"/> <Label value="Eye" background="yellow"/> <Label value="Hand" background="purple"/> </KeyPointLabels>

1. Annotation Layout Behavior:

It’s important to note that in Label Studio, keypoint annotations are not “nested” inside the drawn bounding box in the UI; they overlay the entire image. Their placement depends on the coordinates returned by the model or the ones you manually add. If you’re seeing keypoints only outside the bounding box, it might be due to how the model predictions are being merged or the region positions are calculated. This behavior can sometimes be influenced by extensive ML pre-annotations. In such cases, manually reviewing and adjusting the positions using the region editor can help.
2. Integrated Model Predictions Impact:

Since you mentioned that model predictions are integrated into your task, it’s possible that the pre-annotation results are affecting the interactive behavior. When predictions are present, Label Studio groups results based on their region IDs. If the keypoints are coming as separate predictions, they might not be automatically "attached" or overlaid to the corresponding bounding box region. Double-check that your ML backend’s output JSON correctly associates keypoints with the same region (i.e., same annotation ID) as the bounding box. If not, you may consider re-running the ML prediction with settings adjusted for grouping.

I hope these insights help clarify the behavior and assist you in refining your project setup. Please let me know if you have further questions or need additional assistance!

Comment by Oussama Assili
Workflow Run

[JasonSloan]

Hello Jason,

Based on the feedback you provided, here are a few points and additional insights that may help resolve the issue when combining ML-predicted bounding boxes with keypoint annotations:

1. Consistent toName References:

Make sure that both your and tags share the same toName attribute (e.g., "image"). This is essential so that Label Studio links annotations to the same image and can group them correctly. 2. Model Predictions and Keypoint Grouping:

When using an ML backend that provides both bounding box and keypoint predictions, keypoints are typically grouped with their corresponding object if the predictions include a shared region ID. If you integrate model predictions, ensure that your keypoint labeling element includes the parameter model_add_bboxes="true". This tells Label Studio to include (or group) the bounding boxes from keypoint detections together with the keypoint results. For example:

<KeyPointLabels name="kp" toName="image" model_add_bboxes="true" model_score_threshold="0.75" model_point_threshold="0.5" model_point_size="1" model_path="yolov8n-pose.pt"> <Label value="Nose" background="red"/> <Label value="Eye" background="yellow"/> <Label value="Hand" background="purple"/> </KeyPointLabels>

1. Annotation Layout Behavior:

It’s important to note that in Label Studio, keypoint annotations are not “nested” inside the drawn bounding box in the UI; they overlay the entire image. Their placement depends on the coordinates returned by the model or the ones you manually add. If you’re seeing keypoints only outside the bounding box, it might be due to how the model predictions are being merged or the region positions are calculated. This behavior can sometimes be influenced by extensive ML pre-annotations. In such cases, manually reviewing and adjusting the positions using the region editor can help. 2. Integrated Model Predictions Impact:

Since you mentioned that model predictions are integrated into your task, it’s possible that the pre-annotation results are affecting the interactive behavior. When predictions are present, Label Studio groups results based on their region IDs. If the keypoints are coming as separate predictions, they might not be automatically "attached" or overlaid to the corresponding bounding box region. Double-check that your ML backend’s output JSON correctly associates keypoints with the same region (i.e., same annotation ID) as the bounding box. If not, you may consider re-running the ML prediction with settings adjusted for grouping.

I hope these insights help clarify the behavior and assist you in refining your project setup. Please let me know if you have further questions or need additional assistance!

Comment by Oussama Assili
Workflow Run

Hello! Thanks for your help!

Id like to clarify that in this case, I only integrated bounding boxes predctions into the task while keypoints were manually added.

And I had studied and tried the your suggestions, here are my corresponding experiment results:

1. Consistent toName References:

I double checked the toName attribute, both the object detection and keypoint detection tasks are the same. Shown below:
<View> <Image name="image" value="$image"/> <RectangleLabels name="label" toName="image" model_score_threshold="0.2" model_path="" opacity="0.1"> <Label value="person_easy" background="red"/> <Label value="person_hard" background="red"/> <Label value="person_uniform" background="yellow"/> <Label value="person_nouniform" background="green"/> <Label value="head_easy" background="blue"/> <Label value="head_hard" background="blue"/> <Label value="helmet" background="purple"/> <Label value="lookout" background="orange"/> <Label value="mobilephone_easy" background="cyan"/> <Label value="mobilephone_hard" background="cyan"/> <Label value="cigarette" background="magenta"/> <Label value="safebelt" background="black"/> <Label value="vessel" background="brown"/> </RectangleLabels> <KeyPointLabels name="kp" toName="image" model_add_bboxes="true"> <Label value="Nose" background="red"/> <Label value="Eye" background="yellow"/> <Label value="Hand" background="purple"/> </KeyPointLabels> </View>

2. Model Predictions and Keypoint Grouping:

I had tried the code above you provided but didnt work, still couldnt add keypoints inside the predicted bounding boxes.

3. Annotation Layout Behavior:

Since I just wanted to add keypoints manually, this would likely not be a problem.

4. Integrated Model Predictions Impact:

I thought this could be the reason. But I didnt figure out the concept of group. And I had also tried that if I just put the object detection and keypoints togother and all labeled by hand, they two could work fine. So there must be something I forgot to configure in ML backend, probably was "group" like you said.
So can you help me go further about this? Thanks very much!
Besides, below is an example of my ML backend prediction, and I cant see group in it.
[ { "model_version": "0.0.1", "result": [ { "from_name": "label", "score": 0.8345159888267517, "to_name": "image", "type": "rectanglelabels", "value": { "height": 6.238238016764322, "rectanglelabels": [ "head_easy" ], "rotation": 0, "width": 2.9478073120117188, "x": 39.03620719909668, "y": 14.393158665409794 } }, { "from_name": "label", "score": 0.8938227891921997, "to_name": "image", "type": "rectanglelabels", "value": { "height": 39.395345052083336, "rectanglelabels": [ "person_easy" ], "rotation": 0, "width": 5.971978505452474, "x": 36.912740071614586, "y": 14.427541097005209 } } ], "score": 0.8641693890094757 } ]

[heidi-humansignal]

Hello Jason,

Thank you for the extra details and inputs, that was greatly appreciated!

I understand that you’re manually adding keypoints while using ML-predicted bounding boxes, but the keypoints aren’t “grouped” with their corresponding boxes as you’d expect.
Here’s some further insight to help you move forward:

1. How Grouping Works:

In Label Studio, “grouping” of annotations (linking keypoints to a bounding box) is typically achieved when both the keypoints and bounding boxes are generated together by the ML backend. In that case, the backend output includes a shared identifier (often part of the prediction’s metadata) that tells Label Studio which keypoints belong to which bounding box. When you add keypoints manually, they are created as separate annotations and won’t automatically attach to the pre-annotated bounding boxes unless they share a common “group” (or region ID).
2. Current ML Output:

Your provided ML prediction JSON only includes bounding boxes without any grouping field (no key like “group” or annotation id linking to keypoint predictions). This means that even if you enable model_add_bboxes="true" on your , it won’t “attach” manually added keypoints to the predicted boxes because the ML backend isn’t returning both types of predictions together.
3. Next Steps:

• Option A: Integrate Both Predictions
  If possible, consider updating your ML backend so that it returns both the keypoint predictions and the bounding box predictions in the same response. They must share a common identifier so that Label Studio can group them automatically. For example, a modified prediction JSON might include an extra property (such as "group": "annotation_X") for both the bounding box and its related keypoints.

• Option B: Manual Post-Processing
  If you must combine an ML backend that only supplies bounding boxes with manually added keypoints, then grouping won’t occur automatically. In this case, you may need to adjust post-annotation by manually linking the keypoints with their corresponding bounding box regions (using Label Studio’s region editor) or consider a custom script using the SDK to “merge” annotations based on spatial overlap.

1. ML Backend Configuration:
   Since you’re currently not seeing any grouping in the ML predictions, examine your ML backend configuration and check if it supports multi-task outputs that include keypoints along with bounding boxes. Sometimes, additional parameters or updates (for example, setting a “group” flag in the output) are required. If the backend only returns bounding boxes, then grouping won’t be triggered regardless of the label configuration.
   I hope these suggestions help you move forward. Please let me know if you need further details

Comment by Oussama Assili
Workflow Run

[JasonSloan]

Hello Jason,

Thank you for the extra details and inputs, that was greatly appreciated!

I understand that you’re manually adding keypoints while using ML-predicted bounding boxes, but the keypoints aren’t “grouped” with their corresponding boxes as you’d expect. Here’s some further insight to help you move forward:

1. How Grouping Works:

In Label Studio, “grouping” of annotations (linking keypoints to a bounding box) is typically achieved when both the keypoints and bounding boxes are generated together by the ML backend. In that case, the backend output includes a shared identifier (often part of the prediction’s metadata) that tells Label Studio which keypoints belong to which bounding box. When you add keypoints manually, they are created as separate annotations and won’t automatically attach to the pre-annotated bounding boxes unless they share a common “group” (or region ID). 2. Current ML Output:

Your provided ML prediction JSON only includes bounding boxes without any grouping field (no key like “group” or annotation id linking to keypoint predictions). This means that even if you enable model_add_bboxes="true" on your , it won’t “attach” manually added keypoints to the predicted boxes because the ML backend isn’t returning both types of predictions together. 3. Next Steps:

• Option A: Integrate Both Predictions
  If possible, consider updating your ML backend so that it returns both the keypoint predictions and the bounding box predictions in the same response. They must share a common identifier so that Label Studio can group them automatically. For example, a modified prediction JSON might include an extra property (such as "group": "annotation_X") for both the bounding box and its related keypoints.
• Option B: Manual Post-Processing
  If you must combine an ML backend that only supplies bounding boxes with manually added keypoints, then grouping won’t occur automatically. In this case, you may need to adjust post-annotation by manually linking the keypoints with their corresponding bounding box regions (using Label Studio’s region editor) or consider a custom script using the SDK to “merge” annotations based on spatial overlap.

1. ML Backend Configuration:
   Since you’re currently not seeing any grouping in the ML predictions, examine your ML backend configuration and check if it supports multi-task outputs that include keypoints along with bounding boxes. Sometimes, additional parameters or updates (for example, setting a “group” flag in the output) are required. If the backend only returns bounding boxes, then grouping won’t be triggered regardless of the label configuration.
   I hope these suggestions help you move forward. Please let me know if you need further details

Comment by Oussama Assili
Workflow Run

Thanks for your suggestions.

Now maybe I think I should give up attempting to integrate ML-backend detection with mannually labeling keypoints.

It will be more easier to seperate them.

Thanks a lot after all! Close this issue now.