Can't run Video Object Tracking SAM-2 pre-annotation

[swap-10]

Describe the bug
Using the video object tracking feature with sam-2 as described here: https://docs.humansignal.com/tutorials/segment_anything_2_video

To Reproduce
Steps to reproduce the behavior:

1. Clone and run label studio and label studio ML backend.
2. Run the segment_anything_2_video example from label studio ML backend
3. Follow all the steps in the tutorial.
4. See error in ml backend:

path/to/model.py in get_prompts():
for ctx in context['result']
TypeError: NoneType object is not subscriptable

Expected behavior
Predictions get computed and appear on the interface as shown in the tutorial

Environment (please complete the following information):

• OS: [e.g. iOS] label studio on windows 11, label studio ml backend on Ubuntu 22.04
• Label Studio Version [e.g. 0.8.0]: Cloned at commit 6da49c1

Additional context
Looks like interactive annotation is not sending any requests. The errors seem to have been caused by pre-labelling requests.
On disabling pre-labelling, and following the interactive annotation process as shown in the tutorial GIF:

1. Toggle auto-annotation in labelling interface
2. Select label
3. Draw bounding box

No requests seem to be sent to the ML backend.

[heidi-humansignal]

Hello,

Do you have GPU setup? Are you using docker to setup either LS or ML backend?

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run

[swap-10]

Yes, I have GPU setup.
I'd like to make some corrections to my first comment.

1. VOT is working for interactive annotation using ML backend.
2. VOT is not working for pre-annotation by selecting 'retrieve predictions for all tasks' in the data manager.
   API request is sent to the ML backend in (2) but it errors out with
   TypeError: NoneType object is not subscriptable
   in segment_anything_2_video/model.py get_prompts()

I guess the API call is not receiving the context object properly.

Neither LS nor ML backend is running in docker.

Thank you for your response!

[heidi-humansignal]

Hello,

Try this suggestion:

1. Handle “None” Context for Pre-Labeling
   Modify the parts of your SAM2 code (in the model’s “get_prompts” or “predict” method) so that it checks if “context” or “context['result']” is present before accessing them. For example:

   def get_prompts(self, context): if not context or 'result' not in context: return [] # existing code that loops over context['result']

When you run a full “retrieve predictions” job, often there’s no bounding box or keypoint input yet, so the “context” can come through as None.

Thank you,
Abu

Comment by Abubakar Saad
Workflow Run