Frame Rate Settings NDI VEGA

[samuv01]

Hello everyone,

first of all, thank you for the resources and for the work that has gone into this library.

I’m working with an NDI Vega to track the movement of a marker placed behind a camera, which is itself tracked using a ChAruco board. My goal is to compute the transform between the marker reference frame and the camera’s frame.

While I can configure and adjust the FPS of my camera, I’m struggling to do the same with the Vega. According to the documentation and factory specifications, the Vega should provide tracking updates at 60 Hz, but when I log my data the effective frame rate is only around 20 Hz on average.

My question is: how can I configure or force the Vega tracker to stream at the full 60 Hz rate?

• Is there a setting in the API/commands that I need to change?

• Or is the 20 Hz limitation coming from the Python interface (e.g. blocking calls, polling speed), and not the device itself?

Any suggestions, experiences, or pointers to relevant documentation would be greatly appreciated.

Thanks in advance!

[thompson318]

One thing to look at might be the type of transforms you are using, have a look at

scikit-surgerynditracker/sksurgerynditracker/nditracker.py

Lines 182 to 186 in c52694f

	if firmware in (' AURORA Rev 007', ' AURORA Rev 008',
	' Polaris Vega 008',
	' Polaris Spectra Rev 006', ' Polaris Spectra Rev 007'):
	self._get_frame = getattr(ndicapy, 'ndiGetTXFrame')
	self._get_transform = getattr(ndicapy, 'ndiGetTXTransform')

If you've got a Vega 008 you'll be using TX (text) transforms which may be slower than BX (binary) transforms. You could delete line 183 and see if that speeds things up (though it may not work).

If that fails you may have to look at the upstream library ndicapi that we use for communication. I've never tried to do anything too clever with it as our library is primarily aimed at being easy to use rather than superfast, but there are functions to use threading ndiSetThreadMode that may improve performance depending on your application. You could try something like:

import nidicapy

ndicapy.ndiSetThreadMode(...)

Though I'm not sure what the parameters would be, you'd have to the ndicapi documentation. ndicapi is part of the PlusToolkit which has more features than our library, so you may find more useful information there.

[samuv01]

Thank you for the quick answer.

I tried both methods but I wasn't able to solve my issue.

I read through the NDI functions and understood that in order to evaluate the frequency of the camera I should look only at the framenumbers (which increase at 60Hz). I don't understand why the get_frame function in nditracker.py skips some frames and duplicates others. Am I doing something wrong?

Thanks again,
S

[thompson318]

Could you post some of the output of get_frame here (a list of the time_stamps and frame_numbers)? How many tools are you tracking?

[samuv01]

Hello,
here it is:
[{'dev': 'vega', 'transform': array([[-9.9718020e-01, -4.2781240e-02, 6.2293680e-02, -1.5913000e+02],
[ 5.2426800e-02, -9.8535060e-01, 1.6252460e-01, -7.7580000e+01],
[ 5.4425800e-02, 1.6532568e-01, 9.8473644e-01, -1.1534800e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.3283768, 'framenumber': 1449617635.0, 'quality': 0.1562}, {'dev': 'vega', 'transform': array([[-9.9718020e-01, -4.2781240e-02, 6.2293680e-02, -1.5913000e+02],
[ 5.2426800e-02, -9.8535060e-01, 1.6252460e-01, -7.7580000e+01],
[ 5.4425800e-02, 1.6532568e-01, 9.8473644e-01, -1.1534800e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.347566, 'framenumber': 1449617635.0, 'quality': 0.1562}, {'dev': 'vega', 'transform': array([[-9.9734530e-01, -4.3614900e-02, 6.1587100e-02, -1.5901000e+02],
[ 5.3186580e-02, -9.8530404e-01, 1.6351650e-01, -7.7470000e+01],
[ 5.3538940e-02, 1.6632558e-01, 9.8461790e-01, -1.1536000e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.3675764, 'framenumber': 1449617638.0, 'quality': 0.1578}, {'dev': 'vega', 'transform': array([[-9.9734530e-01, -4.3614900e-02, 6.1587100e-02, -1.5901000e+02],
[ 5.3186580e-02, -9.8530404e-01, 1.6351650e-01, -7.7470000e+01],
[ 5.3538940e-02, 1.6632558e-01, 9.8461790e-01, -1.1536000e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.39057, 'framenumber': 1449617638.0, 'quality': 0.1578}, {'dev': 'vega', 'transform': array([[-9.9734530e-01, -4.3614900e-02, 6.1587100e-02, -1.5901000e+02],
[ 5.3186580e-02, -9.8530404e-01, 1.6351650e-01, -7.7470000e+01],
[ 5.3538940e-02, 1.6632558e-01, 9.8461790e-01, -1.1536000e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.407569, 'framenumber': 1449617638.0, 'quality': 0.1578}, {'dev': 'vega', 'transform': array([[-9.9731220e-01, -4.3222320e-02, 6.1549160e-02, -1.5908000e+02],
[ 5.2782440e-02, -9.8530404e-01, 1.6332888e-01, -7.7670000e+01],
[ 5.3576880e-02, 1.6611484e-01, 9.8465100e-01, -1.1535500e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.4282844, 'framenumber': 1449617641.0, 'quality': 0.1519}, {'dev': 'vega', 'transform': array([[-9.9731220e-01, -4.3222320e-02, 6.1549160e-02, -1.5908000e+02],
[ 5.2782440e-02, -9.8530404e-01, 1.6332888e-01, -7.7670000e+01],
[ 5.3576880e-02, 1.6611484e-01, 9.8465100e-01, -1.1535500e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.4481394, 'framenumber': 1449617641.0, 'quality': 0.1519}, {'dev': 'vega', 'transform': array([[-9.9724612e-01, -4.2486660e-02, 6.0875980e-02, -1.5931000e+02],
[ 5.1924660e-02, -9.8526954e-01, 1.6296786e-01, -7.8010000e+01],
[ 5.3054980e-02, 1.6567914e-01, 9.8475158e-01, -1.1533900e+03],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.0000000e+00]]), 'timestamp': 1759849930.4681368, 'framenumber': 1449617644.0, 'quality': 0.15610000000000002}

I am tracking only 1 tool.
This output is obtained with only the NDI Vega ST connected to the laptop. I don't have an ethernet port so I'm using an ethernet adaptor to usb A.

From what I understood the NDI is tracking at 20 Hz and I'm polling at a higher frequency (50 Hz if we look at the CPU timestamps), so frame duplicates are returned because the NDI doesn't have anything else available.

I modified and saved the frame rate settings on NDI Track and NDI Configure as shown in the picture below, but everytime I initialize the device these reset to the initial ones.

Thank you,
let me know if you have any questions.

[samuv01]

These are my results if I use BX transforms:
[{'dev': 'vega', 'transform': array([[-9.97074437e-01, -7.64253158e-02, 1.31408213e-03,
-1.62312683e+02],
[ 7.55051330e-02, -9.87453974e-01, -1.38685297e-01,
1.90922897e+02],
[ 1.18966633e-02, -1.38180345e-01, 9.90335631e-01,
-1.21653162e+03],
[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
1.00000000e+00]]), 'timestamp': 1759852470.21958, 'framenumber': 1449770035.0, 'quality': 0.16095367074012756}, {'dev': 'vega', 'transform': array([[-9.98233817e-01, -5.88681326e-02, 7.97423743e-03,
-1.61525101e+02],
[ 5.71583348e-02, -9.88348987e-01, -1.41063835e-01,
1.89050247e+02],
[ 1.61854952e-02, -1.40358911e-01, 9.89968386e-01,
-1.21568103e+03],
[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
1.00000000e+00]]), 'timestamp': 1759852470.437586, 'framenumber': 1449770050.0, 'quality': 0.15711364150047302}, {'dev': 'vega', 'transform': array([[-9.98233817e-01, -5.88681326e-02, 7.97423743e-03,
-1.61525101e+02],
[ 5.71583348e-02, -9.88348987e-01, -1.41063835e-01,
1.89050247e+02],
[ 1.61854952e-02, -1.40358911e-01, 9.89968386e-01,
-1.21568103e+03],
[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
1.00000000e+00]]), 'timestamp': 1759852470.4492407, 'framenumber': 1449770050.0, 'quality': 0.15711364150047302}, {'dev': 'vega', 'transform': array([[-9.98233817e-01, -5.88681326e-02, 7.97423743e-03,
-1.61525101e+02],
[ 5.71583348e-02, -9.88348987e-01, -1.41063835e-01,
1.89050247e+02],
[ 1.61854952e-02, -1.40358911e-01, 9.89968386e-01,
-1.21568103e+03],
[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
1.00000000e+00]]), 'timestamp': 1759852470.4653137, 'framenumber': 1449770050.0, 'quality': 0.15711364150047302}, {'dev': 'vega', 'transform': array([[-9.97918016e-01, -6.34559995e-02, 1.15369658e-02,
-1.60912857e+02],
[ 6.13204679e-02, -9.88917366e-01, -1.35212334e-01,
1.89548401e+02],
[ 1.99891386e-02, -1.34223363e-01, 9.90749476e-01,
-1.21508984e+03],
[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
1.00000000e+00]]), 'timestamp': 1759852470.4814553, 'framenumber': 1449770053.0, 'quality': 0.16213876008987427},

Once we solve this issue I would also like to understand why the qualities are so low and how I could improve them (I'm guessing the range is 0-1).

[thompson318]

Just checking I've understood the output right.

• You've grabbed 8 frames in 0.14s seconds CPU which is 50Hz 👍
• In that time the framenumber has gone from 1449617635 to 1449617644 (9 frames 9/0.14 = ~60hz) 👍
• But you've actually only collected 4 unique frames (635, 638, 641, 644) because of repeats and skipped frames. 👎

So to me it appears that the Vega is running at the correct frame rate (60 Hz), but we're not sampling it right. I'll need to think a bit.

[samuv01]

Thank you for the quick answer!
I interpreted the output differently.

I was discarding the frame duplicates and just looking at the unique frames to calculate the Vega frame rate.