[ROS2] Multiple cameras not working on Jetson Orin

[JulioPlaced]

Hello,
We are using two Blackfly S GigE cameras connected to a Jetson Orin AGX (Ubuntu 22 & ROS2 Humble) using a switch (since the Orin only has one ethernet port). The two cameras are correctly configured and we can see both of them at the same time using Spinview.

However, when we launch the driver, only one of the cameras work. Both cameras are found (serials are correct), and the network bandwidth should not be a problem (they work on Spinview and camera rate/image size have been reduced to test this). Using the launch file provided for multiple cameras, the following error shows up:

rate [Hz] in 10.0 out 0.00 drop 0% INCOMPLETE 100%

This page relates this error to a bandwith problem, although we are sure that is not the case.

Any help would be greatly appreciated.

[berndpfrommer]

The incomplete means that at the SDK reports back to the ROS2 driver that incoming image frames are incomplete. That indicates that part of the image frame is lost in transmission.
I assume you have checked that your MTUs are all set to 9000 and you get full bandwidth with SpinView on both cameras individually and combined.
So first thing is to test the two cameras with the ROS2 driver individually (launch in single camera mode) and see that you get full bandwidth by cranking up the frame rate and image size. Does that check out for both cameras?
The multiple cameras launch file that you linked does not set the jumbo parameters, so there is a chance the camera is not running with jumbo frames. You can look at the single camera example launch file to see how to set the camera to use jumbo packets ('gev_scps_packet_size': 9000)

[JulioPlaced]

Hello,
Thanks for the quick reply. Yes, mtu are set to 9000 and jumbo packet configuration enabled in the config file.

I can see both cams individually and together in spinview, and the driver works well with each camera individually. I can send you config/snapshots if that helps.

[berndpfrommer]

Please post your complete multi-camera launch file and the complete startup console log, up to the first few INCOMPLETE messages.
Also check the CPU usage with top. Is the driver running near 100%?
(Side note: Will you run the cameras hardware synchronized? If yes then you should run the synchronized camera driver).

[JulioPlaced]

This is the log:

This is the CPU usage (low):

This is the config in the launch file:

The rest of the file is exactly as in the repo, except for the serial numbers.

I recently noticed that despite MTUs are set to 9000, ifconfig promts:

Could this be reason? For now, I have not been able to set it to exactly 9000.

[berndpfrommer]

You definitely want the MTUs to match. The Jetsons seem to be a bit idiosyncratic here, so try to set the camera to 8966 as well, see if it takes that.
Your config looks fine. The incomplete frames are a sign that things are going wrong at the SDK layer already, the ROS2 driver should only be the culprit insofar it does not set the camera parameters correctly. This really feels like a networking setup issue. I have been able to pull 10 1-gigabit cameras simultaneously at full speed using the synchronized driver and now issue.
Reading up on the Orin AGX I see it has 2 interfaces: 1 GBit and 10 GBit. You are plugging into the 10GBit, right? And you are 100% positive that you can pull 2 GBit total when using SpinView, correct? Will the ROS2 driver work in any two-camera configuration, with low frame rate and low resolution? Sorry to be so distrustful of your statement that bandwidth is not the issue. Whenever I've seen incomplete frames it was dropped UDP packets without fail.
Oh, you can check for RX errors and dropped packets with ifconfig.

[berndpfrommer]

@JulioPlaced were you able to make progress on this?

[JulioPlaced]

Hello,
As far as I know, the Orin AGX usually has only 1 interface accessible to the user, which is 10GBit. There is a second interface (1GBit) but only a few manufacturers make it accessible (which is not our case). The swich used to connect the two cameras to the Orin has 5x1Gbit ports.

[berndpfrommer]

But wouldn't then the two cameras share a 1 Gbit link (because the switch only has 1 Gbit ports)?

[JulioPlaced]

The switch has up to 5Gbit output

[JulioPlaced]

I also checked RX with ifconfig and bmon, but there are no errors/drops.

[berndpfrommer]

What bandwidth utilization do you see?
If the switch drops the packets you would not see that show up as errors using ifconfig on your Orin.