TensorRT convert and Test

[BlackLee68]

hi @wenbowen123 , thanks for the great work!

I tested the model on AGX orin 32G(Jetpack 6.2), I found the 20-30-48 and the 23-36-67 shows different results when using TRT model for testing, I use the same convert code and command to generate TRT engine, but the final results seemed quite different when using run_demo_tensorrt.py, below is the res:

1, 23-36-67 Checkpoint with 4 iters, the result looks good:

2, 20-30-48 Checkpoint with 4 iters, the result looks a little strange:

I checked the post_runner, found the input is different between 23-36-67 and 20-30-48, the "features_left_16" is not used in 20-30-48 TRT engine, is this normal or i missed something when converting the model ?

Thanks a lot！

[te-shi]

Hi, I encountered the same issue. From my testing, if you are using the 20-26-39 or 20-30-48 checkpoints, you need to set normalize=False in the build_gwc_volume_triton function. Hope this helps.

[wenbowen123]

Hi, this should have been fixed in latest update.

[BlackLee68]

thanks, that helps. @wenbowen123 any plan to release training code?

[sunrise2399]

Hello, may I ask if your jetpack is 6.2.2? What version of tensorrt is there? The model operation result I obtained by converting with origin jetpack6.2.2 and tensorrt 10.3 is completely abnormal.

[Iridium771110]

@runner-9 same issue，jetpack 6 with cuda 12.6 & tensorrt 10.3, the result is chaos. Checked the output of feature-runner from run_demo_tensorrt semms different from the result from run_demo.

[Iridium771110]

@runner-9 I have solved it by my side.
It is not a bug from nvidia, just try img0 = img0.contiguous() and img1 = img1.contiguous() after permute operation.
Since in pytorch permutation may do not change the data in memory, but for tensorrt, it directly manages the memory.

[Iridium771110]

@wenbowen123
and one more question. As far as I know, cudagraph is a method for acceleration in inference, so use --useCudaGraph while building engine will not help for load this engine and inference use API individually, unless capture cudagraph and enqueue it use API after load engine. I wonder whether I am right.

[Mikasal]

@Iridium771110 Hello, can you run on the orin platform in real time? Although the result of my run was fine, the FPS (frames per second) was very low.

[Iridium771110]

@Mikasal my orin is AGX-64G，with MAXN-mode and jetson_clocks activated, it can run in real time.
for 23-36-67, 8-iters for post, 100ms per frame excluding img-reading, while 20-30-48 and 20-26-39 take 80ms and 90ms respectively. Since img-reading is a cpu-process, can be paralleled with gpu-process.
so 10FPS can be reached in my sight

[BlackLee68]

@Mikasal my orin is AGX-64G，with MAXN-mode and jetson_clocks activated, it can run in real time. for 23-36-67, 8-iters for post, 100ms per frame excluding img-reading, while 20-30-48 and 20-26-39 take 80ms and 90ms respectively. Since img-reading is a cpu-process, can be paralleled with gpu-process. so 10FPS can be reached in my sight

basically the same, orin AGX-32G, with MAXN-mode, for 23-36-67/8iters, using cudagraph, 11FPS can be reached

[jinyqjinyq]

@Iridium771110 my orin is also AGX-64G，with MAXN-mode and jetson_clocks activated, but i can't run in real time. for 23-36-67, 8-iters for post, 500ms-600ms per frame. The tensorrt version is 8.5.2, what can i try to get the same fps with you?

[DylanCoene]

How long did TRT conversion take on AGX-32G? I'm planning to run this next to Foundation Pose, and was able to get the .pt version running but this maxed out my RAM. I did not yet manage to convert to TRT, because this takes hours on Jetpack 6.1.

[Iridium771110]

@Iridium771110 my orin is also AGX-64G，with MAXN-mode and jetson_clocks activated, but i can't run in real time. for 23-36-67, 8-iters for post, 500ms-600ms per frame. The tensorrt version is 8.5.2, what can i try to get the same fps with you?

@jinyqjinyq
could you test the cost for every part?
in foundation_stereo.py TrtRunner, you can record the time cost for feature-runner, gwc-volume & post-runner.
and for tensorrt engine, you can check the inference cost using trtexec.
one possible thing is that 8.5.2 is the version without kernel-fusion for transformer-structure, so low FPS is expected.
with 8.6, in my test, 8.5hz can be reached, so just try update your jetpack which support tensorrt higher vision.
or you should first test the time cost every part to localize the source that takes large cost.