Why RetinaNet is slower than Faster RCNN?

[alyssahyq]

In the COCO Object Detection Baselines, I find Faster RCNNs have lower inference time than RetinaNets with the same backbone.
However, RetinaNet is supposed to run faster than Faster RCNN according to the paper.

I checked the configuration and I didn't get any clue yet. Any ideas are welcome!

Here's a snippet of the baseline table:

Configs	lr sched	train time (s/iter)	inference time (s/im)	train mem (GB)	box AP	model id	download
faster_rcnn_R_50_FPN	1x	0.210	0.038	3.0	37.9	137257794	model | metrics
retinanet_R_50_FPN	1x	0.205	0.041	4.1	37.4	190397773	model | metrics
faster_rcnn_R_101_FPN	3x	0.286	0.051	4.1	42.0	137851257	model | metrics
retinanet_R_101_FPN	3x	0.291	0.054	5.2	40.4	190397697	model | metrics

[ppwwyyxx]

RetinaNet is supposed to run faster than Faster RCNN according to the paper.

The original paper uses old implementations & old frameworks, so the conclusion is different.

Page 128-132 of https://www.dropbox.com/s/6izllk62dobao9n/ICCV19_generalized_rcnn_tutorial_ross_girshick.pptx?dl=0 also explains this story.