inference extra latency outside of a continuous loop

[SpitchAG]

Hello
I would like to understand where comes an extra latency when executing a model inference:
if I run for (N times) model.infer(); sleep(1s); repeat ...
the very first model.infer() (within a loop) has always a penalty (instead of 60ms, it goes up to 110ms). Without the sleep(), no penalty (the first of the first loop always has, but then the repeated loops dont show this latency).
is this expected? how to mitigate ? (without doing busy dummy infer)
is that related to how tbb worker threads are rescheduled/initialized ? something else?
thanks!

[Ganesh-403]

@SpitchAG From what you describe, this actually sounds like expected behavior, although the magnitude of the latency increase (60 ms → ~110 ms) will depend on the platform.

Since the penalty appears only after the application has been idle for ~1 second, but disappears when inference runs continuously, I'd suspect it's related to the runtime and CPU scheduling rather than the model itself.

A few possible causes:

1. Thread pool wake-up

   OpenVINO's CPU plugin uses oneTBB by default. After inference completes, worker threads don't necessarily remain active indefinitely. Following an idle period, there is some overhead to wake up and reschedule the worker threads before the next inference. The OpenVINO documentation discusses oneTBB's worker scheduling behavior and its interaction with application threads. :contentReference[oaicite:0]{index=0}

2. CPU power management

   Modern CPUs may enter lower power states (C-states/P-states) during the 1-second sleep. The next inference then pays the cost of ramping the CPU back to its operating frequency. This often appears as an increased latency only on the first inference after an idle period.

3. Caches

   During continuous inference, caches remain "warm". After an idle period, some cache state may be lost, so the first inference incurs additional memory/cache misses before subsequent inferences return to the steady-state latency.

One thing I'd be interested in is:

• Which device are you using (CPU, GPU, NPU)?
• Which OpenVINO version?
• Are you calling InferRequest::infer() synchronously, or using start_async()?

If this is the CPU plugin, you could also try compiling the model with the latency hint:

ov::Core core;

auto compiled = core.compile_model(
    model,
    "CPU",
    ov::hint::performance_mode(ov::hint::PerformanceMode::LATENCY)
);

The latency hint configures the runtime for latency-oriented execution and is generally preferred over manually tuning low-level threading properties. :contentReference[oaicite:1]{index=1}

One experiment that would help narrow it down is to vary the sleep duration:

• sleep(10 ms)
• sleep(100 ms)
• sleep(500 ms)
• sleep(1 s)
• sleep(5 s)

If the latency penalty starts appearing only after some idle threshold, that would strongly suggest thread wake-up and/or CPU power management rather than any model-specific behavior.

I couldn't find documentation stating that OpenVINO intentionally "parks" inference threads after one second of inactivity, so I'm not sure whether the observed latency is entirely due to oneTBB scheduling, CPU power states, or a combination of both.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.