Add metric to track inflight visibility tasks

[rodrigozhou]

What changed?

Added metric to track number of inflight visibility tasks.

Why?

Help debug bottlenecks in the visibility task flow: task queue or bulk processor.

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

Potential risks

[yycptt]

Can you provide some context for this metric? I mean it will limited by the visibility task gorouting pool size.

FWIW, we already have pending_tasks metric and can be scoped to visibility queue, though it's a histogram for pending task count across shards.

[rodrigozhou]

IIUC, pending_tasks included all tasks that were load into memory, ie., not only the tasks dispatched to the executor. I wanted to track tasks that are in the executor to check if we are exhausting the history.visibilityProcessorSchedulerWorkerCount value. Basically, help diagnose where the bottleneck is in visibility: task queue or bulk processor.