[WIP] Replace DCGM_FI_PROF_GR_ENGINE_ACTIVE with DCGM_FI_DEV_GPU_UTIL

[empovit]

• Profiling metrics DCGM_FI_PROF_* aren't available on pre-Volta GPUs
• Consolidate the default metrics into the configmap so that installing the console plugin doesn't break other tools

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: empovit
Once this PR has been reviewed and has the lgtm label, please assign mareklibra for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[empovit]

/cc @mresvanis

[mresvanis]

II{U,R}C we chose DCGM_FI_PROF_GR_ENGINE_ACTIVE because it is available also when using MIG, while DCGM_FI_DEV_GPU_UTIL is not.

If we would like to drop MIG support for other NVIDIA GPU models, then no objection from my part (whatever makes more sense business-wise).

[empovit]

/hold until we test the changes with MIG-supported GPUs

[empovit]

There are two DCGM metrics for GPU utilization, depending on the GPU model/architecture:

1. DCGM_FI_PROF_GR_ENGINE_ACTIVE is the "modern" one, better reflects GPU utilization, and works with MIG. However, it does not work on older GPUs.
2. DCGM_FI_DEV_GPU_UTIL is the "old" metrics, works on older GPUs but does not work with MIG.

The entire Profiling group (Group ID 8, DCGM_FI_PROF_* metrics) is not supported on pre-Volta GPUs.