[dashboard] Remove ReportHead usage of DataSource.agents.

[rynewang]

ReportHead used to subscribe to DataSource.agents changes, to maintain a connection to every single O(#node) agents, just to make grpc calls when needed. This PR removes it, now it only make on demand connections when a profiling request comes.

Changes:

• No longer listen to DataSource.agents.
• On profiling request, make 1 rpc to InternalKV, to get IP and agent port. Then create a stub on the fly and use it to send outbound requests.
• Changed InternalKV:
  • key: DASHBOARD_AGENT_PORT_PREFIX -> DASHBOARD_AGENT_ADDR_PREFIX
  • value: json.dumps([http_port, grpc_port]) -> json.dumps([ip, http_port, grpc_port])
• Changed all profiling requests from giving param ip to give param node_id.
• Added NodeID filter to GcsClient API.
• Moved updates of DataSource.node_physical_stats to NodeHead.

After this PR, ReportHead no longer needs DataSource.

Smoke tested UI locally.

[jjyao]

as long as you have a gcs client, you have all nodes cached, we can just use that?

[rynewang]

That thing only have value if you subscribe to node change:

ray/src/ray/gcs/gcs_client/accessor.h

Lines 398 to 399 in 0022380

	/// Note, the local cache is only available if `AsyncSubscribeToNodeChange`
	/// is called before.

a python side gcs client by default does not have that value.

[jjyao]

nit: given filters are optional we can do

virtual Status AsyncGetAll(
                             const MultiItemCallback<rpc::GcsNodeInfo> &callback,
                             int64_t timeout_ms,
                             std::optional<NodeID> node_id = std::nullopt);

so you don't need to update all callers.

[rynewang]

since we already changed, and there's only a few of them, let's just keep them

[rynewang]

you are right we need to do the defaults; cpp tests have a lot of callsites

[rynewang]

@alanwguo this PR changes some HTTP endpoints, and I have changed callsites in frontend accordingly. PTAL!

[jjyao]

LGTM

[jjyao]

seems in the code code it's not required?

[rynewang]

I think pid is required, otherwise we don't know what pid to traceback