need to improve monitoring HTTP APIs for health/diagnostics

[plebhash]

currently, on all apps /api/v1/health naively returns ok regardless of application state, which can be a bit misleading UX... we should start thinking about:

• which abstractions should be introduced to represent edge-case application states
• how to encode edge-case application states as HTTP responses

this also has some conceptual overlap with a potential "diagnostics" API endpoint that we've discussed in the past (for which I couldn't find a tracker issue)

as discussed in #560 one possible edge-case for Pool and JDC is when TP is still doing IBD (although this would only work for Bitcoin Core v31+)... although there could be more

overall for now this is just a high level tracker, this probably deserves deeper brainstorming for the ideal approach

---

perhaps we should rename the endpoint that's currently named as health into something like alive or responsive... so it can always return ok as a way to signal the app is running

and then we can add another endpoint (called health or diagnostics) which will return data with more fine-grained details

[plebhash]

brainstorming with @lucasbalieiro during vinteum OH about this

we looked into this edge case:

• Add PyBlock as a solo pool, fix Public Pool port, rm duplicate text sv2-ui#203 (comment)
• relax legacy solo payout verification in tProxy #589

assuming a malicious solo pool is paying < 90% to the user, we expect tProxy to trigger fallback/shutdown

so we were thinking about how sv2-ui would signal that to the user

at a first glance, my mind came to this issue thinking that a diagnostics endpoint could help sv2-ui show that

but then @lucasbalieiro reminded me that assuming there's no secondary pool, there's no fallback but shutdown

so there's no diagnostics endpoint to probe, and we're back at square 0... so it might be the case that the only viable approach is for sv2-ui to parse logs? e.g.: https://github.com/stratum-mining/sv2-ui/blob/42ae72ac23dee084bb03b63cda77d0f2ef878e04/server/src/logs/parsers.ts#L9-L28

cc @GitGab19

[plebhash]

(first, let's assume we rename /api/v1/health to /api/v1/responsive)

we did some more brainstorming on this during dev call today

the main takeaway was that maybe shutdown should no longer mean "stop the process at an OS level"

perhaps, whenever shutdown is triggered (either via fallback, API errors or other edge cases), it should stop all subcomponents internally and put the application into a state where:

• /api/v1/responsive returns ok
• /api/v1/diagnostics returns the reason for shutdown

this also has some synergy with the work that's being done on:

• need to refactor PoolSv2::start #528
• need to refactor JobDeclaratorClient::start #526
• need to refactor TranslatorSv2::start #527

more specifically, the pattern that's being established on the issues above consists of the introduction of:

• PoolRuntime<State>
• JdcRuntime<State>
• TranslatorRuntime<State>

the specific ways that this would be leveraged for /api/v1/diagnostics are yet to be discussed... just taking note of the brainstorming we did today

[GitGab19]

Cross-referencing a comment on sv2-ui, which is about exposing the information related to the upstream where a client is connected to via APIs: stratum-mining/sv2-ui#211 (comment)

[GitGab19]

Cross-referencing another issue on sv2-ui which requires some new information from the monitoring APIs: stratum-mining/sv2-ui#129

This has been partially discussed above, since it involves fallback and eventually fallback to solo mining.

The only sustainable solution to that seems to be what @plebhash described above: #577 (comment)