Testnet faucet failing to provision wallet

[toliaqat]

Emerynet intermittently experiences issues with wallet provisioning, which can be traced back to two main causes:

1. Connection Failure with the Emerynet Node: The faucet server logs frequently capture an error message indicating a failure to connect with the Emerynet node:

Error: post failed: Post "http://localhost:26657": EOF

This error may appear when the RPC node is not accepting requests, possibly because the node is being upgraded, is catching up, or is down due to some other issue.

2. Timeouts: Another common error observed in the server logs pertains to timeouts:

 RPC error -32603 - Internal error: timed out waiting for tx to be included in a block

This issue warrants further investigation to understand why the RPC node could not include the transaction. A prevailing theory suggests that the validator node might be intermittently hitting its resource limits (CPU, memory, bandwidth), which could sporadically prevent it from processing or broadcasting transactions promptly. However, other underlying causes for this error message should also be explored.

For further investigation:
The faucet server logs are available at /state/server.log
The faucet code is located at https://github.com/Agoric/instagoric/tree/main/bases/shared/instagoric-server