Describe the bug
Following this solved issue #1003, 2 new issues have been detected:
- DA light node connection can be idle from time to time. I got 2 times and one time by a partner. The Follower node DA connection stays open but no blob or heartbeat arrives. The connection stays idle until the node is restarted. When the node restarts, it syncs to the leader and returns to the right state.
- Leader node update generates an incorrect state in the followers. 3 times after updating the leader while the follower is syncing, after the update the follower stays in an incorrect state. The followers get all the blob from the leader but the block and ledget height stay behind. They can not sync to the leader even if the get all the blob. If I restore the node from a DB done before the leader update, it doesn't work, the follower always stays behind. To solve the issue, a new DB backup must be done after the update and the follower node restore this DB. The main reason is that during the update, the node executes blocks with a different version of the leader node, so it generates a different state. I didn't test doing only a stop/start of the leader node.
To Reproduce
For the idle, it arrives from time to time. There's no procedure to reproduce.
For the behind state, do an update of the leader while the follower node is connected.
Expected behavior
A clear and concise description of what you expected to happen.
Proposed solution
For the idle connection, I see 2 possibilities:
- Monitor the heartbeat and it none arrive for some time send an alert
- monitored the state between the follower and the leader and detect when the diff become too important. In this case, send an alert.
The solution must be compatible with the partner follower node use.
For the block execution with a different version. Some more test must be done to confirm the cause. What is sure is that a follower must execute the block with the same version as the leader. So the block notification must contain more information to detect a different version and a different state after execution.
A more general update can be to change the stream of DA blob to a stream of executed state changes from the leader. In this case, the follower never executes blocks but only updates its state with the leader one.
Describe the bug
Following this solved issue #1003, 2 new issues have been detected:
To Reproduce
For the idle, it arrives from time to time. There's no procedure to reproduce.
For the behind state, do an update of the leader while the follower node is connected.
Expected behavior
A clear and concise description of what you expected to happen.
Proposed solution
For the idle connection, I see 2 possibilities:
The solution must be compatible with the partner follower node use.
For the block execution with a different version. Some more test must be done to confirm the cause. What is sure is that a follower must execute the block with the same version as the leader. So the block notification must contain more information to detect a different version and a different state after execution.
A more general update can be to change the stream of DA blob to a stream of executed state changes from the leader. In this case, the follower never executes blocks but only updates its state with the leader one.