Justification and finalisation fail if even one node encounters an error.

[ch4r10t33r]

Zeam logs:

+---------------------------------------------------------------+
  Connected Peers:    3
+---------------------------------------------------------------+
  Head Block Root:    0x3bba23d6504533e560b6ae6cb839df2d468e39d0a68e097b37c8a9ab77ec3ca2
  Parent Block Root:  0xcdbdedc0204e621c7c4f04223e3802ee7825b906bbdef547f26c93680eed0197
  State Root:         0x32dbc140d1a278b0fcf946ad5c38f09d834ebf7068eb1b387ca722cb3990f4a6
  Timely:             YES
+---------------------------------------------------------------+
  Latest Justified:   Slot      0 | Root: 0x5909f25c0cad61d89f249a4f04202e95e45411a5c3f748cbb081a370bc43d3de
  Latest Finalized:   Slot      0 | Root: 0x5909f25c0cad61d89f249a4f04202e95e45411a5c3f748cbb081a370bc43d3de
+===============================================================+

Dec-23 15:06:03.015 [info] (zeam): [validator] constructing attestation message for slot=267
Dec-23 15:06:03.252 [info] (zeam): [network] rust-bridge: Attempting reconnection to /ip4/46.224.135.177/udp/9001/quic-v1/p2p/16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv (attempt 1/5)
Dec-23 15:06:03.256 [error] (zeam): [network] rust-bridge: Failed to dial peer 16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv at /ip4/46.224.135.177/udp/9001/quic-v1: DialPeerConditionFalse(DisconnectedAndNotDialing)
Dec-23 15:06:03.260 [info] (zeam): [network] rust-bridge: Scheduling reconnection to peer /ip4/46.224.135.177/udp/9001/quic-v1/p2p/16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv (attempt 2/5) in 10s
Dec-23 15:06:03.268 [warning] (zeam): [network] rust-bridge: [reqresp] Request 253 timed out after 15s
Dec-23 15:06:03.289 [warning] (zeam): [network] network-0:: Received RPC error for request_id=253 protocol=/leanconsensus/req/lean_blocks_by_root/1/ssz_snappy code=408 from peer=16Uiu2HAm7TYVs6qvDKnrovd9m4vvRikc4HPXm1WyLumKSe5fHxBv(lantern_0)

Ream logs:

============================================================
REAM's CHAIN STATUS: Next Slot: 235 | Head Slot: 232
------------------------------------------------------------
Connected Peers:   4
------------------------------------------------------------
Head Block Root:   0xab2ab7442ac66f5cc76240d0e925d68a6adb139eab52d4a1f4b85cbf5edc88d1
Parent Block Root: 0xd938dfca9fc9b4bc9eafd11688c66e15135be0a58557b41dfac195fac14efd46
State Root:        0xb02447fe8eeb90a1dda5f226d104d4cf1864e1015a313009f777a2ce6548d749
------------------------------------------------------------
Latest Justified:  Slot 0 | Root: 0x5909f25c0cad61d89f249a4f04202e95e45411a5c3f748cbb081a370bc43d3de
Latest Finalized:  Slot 0 | Root: 0x5909f25c0cad61d89f249a4f04202e95e45411a5c3f748cbb081a370bc43d3de
============================================================
2025-12-23T15:03:54.228311Z  INFO ream_chain_lean::service: Processing attestation by Validator 3 slot=234 source_slot=0 target_slot=16
2025-12-23T15:03:54.229968Z  WARN libp2p_gossipsub::behaviour: Not publishing a message that has already been published message_id=c36fcbc705779d7fe9ee794b838b34d5a20a9492
2025-12-23T15:03:54.229995Z  WARN ream_p2p::network::lean: Publish attestation failed slot=234 error=Duplicate
2025-12-23T15:03:55.001492Z  INFO ream_validator_lean::service: Starting attestation phase: 1 validator(s) voting slot=235 tick=941
2025-12-23T15:03:55.005263Z  INFO ream_validator_lean::service: Building attestation data finished slot=235 head_slot=232 source_slot=0 target_slot=16

Qlean logs:

25.12.23 15:07:30.001092  worker.1         Info      ForkChoiceStore  Slot 289 started with time 1156
25.12.23 15:07:31.001557  worker.1         Info      ForkChoiceStore  For slot 289: head is 0xcc8a…50f9 @ 287, target is 0x8c6e…178f @ 16, source is 0x5909…d3de @ 0
25.12.23 15:07:31.008131  worker.1         Info      ForkChoiceStore  Produced vote for target 0x8c6e…178f @ 16
25.12.23 15:07:32.001644  worker.1         Info      ForkChoiceStore  Interval two of slot 289 at time 1158
25.12.23 15:07:32.080817  worker.1         Warning   Networking  Error processing vote for target 0xbf8c…c2ad @ 240: Invalid attestation
25.12.23 15:07:33.001250  worker.1         Info      ForkChoiceStore  Interval three of slot 289 at time 1159
25.12.23 15:07:34.001202  worker.1         Info      Timeline  Next slot is 291 in 3999ms
25.12.23 15:07:34.001412  worker.1         Info      ForkChoiceStore  Slot 290 started with time 1160
25.12.23 15:07:35.000930  worker.1         Info      ForkChoiceStore  For slot 290: head is 0xcc8a…50f9 @ 287, target is 0x8c6e…178f @ 16, source is 0x5909…d3de @ 0
25.12.23 15:07:35.007092  worker.1         Info      ForkChoiceStore  Produced vote for target 0x8c6e…178f @ 16
25.12.23 15:07:36.000572  worker.1         Info      ForkChoiceStore  Interval two of slot 290 at time 1162
25.12.23 15:07:37.000536  worker.1         Info      ForkChoiceStore  Interval three of slot 290 at time 1163
25.12.23 15:07:38.000667  worker.1         Info      Timeline  Next slot is 292 in 4000ms

Lighthouse logs:

Dec 23 15:08:27.023 INFO  Processing lean attestation                   slot: 303, validator_id: 1, is_from_block: false
Dec 23 15:08:27.051 INFO  Forwarding attestation to attestation service
Dec 23 15:08:27.051 INFO  Processing lean attestation                   slot: 303, validator_id: 3, is_from_block: false
Dec 23 15:08:28.061 INFO  Forwarding attestation to attestation service
Dec 23 15:08:28.062 INFO  Processing lean attestation                   slot: 54, validator_id: 1, is_from_block: false
Dec 23 15:08:29.063 INFO  CHAIN STATUS: Current Slot: 303 | Head Slot: 299 | Behind: 0  separator: ===============================================================, current_slot: 303, head_slot: 299, head_block_root: 0xd2dd24e874ae4a3cf4cbcf6da95c946247b5421fbe9786baff0713d6e19eb2b3, parent_block_root: 0xb5d97374ac68b80cbd41c18c604b1eedf2d3317aa6bc5d76781f9307cb40c80d, state_root: 0x83b5d3145ea5e765532f914bc6fa4293871d461c97d71fd2f6bfc53baeddaab2, justified_slot: 256, justified_root: 0xd71093a4e0fa2ecd4d8997444e04d546f39aecb64bced9770c99b66404da760f, finalized_slot: 225, finalized_root: 0x5b71ec85aaefb8d12cfdba2e6ff8f0fb65eee4344ab1ad39d5dd43ef203c6986
Dec 23 15:08:29.063 INFO  Interval 3: Accepting accumulated attestations  slot: 303
Dec 23 15:08:29.063 WARN  Failed to apply weight for promoted attestation  validator_id: 3, head_root: 0x08821f13ca1bfce0c1daaf1c9fe4415e13c8f288e04968927777166ce3485852, error: Block 0x08821f13ca1bfce0c1daaf1c9fe4415e13c8f288e04968927777166ce3485852 not found in cache or store
Dec 23 15:08:29.063 WARN  Failed to apply weight for promoted attestation  validator_id: 1, head_root: 0x63670705355974e1ee4b18dc121b4b6f1f8ea9542ffd0694d218d2b7ed6eb44c, error: Block 0x63670705355974e1ee4b18dc121b4b6f1f8ea9542ffd0694d218d2b7ed6eb44c not found in cache or store
Dec 23 15:08:29.063 WARN  Failed to apply weight for promoted attestation  validator_id: 2, head_root: 0x63670705355974e1ee4b18dc121b4b6f1f8ea9542ffd0694d218d2b7ed6eb44c, error: Block 0x63670705355974e1ee4b18dc121b4b6f1f8ea9542ffd0694d218d2b7ed6eb44c not found in cache or store
Dec 23 15:08:29.064 WARN  Failed to ensure candidate root exists in proto array  candidate: 0xd71093a4e0fa2ecd4d8997444e04d546f39aecb64bced9770c99b66404da760f, error: Block 0xd71093a4e0fa2ecd4d8997444e04d546f39aecb64bced9770c99b66404da760f not found in cache or store
Dec 23 15:08:30.066 INFO  Produced block                                slot: 304, proposer_index: 4, block_root: 0x98c51419ea23d9fab399f00c1e54aba3518486f100ac533eb9170ea49163f3f5, state_root: 0x370e1943c40cc192ada0c9de1cc17fb11b0793a20c9aebb6302544cf2892b758, parent_root: 0xd2dd24e874ae4a3cf4cbcf6da95c946247b5421fbe9786baff0713d6e19eb2b3
Dec 23 15:08:30.066 INFO  Interval 0: Accepting new attestations        slot: 304
Dec 23 15:08:30.066 WARN  Failed to ensure candidate root exists in proto array  candidate: 0xd71093a4e0fa2ecd4d8997444e04d546f39aecb64bced9770c99b66404da760f, error: Block 0xd71093a4e0fa2ecd4d8997444e04d546f39aecb64bced9770c99b66404da760f not found in cache or store
Dec 23 15:08:30.067 INFO  Publishing block to network                   slot: 304
Dec 23 15:08:31.007 INFO  Forwarding attestation to attestation service
Dec 23 15:08:31.007 INFO  Processing lean attestation                   slot: 304, validator_id: 2, is_from_block: false
Dec 23 15:08:31.020 INFO  Forwarding attestation to attestation service
Dec 23 15:08:31.020 INFO  Processing lean attestation                   slot: 304, validator_id: 3, is_from_block: false
Dec 23 15:08:31.020 INFO  Forwarding attestation to attestation service
Dec 23 15:08:31.020 INFO  Processing lean attestation                   slot: 304, validator_id: 1, is_from_block: false
Dec 23 15:08:32.491 INFO  Forwarding attestation to attestation service
Dec 23 15:08:32.491 INFO  Processing lean attestation                   slot: 303, validator_id: 0, is_from_block: false
Dec 23 15:08:32.493 INFO  Forwarding block to attestation service     
Dec 23 15:08:32.493 INFO  Forwarding attestation to attestation service
Dec 23 15:08:32.493 INFO  Processing lean block from network (on_block)  slot: 193, proposer_index: 3, block_root: 0xfb2f0bc61264521a0371a6e2fbff046a5c83516415b337233f92e8f79d766fae
Dec 23 15:08:32.493 INFO  Parent state missing, requesting block from peer  slot: 193, parent_root: 0xe2e8af8194cfcd20224a8709964f1b4e9fabe468215889f5ff9158d797a5eec3, peer: PeerId("16Uiu2HAkvi2sxT75Bpq1c7yV2FjnSQJJ432d6jeshbmfdJss1i6f")
Dec 23 15:08:32.493 INFO  Processing lean attestation                   slot: 193, validator_id: 2, is_from_block: false
Dec 23 15:08:32.508 INFO  Forwarding attestation to attestation service
Dec 23 15:08:32.508 INFO  Processing lean attestation                   slot: 110, validator_id: 2, is_from_block: false
Dec 23 15:08:32.508 INFO  Forwarding attestation to attestation service
Dec 23 15:08:32.508 INFO  Processing lean attestation                   slot: 110, validator_id: 3, is_from_block: false

Lantern logs:

2025-12-23 15:09:43.025 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.029 INFO [gossip] processed vote validator=1 slot=322 head=0x7795bdd79f4c140b43be3e2c2c679bc5c69ff86d9278ae321fef5eac0f33fbd9 target=0x8c6e10ca856dc8d6f1fde999fc668efc9c33fcdd4b328b1358fa90adc80a178f@16 source=0x5909f25c0cad61d89f249a4f04202e95e45411a5c3f748cbb081a370bc43d3de@0  validator=lantern_0 peer=16Uiu2HA..
2025-12-23 15:09:43.043 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.043 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.043 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.043 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.043 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.044 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.044 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.044 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.044 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.044 WARN [gossipsub] rpc decode error from peer (rc=-7)
2025-12-23 15:09:43.095 INFO [gossip] received block slot=203 proposer=3 root=0x58ad0ad5ad0a42c5e10018534fc6f45dcb19480913c48944e094c3c1cd5bb390 source=gossip  validator=lantern_0 peer=16Uiu2HA..
2025-12-23 15:09:43.109 INFO [gossip] rejected vote validator=2 slot=58 head=0xc673dd1d7757e6a7904e6c79654cd6d30e0be9ee8daee02f8df4f9ae01c2ed04 target=0x74a9c6b8232f903af042cdd658a08436f1b6bf70cbb7e18d969b1a6607c64f08@42 source=0x5909f25c0cad61d89f249a4f04202e95e45411a5c3f748cbb081a370bc43d3de@0 reason=unknown target root=0x74a9c6b8232f903af042cdd658a08436f1b6bf70cbb7e18d969b1a6607c64f08 slot=42  validator=lantern_0 peer=16Uiu2HA..
2025-12-23 15:09:43.109 INFO [gossip] rejected vote validator=0 slot=141 head=0x9abfddea45796d928ad1313dfe0ddb7a76a40737b652f777776cefaa4650afc1 target=0xdb6f4d1a487a758df5c3d6b7df6e9c32cf7e50010faa69f3df985be0c7a301a8@100 source=0x5909f25c0cad61d89f249a4f04202e95e45411a5c3f748cbb081a370bc43d3de@0 reason=unknown target root=0xdb6f4d1a487a758df5c3d6b7df6e9c32cf7e50010faa69f3df985be0c7a301a8 slot=100  validator=lantern_0 peer=16Uiu2HA..

[KatyaRyazantseva]

Even though the head is growing, finality is being lost with 3 clients running continuously and one client being restarted.

[KatyaRyazantseva]

From the clients logs I see that they don't receive attestations from Lighthouse. But the chain is configured as 5 validators. When Zeam gets disconnected, other clients have only 3/5 votes which is 0.6, less than 0.66. Even if Lighthouse stays connected with other peers, the total number of attestations is not enough for finalization.

Ream's logs on receiving LH gossip message:

2025-12-23T23:30:15.067118Z  WARN libp2p_gossipsub::protocol: Signature field was non-empty and anonymous validation mode is set
2025-12-23T23:30:15.067249Z  WARN libp2p_gossipsub::behaviour: Invalid message from peer. Reason: SignaturePresent peer=16Uiu2HAmTrYwvBDY3GLEv5zR48ZfBwCYyKgmuLTDsCdBJUo127HY source=None

I checked other clients' logs. Lighthouse publishes the blocks and attestations (rarely), but other clients don't receive them.

Lantern logs on receiving the attestation from Lighthouse:

2025-12-23 18:04:40.015 INFO [gossip] rejected vote validator=4 slot=0 head=0x6691eb3ea2573128e6d942555072d9e15b5d239de6656fbd28d8ce16404e3738 target=0x6691eb3ea2573128e6d942555072d9e15b5d239de6656fbd28d8ce16404e3738@0 source=0x0@0 reason=source checkpoint root zero slot=0 root=0x0  validator=lantern_0 peer=16Uiu2HA..

Lighthouse logs:

Dec 23 18:04:55.028 INFO  Produced block                                slot: 4, proposer_index: 4, block_root: 0x6d78f7867e146068865ee5dfbd2912d0dcca7d806ea64c208b61392ea90254ff, state_root: 0xad2ef31b375de11db49abc52c3efa84cfcf78d56277d719f0b4fde88277ca2c8, parent_root: 0x6691eb3ea2573128e6d942555072d9e15b5d239de6656fbd28d8ce16404e3738
Dec 23 18:04:55.028 INFO  Interval 0: Accepting new attestations        slot: 4
Dec 23 18:04:55.028 INFO  Publishing block to network                   slot: 4
...
Dec 23 18:04:40.011 INFO  Publishing attestation to network             slot: 0, validator_id: 4

Lighthouse can't accept other clients attestations:

Dec 23 18:04:54.026 INFO  Interval 3: Accepting accumulated attestations  slot: 3
Dec 23 18:04:54.026 WARN  Failed to apply weight for promoted attestation  validator_id: 3, head_root: 0x639940e5daeec5a55dfe3911c6f3ca4b00d3a1c33ad1252a75d4f6bdd747f095, error: Block 0x639940e5daeec5a55dfe3911c6f3ca4b00d3a1c33ad1252a75d4f6bdd747f095 not found in cache or store
Dec 23 18:04:54.026 WARN  Failed to apply weight for promoted attestation  validator_id: 0, head_root: 0x2cd76b5b27d67603fe56b3ff043d895079df10f394ae028f25f2f4e2f87dda57, error: Block 0x2cd76b5b27d67603fe56b3ff043d895079df10f394ae028f25f2f4e2f87dda57 not found in cache or store
Dec 23 18:04:54.026 WARN  Failed to apply weight for promoted attestation  validator_id: 1, head_root: 0x2cd76b5b27d67603fe56b3ff043d895079df10f394ae028f25f2f4e2f87dda57, error: Block 0x2cd76b5b27d67603fe56b3ff043d895079df10f394ae028f25f2f4e2f87dda57 not found in cache or store
Dec 23 18:04:54.026 WARN  Failed to apply weight for promoted attestation  validator_id: 2, head_root: 0x2cd76b5b27d67603fe56b3ff043d895079df10f394ae028f25f2f4e2f87dda57, error: Block 0x2cd76b5b27d67603fe56b3ff043d895079df10f394ae028f25f2f4e2f87dda57 not found in cache or store
D

My suggestion is to restart the devnet of 4 clients excluding Lighthouse and check if we still have finalization after shutting down Zeam.