Problems with IPNIsync

[cryptowhizzard]

Checklist

• This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to boost.

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• Other

Boost Version

Latest master of 13 nov 2023

Describe the Bug

Boost generates an error regenerated link from multihash lister did not match the original link; multihashes returned by the lister for the same key are not consistent causing IPNIsync to halt on Indexer side.

Logging Information

2023-11-13T12:46:10.588Z	DEBUG	index-provider-wrapper	indexprovider/wrapper.go:575	returning piece iterator	{"propCid": "bafyreicpfl4qsu4y723rhdu3njkrvvx3uh74tlpxmwr56v4mzehphz242e", "piece": "baga6ea4seaqkljwlwpcjsyol3bxudck5wvzwo3zsnxfocg6ctbgk3xdlljyseiy", "err": null}
2023-11-13T12:46:17.534Z	ERROR	provider/engine	engine/linksystem.go:129	Regeneration of entries link from multihash iterator did not match the original link. Check that multihash iterator consistently returns the same entries for the same key.	{"want": "baguqeera4v4fm56ddi4hrytmkwn2e64hd6wgwmcze4s6fph36ldv55gsvnya", "got": "baguqeeraclox2bkja373gehvxzflg2nvere6dtjagf4arhfdlzlq6mas57gq"}
2023-11-13T12:46:17.535Z	ERROR	dagsync/ipnisync	ipnisync/publisher.go:235	Failed to load requested block	{"err": "regenerated link from multihash lister did not match the original link; multihashes returned by the lister for the same key are not consistent", "cid": "baguqeera4v4fm56ddi4hrytmkwn2e64hd6wgwmcze4s6fph36ldv55gsvnya"}

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

[cryptowhizzard]

Apart from this problem, there is a second problem emerging.

Not all deals are getting announced to the network indexer.

error while ingesting ad baguqeerazpknqo4o4ok6oax7tmpib4ebekuahg2i35gvubbpex7rwy55chhq: syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeeramden5stz3kbgzb7abpl3kastbenkruh36uo372lhrv5mrpek4rta: failed to fetch block for cid baguqeeramden5stz3kbgzb7abpl3kastbenkruh36uo372lhrv5mrpek4rta: fetch request failed: Get "http://212.6.53.92:3104/ipni/v1/ad/baguqeeramden5stz3kbgzb7abpl3kastbenkruh36uo372lhrv5mrpek4rta": net/http: request canceled (Client.Timeout exceeded while awaiting headers) (19h ago)

These timeout are blocking further ingestion of ads.

[masih]

Latest master of 13 nov 2023

Can you provide the concrete commit ID please?

[cryptowhizzard]

To provide some more context:

__Edit removed the screenshot due sensitive data. Please refer to the ASCII below.

This example has not made it to the indexer. Checking bafkreigjnu72cdjpzfjs55xurvif2dqip62zfbwpypohcvpe4kgettvtc4 on cid.contact gives no record for the machine with ip ending on 53.92.

[cryptowhizzard]

In ASCII for the record:

Deal 3e500432-453f-4ea8-9b6c-75c31351d831

CreatedAt | 2023-11-12 21:50:01.830 (3 days ago) -- | -- f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i 12D3KooWF3msa2qui4j5TMcK4ua7ebTscGGXmZSeG1UizLcWVnqM bafyreiak6opod5caczdl7kgtz5vl2beypxngzx7q4gz4rumonjbd2o5an4 bafkreigjnu72cdjpzfjs55xurvif2dqip62zfbwpypohcvpe4kgettvtc4 Yes Yes Yes baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa 32 GiB (34,359,738,368 bytes) 0 atto 0.007 FIL 0 atto 3,391,587 3,404,020 (in 4 days) 4,918,900 (in a year) 1,514,880 (526 days) 0 atto (Price per epoch x Piece Size x Duration) Online http 16.5 GiB (17,723,430,495 bytes) URL: https://XXXXXXXXXdataset.twinquasar.io/nasanex/baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa.car 0 B (0 bytes) /root/.boost/incoming/3e500432-453f-4ea8-9b6c-75c31351d831.download Yes 38763 0 34,359,738,368 bafy2bzacebyqozi2pcto3kr3k33egtka6qq4ukht4htujktsp2mb6f2cvkdkm 62,966,623 IndexedAndAnnounced (3 days ago) Sealer: Proving

Deal 3e500432-453f-4ea8-9b6c-75c31351d831
CreatedAt 2023-11-12 21:50:01.830 (3 days ago)
Client Address f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i
Client Peer ID 12D3KooWF3msa2qui4j5TMcK4ua7ebTscGGXmZSeG1UizLcWVnqM
Signed Proposal CID bafyreiak6opod5caczdl7kgtz5vl2beypxngzx7q4gz4rumonjbd2o5an4
Label bafkreigjnu72cdjpzfjs55xurvif2dqip62zfbwpypohcvpe4kgettvtc4
Verified Yes
Keep Unsealed Copy Yes
Announce To IPNI Yes
Piece CID baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa
Piece Size 32 GiB (34,359,738,368 bytes)
Client Collateral 0 atto
Provider Collateral 0.007 FIL
Storage Price / epoch / GiB 0 atto
Current Epoch 3,391,587
Start Epoch 3,404,020 (in 4 days)
End Epoch 4,918,900 (in a year)
Duration 1,514,880 (526 days)
Storage Fee 0 atto (Price per epoch x Piece Size x Duration)
Transfer Mode Online
Transfer Type http
Transfer Size 16.5 GiB (17,723,430,495 bytes)
Transfer Params
URL: https://XXXXXXXXXX.twinquasar.io/nasanex/baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa.car
Transferred 0 B (0 bytes)
Inbound File Path /root/.boost/incoming/3e500432-453f-4ea8-9b6c-75c31351d831.download
Delete After Add Piece Yes
Sector ID 38763
Sector Data Offset 0
Sector Data Length 34,359,738,368
Publish Message CID bafy2bzacebyqozi2pcto3kr3k33egtka6qq4ukht4htujktsp2mb6f2cvkdkm
Chain Deal ID 62,966,623
Checkpoint IndexedAndAnnounced (3 days ago)
Status Sealer: Proving