[Support Ticket]: bitswap stops responding after sector is tried but not found

[bobdubois]

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• booster-http
• booster-bitswap
• LID Database - Yugabyte/LevelDB
• boostd-data
• Other

Boost Version

boostd version 2.1.0-rc1+mainnet+git.0cd9d5d.dirty

Describe the problem

Bitswap stops working after a sector is tried but not found. No other logs found, retrievals work just before that but then no more. The service runs without other errors but retrievals time out. Sector 4011 is NOT available on the system, so shouldn't even be asked for.
Restarting booster-bitswap solves the issue but is not workable.

Logging Information

booster-bitswap log:
2023-11-08T03:04:13.691+0100	INFO	remote-blockstore	remoteblockstore/remoteblockstore.go:68	Get failed	{"cid": "bafkreidrspvkbkkwugvvf4peo3fx6d3u2ac4taiz3lrqwr52alql5mpr7a", "error": "1 error occurred:\n\t* getting piece reader: 1 error occurred:\n\t* getting reader over sector 4011: sector is not unsealed\n\n\n\n"}
2023-11-08T03:04:13.691+0100	ERROR	engine	decision/blockstoremanager.go:121	blockstore.Get(bafkreidrspvkbkkwugvvf4peo3fx6d3u2ac4taiz3lrqwr52alql5mpr7a) error: 1 error occurred:
	* getting piece reader: 1 error occurred:
	* getting reader over sector 4011: sector is not unsealed

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

[bobdubois]

and it's not just 1 sector:
2023-11-08T08:54:02.571+0100 INFO remote-blockstore remoteblockstore/remoteblockstore.go:68 Get failed {"cid": "bafkreigxgnwwhkv3osgskxtjpcuqzvp53pry5gcndnzr45pef3kwgzudwm", "error": "1 error occurred:\n\t* getting piece reader: 1 error occurred:\n\t* getting reader over sector 4010: sector is not unsealed\n\n\n\n"}
2023-11-08T08:54:02.571+0100 ERROR engine decision/blockstoremanager.go:121 blockstore.Get(bafkreigxgnwwhkv3osgskxtjpcuqzvp53pry5gcndnzr45pef3kwgzudwm) error: 1 error occurred:
* getting piece reader: 1 error occurred:
* getting reader over sector 4010: sector is not unsealed

it retries the sector multiple times, bitswap doesn't crash on the first occurrence of this log entry.

[bobdubois]

Boost announces to IPNI before handing off to the sealing pipeline. What if the sector does not get sealed (like in this case) and IPNI is not aware? Is this the reason this happens? Still bitswap should choke on it.

[bobdubois]

Graphsync retrievals stopped working as well. No logs found in boostd.log though that are related.

[ischasny]

Seems to have been resolved by updating to RC3.