Skip to content

ContentUploaddService: Retry uploads#339

Open
mnaamani wants to merge 3 commits intoJoystream:mainfrom
mnaamani:retry-uploads
Open

ContentUploaddService: Retry uploads#339
mnaamani wants to merge 3 commits intoJoystream:mainfrom
mnaamani:retry-uploads

Conversation

@mnaamani
Copy link
Member

Some storage nodes that have a slow storage-squid that might be lagging a few blocks behind, will fail to accept upload of a dataobject if the object was created just before the upload attempt for example:

{"@timestamp":"2024-10-17T04:42:43.230Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920679 doesn't exist in storage bag {\"dynamic\":{\"channel\":60210}}"}
{"@timestamp":"2024-10-17T13:54:48.899Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920740 doesn't exist in storage bag {\"dynamic\":{\"channel\":65925}}"}
{"@timestamp":"2024-10-17T13:55:18.518Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920748 doesn't exist in storage bag {\"dynamic\":{\"channel\":65925}}"}
{"@timestamp":"2024-10-17T14:00:48.856Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920788 doesn't exist in storage bag {\"dynamic\":{\"channel\":65925}}"}
{"@timestamp":"2024-10-17T22:10:55.624Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920855 doesn't exist in storage bag {\"dynamic\":{\"channel\":59045}}"}

This PR simply adds a retry mechanism.

@mnaamani mnaamani changed the title Retry uploads ContentUploaddService: Retry uploads Oct 24, 2024
@Lezek123 Lezek123 mentioned this pull request Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant