Timeout while restore from google storage bucket

[marcuslinke]

Project board link

The following exception is thrown with latest medusa version 0.22.3 while trying to restore from a remote backup stored in google cloud storage. Is there any possibility to increase/customize the timeout or any other workaround available?

 |  medusa-restore Traceback (most recent call last):                                                                                                                                                                                                    │
│ medusa-restore   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main                                                                                                                                                               │
│ medusa-restore     return _run_code(code, main_globals, None,                                                                                                                                                                                        │
│ medusa-restore   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code                                                                                                                                                                          │
│ medusa-restore     exec(code, run_globals)                                                                                                                                                                                                           │
│ medusa-restore   File "/home/cassandra/medusa/service/grpc/restore.py", line 120, in <module>                                                                                                                                                        │
│ medusa-restore     output_message = restore_backup(in_place, config)                                                                                                                                                                                 │
│ medusa-restore   File "/home/cassandra/medusa/service/grpc/restore.py", line 99, in restore_backup                                                                                                                                                   │
│ medusa-restore     medusa.restore_node.restore_node(config, tmp_dir, backup_name, in_place, keep_auth,                                                                                                                                               │
│ medusa-restore   File "/home/cassandra/medusa/restore_node.py", line 50, in restore_node                                                                                                                                                             │
│ medusa-restore     restore_node_locally(config, temp_dir, backup_name, in_place, keep_auth, seeds, storage,                                                                                                                                          │
│ medusa-restore   File "/home/cassandra/medusa/restore_node.py", line 90, in restore_node_locally                                                                                                                                                     │
│ medusa-restore     download_data(config.storage, node_backup, fqtns_to_restore, destination=download_dir)                                                                                                                                            │
│ medusa-restore   File "/home/cassandra/medusa/download.py", line 56, in download_data                                                                                                                                                                │
│ medusa-restore     storage.storage_driver.download_blobs(srcs, dst)                                                                                                                                                                                  │
│ medusa-restore   File "/home/cassandra/medusa/storage/abstract_storage.py", line 150, in download_blobs                                                                                                                                              │
│ medusa-restore     loop.run_until_complete(self._download_blobs(srcs, dest))                                                                                                                                                                         │
│ medusa-restore   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete                                                                                                                                                  │
│ medusa-restore     return future.result()                                                                                                                                                                                                            │
│ medusa-restore   File "/home/cassandra/medusa/storage/abstract_storage.py", line 158, in _download_blobs                                                                                                                                             │
│ medusa-restore     await asyncio.gather(*coros)                                                                                                                                                                                                      │
│ medusa-restore   File "/home/cassandra/medusa/storage/google_storage.py", line 166, in _download_blob                                                                                                                                                │
│ medusa-restore     chunk = await stream.read(DOWNLOAD_STREAM_CONSUMPTION_CHUNK_SIZE)                                                                                                                                                                 │
│ medusa-restore   File "/home/cassandra/.venv/lib/python3.10/site-packages/gcloud/aio/storage/storage.py", line 140, in read                                                                                                                          │
│ medusa-restore     chunk = await self._response.content.read(size)                                                                                                                                                                                   │
│ medusa-restore   File "/home/cassandra/.venv/lib/python3.10/site-packages/aiohttp/streams.py", line 393, in read                                                                                                                                     │
│ medusa-restore     await self._wait("read")                                                                                                                                                                                                          │
│ medusa-restore   File "/home/cassandra/.venv/lib/python3.10/site-packages/aiohttp/streams.py", line 311, in _wait                                                                                                                                    │
│ medusa-restore     with self._timer:                                                                                                                                                                                                                 │
│ medusa-restore   File "/home/cassandra/.venv/lib/python3.10/site-packages/aiohttp/helpers.py", line 735, in __exit__                                                                                                                                 │
│ medusa-restore     raise asyncio.TimeoutError from None

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: MED-117

[marcuslinke]

Seems related to k8ssandra/k8ssandra-operator#1353 and #763 (comment)

[marcuslinke]

After manually editing the medusa ConfigMap (containing medusa.ini) and setting read_timeout = 600 as a workaround everything works as expected. When the new medusa version containing the fix (#813) will be released?