use custom gzip compression to tolerate FUSE-mounted OSS/S3#980
Conversation
Loguru's built-in "gz" compression fails on K8s + Fluid OSS/S3 mounts because the post-compression unlink returns OSError errno 34 even though the source file is removed. Replace it with a custom callback that swallows that specific failure so log compression works under FUSE.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a custom gzip_compression function to replace the default loguru 'gz' compression, specifically addressing an issue where file removal fails on FUSE-mounted filesystems with a 'Numerical result out of range' error. The feedback suggests refining the exception handling in this new function by importing the errno module and explicitly checking for errno.ERANGE instead of catching all OSError instances, which ensures that other legitimate filesystem errors are not masked.
Swallowing all OSError can mask real failures (permission denied, disk full, etc.). Restrict the suppression to the specific errnos observed on FUSE-mounted OSS/S3 (errno 34 / ERANGE, plus ENOSYS) and re-raise anything else.
|
Thanks for the detailed fix and context. I agree this is a real issue, but the root cause appears to be the FUSE/Fluid mount returning On the Data-Juicer side, I’m a bit concerned that swallowing filesystem errors here (even if limited to I’d prefer to address this in the FUSE/Fluid layer if possible. |
2 participants
Loguru's built-in "gz" compression fails on K8s + Fluid OSS/S3 mounts because the post-compression unlink may returns OSError errno 34 even though the source file is removed. We replace it with a custom callback that swallows that specific failure so log compression works under FUSE.
e.g.:
