Fix potential OSError when removing log file on FUSE device

[ideal]

In some cases when we write logs to directory mounted with FUSE device using OSS (or AWS S3) + Fluid, here we may encounter: OSError: [Errno 34] Numerical result out of range, but the log file is removed successfully actually.

[Delgan]

Thanks for the PR. However, it looks to me like something that should be fixed upstream, either in your FUSE implementation or in CPython. If os.remove() raises an exception, I'd rather not silently ignore it, especially because this risks shadowing legitimate errors from others filesystems.

[ideal]

The problem is that if we add logger with compression="gz", it throws an OSError in atexit callback, do we have a better way here?

[Delgan]

The problem is that if we add logger with compression="gz", it throws an OSError in atexit callback, do we have a better way here?

I do not know what is causing this on your specific filesystem. If you cannot pinpoint exactly the source of error, but are confident os.remove() are harmless and can be ignored, I suggest implementing your own custom compression function. It will work the same way as Loguru does natively, but you'll have more control over how the files are handled.

from loguru import logger
import gzip
import shutil
import os


def compression(file_path):
    gz_path = file_path + ".gz"
    with open(file_path, "rb") as f_in:
        with gzip.open(gz_path, "wb") as f_out:
            shutil.copyfileobj(f_in, f_out)

    # In some cases when we write logs to directory mounted with
    # FUSE device using OSS (or AWS S3) + Fluid, here we may encounter:
    # OSError: [Errno 34] Numerical result out of range,
    # but the log file is removed successfully actually
    try:
        os.remove(file_path)
    except OSError:
        pass


logger.remove()
logger.add("file.log", compression=compression)

logger.info("Hello, World!")

Note: make sure your program isn't inadvertently logging to the same file from multiple child process at the same time. This is a known source of errors while using multiprocessing which causes conflicts and race conditions, due to duplicated handlers trying to write/move/erase the same file in parallel. See the documentation for more details.

[ideal]

I do not know what is causing this on your specific filesystem. If you cannot pinpoint exactly the source of error, but are confident os.remove() are harmless and can be ignored, I suggest implementing your own custom compression function. It will work the same way as Loguru does natively, but you'll have more control over how the files are handled.

OK, Thanks, I will try soon.

[ideal]

Related PR: datajuicer/data-juicer#980