Unable to build on macOS with Apple Clang 12

[adamjstewart]

I'm trying to build the numcodecs Python library, which has a git submodule for c-blosc. I see the following build error on macOS:

/Users/Adam/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zstd-1.4.4 -Ic-blosc/internal-complibs/lz4-1.9.2 -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.4.4/compress -Ic-blosc/internal-complibs/zstd-1.4.4/dictBuilder -Ic-blosc/internal-complibs/zstd-1.4.4/decompress -Ic-blosc/internal-complibs/zstd-1.4.4/legacy -Ic-blosc/internal-complibs/zstd-1.4.4/common -Ic-blosc/internal-complibs/zstd-1.4.4/dll -Ic-blosc/internal-complibs/zstd-1.4.4/deprecated -I/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/python-3.8.7-3auqbnqcq3ti45v6unf4rcyrfkvwmnvo/include/python3.8 -c c-blosc/internal-complibs/zlib-1.2.8/gzread.c -o build/temp.macosx-10.15.7-x86_64-3.8/c-blosc/internal-complibs/zlib-1.2.8/gzread.o -msse2 -stdlib=libc++ -DSHUFFLE_SSE2_ENABLED
c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: error: implicit declaration of function 'read' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = read(state->fd, buf + *have, len - *have);
              ^
c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: note: did you mean 'fread'?
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/stdio.h:158:9: note: 'fread' declared here
size_t   fread(void * __restrict __ptr, size_t __size, size_t __nitems, FILE * __restrict __stream);
         ^
c-blosc/internal-complibs/zlib-1.2.8/gzread.c:591:11: error: implicit declaration of function 'close' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    ret = close(state->fd);
          ^
2 errors generated.
[numcodecs] command '/Users/Adam/spack/lib/spack/env/clang/clang' failed with exit status 1

Apple Clang 12 (the current release) is more strict than previous releases. This can be solved with the following patch:

diff --git a/internal-complibs/zlib-1.2.8/gzguts.h b/internal-complibs/zlib-1.2.8/gzguts.h
index d87659d..f6823ed 100644
--- a/internal-complibs/zlib-1.2.8/gzguts.h
+++ b/internal-complibs/zlib-1.2.8/gzguts.h
@@ -26,6 +26,7 @@
 #  include <limits.h>
 #endif
 #include <fcntl.h>
+#include <unistd.h>

 #ifdef _WIN32
 #  include <stddef.h>

Idea comes from https://stackoverflow.com/questions/15296067/how-to-handle-a-warning-from-the-clang-compiler

Would it be possible to update from zlib 1.2.8 to 1.2.11? Not sure if this will fix it or not, you may also need to run configure. Ideally, I could simply use an external zlib installation since I don't want to be stuck with an older broken version of zlib.

[adamjstewart]

Also pinging @alimanfoo and @jakirkham from the numcodecs repo as it's not clear exactly who's fault this is. Even if it's a c-blosc bug, numcodecs will need to update their submodule once it is fixed.

[FrancescAlted]

Hi @adamjstewart . Just updated to latest MacOSX Big Sur 11.2.2 and Apple clang 12:

faltet@Francescs-MacBook-Pro ~/b/c/build (master)> clang -v                                                        (base)
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

But I can compile C-Blosc without problems.

Which version of C-Blosc library are you using?

[FrancescAlted]

I have also tried out zlib 1.2.11, but it needs an additional configure step, which is new wrt 1.2.8, and the current configuration method is not prepared for that. If you want to create a PR that uses zlib 1.2.11, I am open to merge it.

Another possibility is to try out zlib-ng which is a modern re-implementation of zlib and that is almost in production state. Maybe it would be nice to experiment with it now and adopt it as soon as it is declared 'production ready'.

[adamjstewart]

Yes, strangely enough c-blosc builds, but numcodecs (which has c-blosc as a vendored git submodule) doesn't build. Maybe numcodecs isn't running the cmake script correctly?

[FrancescAlted]

Ok. Looks like the issue should be in numcodecs. I am going to close this, but feel free to reopen it if necessary.

[adamjstewart]

It looks like this has already been reported to the numcodecs devs: zarr-developers/numcodecs#269