Different libc.a for macOS arm64 and Linux x86_64 #23379

eagleoflqj · 2025-01-13T03:10:01Z

I'm testing reproducibility by compiling on Linux and macOS fcitx-contrib/fcitx5-js#18. Everything works fine except for .wasm. A wasm2wat and diff produces below and more

317,326c317,326
<   (import "wasi_snapshot_preview1" "environ_sizes_get" (func (;139;) (type 1)))
<   (import "wasi_snapshot_preview1" "environ_get" (func (;140;) (type 1)))
<   (import "env" "__syscall_fcntl64" (func (;141;) (type 4)))
<   (import "env" "__syscall_ioctl" (func (;142;) (type 4)))
<   (import "env" "__syscall_openat" (func (;143;) (type 6)))
<   (import "wasi_snapshot_preview1" "fd_close" (func (;144;) (type 0)))
<   (import "wasi_snapshot_preview1" "fd_read" (func (;145;) (type 6)))
<   (import "wasi_snapshot_preview1" "fd_write" (func (;146;) (type 6)))
<   (import "env" "_tzset_js" (func (;147;) (type 8)))
<   (import "wasi_snapshot_preview1" "proc_exit" (func (;148;) (type 3)))
---
>   (import "wasi_snapshot_preview1" "proc_exit" (func (;139;) (type 3)))
>   (import "wasi_snapshot_preview1" "environ_sizes_get" (func (;140;) (type 1)))
>   (import "wasi_snapshot_preview1" "environ_get" (func (;141;) (type 1)))
>   (import "env" "__syscall_fcntl64" (func (;142;) (type 4)))
>   (import "env" "__syscall_ioctl" (func (;143;) (type 4)))
>   (import "env" "__syscall_openat" (func (;144;) (type 6)))
>   (import "wasi_snapshot_preview1" "fd_close" (func (;145;) (type 0)))
>   (import "wasi_snapshot_preview1" "fd_read" (func (;146;) (type 6)))
>   (import "wasi_snapshot_preview1" "fd_write" (func (;147;) (type 6)))
>   (import "env" "_tzset_js" (func (;148;) (type 8)))
923c923
<     call 2185
---
>     call 2186

I also tested the link command below with EMCC_DEBUG. Order of imports differs from /tmp/emscripten_temp/emcc-00-base.wasm.

It seems imports are sorted in test as well.

emscripten/test/test_other.py

Line 8864 in 5ac384b

imports.sort()

Not sure if it's a emscripten issue or llvm issue.

Version of emscripten/emsdk:
Please include the output emcc -v here

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.74 (1092ec30a3fb1d46b1782ff1b4db5094d3d06ae5)
clang version 20.0.0git (https:/github.com/llvm/llvm-project 322eb1a92e6d4266184060346616fa0dbe39e731)
Target: wasm32-unknown-emscripten
Thread model: posix

Failing command line in full:
Not failing.

Full link command and output with -v appended:

$ em++ -v -O3 -DNDEBUG  src/CMakeFiles/Fcitx5.dir/fcitx.cpp.o src/CMakeFiles/Fcitx5.dir/keycode.cpp.o src/CMakeFiles/
Fcitx5.dir/input_method.cpp.o src/CMakeFiles/Fcitx5.dir/config.cpp.o src/CMakeFiles/Fcitx5.dir/action.cpp.o src/CMakeFiles/Fcitx5.dir/event_js.cpp.o src/CMakeFiles/Fcitx
5.dir/__/dummy.cpp.o src/CMakeFiles/Fcitx5.dir/Fcitx5-webpanel-import-addon.cpp.o src/CMakeFiles/Fcitx5.dir/Fcitx5-wasmfrontend-import-addon.cpp.o src/CMakeFiles/Fcitx5.
dir/Fcitx5-keyboard-import-addon.cpp.o -o src/Fcitx5.js  -sMAIN_MODULE=1  -fexceptions  -sSTACK_SIZE=1MB  -sALLOW_MEMORY_GROWTH=1  -sMAXIMUM_MEMORY=4GB  -sEXPORTED_RUNTI
ME_METHODS=['ccall','FS']  -sENVIRONMENT=web  --extern-pre-js ../build/pre.js  --extern-post-js ../build/index.js  -L../build/destdir/usr/lib  --preload-file ../build/sy
sroot/usr/share/X11@/usr/share/X11  --preload-file ../build/sysroot/usr/share/iso-codes/json/iso_3166-1.json@/usr/share/iso-codes/json/iso_3166-1.json  --preload-file ..
/build/sysroot/usr/share/iso-codes/json/iso_639-3.json@/usr/share/iso-codes/json/iso_639-3.json  --preload-file ../build/destdir/usr/share/locale@/usr/share/locale  --pr
eload-file ../build/destdir/usr/share/fcitx5@/usr/share/fcitx5  --preload-file ../build/destdir/usr/lib/fcitx5@/usr/lib/fcitx5  webpanel/libwebpanel.a  wasmfrontend/libw
asmfrontend.a  fcitx5/src/im/keyboard/libkeyboard.a  fcitx5-webview/src/libWebviewCandidateWindow.a  fcitx5/src/lib/fcitx/libFcitx5Core.so  fcitx5/src/lib/fcitx-config/l
ibFcitx5Config.so  fcitx5/src/lib/fcitx-utils/libFcitx5Utils.so  sysroot/usr/lib/libexpat.a  sysroot/usr/lib/libxkbcommon.a  sysroot/usr/lib/libjson-c.a
 /mnt/storage/github/emsdk/upstream/bin/clang --version
 /mnt/storage/github/emsdk/upstream/emscripten/tools/file_packager src/Fcitx5.data --from-emcc --preload ../build/sysroot/usr/share/X11@/usr/share/X11 ../build/sysroot/usr/share/iso-codes/json/iso_3166-1.json@/usr/share/iso-codes/json/iso_3166-1.json ../build/sysroot/usr/share/iso-codes/json/iso_639-3.json@/usr/share/iso-codes/json/iso_639-3.json ../build/destdir/usr/share/locale@/usr/share/locale ../build/destdir/usr/share/fcitx5@/usr/share/fcitx5 ../build/destdir/usr/lib/fcitx5@/usr/lib/fcitx5 --no-node
 /mnt/storage/github/emsdk/upstream/bin/wasm-ld @/tmp/emscripten_qie0voif.rsp.utf-8
 /mnt/storage/github/emsdk/upstream/bin/wasm-ld @/tmp/emscripten_l43olzxs.rsp.utf-8
 /mnt/storage/github/emsdk/upstream/bin/llvm-objcopy src/Fcitx5.wasm src/Fcitx5.wasm --remove-section=.debug* --remove-section=producers
 /mnt/storage/github/emsdk/upstream/bin/wasm-emscripten-finalize --dyncalls-i64 --pass-arg=legalize-js-interface-export-originals --pass-arg=legalize-js-interface-exported-helpers src/Fcitx5.wasm -o src/Fcitx5.wasm --detect-features
 /mnt/storage/github/emsdk/upstream/bin/wasm-emscripten-finalize --dyncalls-i64 --pass-arg=legalize-js-interface-export-originals --pass-arg=legalize-js-interface-exported-helpers src/Fcitx5.wasm -o src/Fcitx5.wasm --detect-features
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/src/compiler.mjs /tmp/tmpzuugy7zv.json
 /mnt/storage/github/emsdk/upstream/bin/wasm-opt --strip-target-features --post-emscripten -O3 --low-memory-unused --zero-filled-memory --pass-arg=directize-initial-contents-immutable --no-stack-ir src/Fcitx5.wasm -o src/Fcitx5.wasm --mvp-features --enable-call-indirect-overlong --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext
 /mnt/storage/github/emsdk/upstream/bin/wasm-opt --strip-target-features --post-emscripten -O3 --low-memory-unused --zero-filled-memory --pass-arg=directize-initial-contents-immutable --no-stack-ir src/Fcitx5.wasm -o src/Fcitx5.wasm --mvp-features --enable-call-indirect-overlong --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.js unsignPointers -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.js unsignPointers -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js --minify-whitespace -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso2.js
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js --minify-whitespace -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso2.js

The text was updated successfully, but these errors were encountered:

kripken · 2025-01-13T21:04:32Z

To find if this is an llvm, emscripten, or binaryen issue, link with EMCC_DEBUG=1 in the env, and view the temp files /tmp/emscripten_temp/emcc-* that are emitted. The logging output will say which tool emits which file.

eagleoflqj · 2025-01-15T04:16:57Z

Thanks for you instruction. There are 2 executions of wasm-ld. The second one causes difference (with same normalized response file provided). A wasm-ld -mllvm -debug @response-file logs different import order.
Definitely a lld issue, not emscripten.

eagleoflqj · 2025-01-15T04:26:23Z

By checking wasm-ld's log, I think the first difference comes from different emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/libc.a.
On macOS arm64 it has md5sum 429f0d49ac576db7afef94c6790acce8, on Linux x64_64 it's 84ea5357d41ab10ef3075ee44234475c

eagleoflqj · 2025-01-15T04:31:31Z

Yeah I think it's exactly the issue. A diff of ar tv libc.a shows

1d0
< rw-r--r-- 0/0   1404 Dec 31 19:00 1969 _Exit.o
73a73
> rw-r--r-- 0/0   1404 Dec 31 19:00 1969 _exit__1.o

eagleoflqj · 2025-01-15T05:44:41Z

emscripten/tools/system_libs.py

Lines 232 to 243 in 6bed395

    
           # Resolve duplicates by appending unique. 
        
           # This is needed on case insensitive filesystem to handle, 
        
           # for example, _exit.o and _Exit.o. 
        
           object_basename = shared.unsuffixed_basename(src) 
        
           if case_insensitive: 
        
             object_basename = object_basename.lower() 
        
           o = os.path.join(build_dir, object_basename + '.o') 
        
           object_uuid = 0 
        
           # Find a unique basename 
        
           while o in objects: 
        
             object_uuid += 1 
        
             o = os.path.join(build_dir, f'{object_basename}__{object_uuid}.o')

So this doesn't seem reproducible cross macOS and Linux
CC @sbc100
Does it make sense to always treat FS as case insensitive?

sbc100 · 2025-01-15T15:50:57Z

Sure that sounds like a good idea. Would you like to send a PR?

Is macOS really case insensitive by default? I had not realized that. Strange.

kripken · 2025-01-15T18:54:28Z

We have an option for that (inspired by MacOS, I think?):

emscripten/src/settings.js

Lines 1013 to 1017 in bcad96d

    
           // If set to nonzero, the provided virtual filesystem if treated 
        
           // case-insensitive, like Windows and macOS do. If set to 0, the VFS is 
        
           // case-sensitive, like on Linux. 
        
           // [link] 
        
           var CASE_INSENSITIVE_FS = false;

Use it with emcc -sCASE_INSENSITIVE_FS. It should work in both the old FS and WasmFS.

kripken · 2025-01-15T18:55:18Z

Oh, wait, ignore my last comment 😆 I was mixed up by the runtime filesystem and the emsdk build.

eagleoflqj · 2025-01-16T00:04:30Z

Would you like to send a PR?

Yes!

Is macOS really case insensitive by default?

Yes. The is_case_insensitive function shows it is.

eagleoflqj closed this as completed Jan 15, 2025

eagleoflqj reopened this Jan 15, 2025

eagleoflqj changed the title ~~order of imports in wasm is different when compiling on macOS and Linux~~ Different libc.a for macOS arm64 and Linux x86_64 Jan 15, 2025

eagleoflqj mentioned this issue Jan 16, 2025

Always treat host FS as case-insensitive to produce deterministic system libraries regardless of OS #23426

Merged

sbc100 closed this as completed in #23426 Jan 16, 2025

sbc100 closed this as completed in 186cae3 Jan 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Different libc.a for macOS arm64 and Linux x86_64 #23379

Different libc.a for macOS arm64 and Linux x86_64 #23379

eagleoflqj commented Jan 13, 2025

kripken commented Jan 13, 2025

eagleoflqj commented Jan 15, 2025

eagleoflqj commented Jan 15, 2025

eagleoflqj commented Jan 15, 2025

eagleoflqj commented Jan 15, 2025

sbc100 commented Jan 15, 2025

kripken commented Jan 15, 2025

kripken commented Jan 15, 2025

eagleoflqj commented Jan 16, 2025

Different libc.a for macOS arm64 and Linux x86_64 #23379

Different libc.a for macOS arm64 and Linux x86_64 #23379

Comments

eagleoflqj commented Jan 13, 2025

kripken commented Jan 13, 2025

eagleoflqj commented Jan 15, 2025

eagleoflqj commented Jan 15, 2025

eagleoflqj commented Jan 15, 2025

eagleoflqj commented Jan 15, 2025

sbc100 commented Jan 15, 2025

kripken commented Jan 15, 2025

kripken commented Jan 15, 2025

eagleoflqj commented Jan 16, 2025