ld.bfd ld.gold missing and build_binutils problem

[xxjoe2]

hi

i tried to build clanv last nite. i followed your building instrutions to build it but i couldn't find the ld.gold nor ld.bfd files. i built without the -p option which i believed bfd and gold files would be created. did i miss something?

in this post #231 you mentioned --enable-gold option. where should i put this option?

also i tried to run build_binutils.py but it returned errors as below. do i need to run it separately after building llvm?

/build-binutils.py -i /tmp/binutils -m ARCH -t arm aarch64

configure: error: in /tmp/tc-build/build/binutils/aarch64':
configure: error: C compiler cannot create executables
See config.log' for more details Traceback (most recent call last): File "./build-binutils.py", line 125, in <module> builder.build() File "/tmp/tc-build/tc_build/binutils.py", line 63, in build self.run_cmd(configure_cmd, cwd=self.folders.build) File "/tmp/tc-build/tc_build/builder.py", line 39, in run_cmd return subprocess.run(cmd, capture_output=capture_output, check=True, cwd=cwd) File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '[PosixPath('/tmp/tc-build/src/binutils-2.41/configure'), '--disable-compressed-debug-sections', '--disable-gdb', '--disable-nls', '--disable-werror', '--enable-deterministic-archives', '--enable-new-dtags', '--enable-plugins', '--enable-threads', '--quiet', '--with-system-zlib', '--disable-multilib', '--with-gnu-as', '--with-gnu-ld', '--prefix=/cache/binutils', '--program-prefix=aarch64-linux-gnu-', '--target=aarch64-linux-gnu', 'CC=gcc', 'CXX=g++', 'CFLAGS=-O2 -march=ARCH -mtune=ARCH', 'CXXFLAGS=-O2 -march=ARCH -mtune=ARCH']' returned non-zero exit status 77.

i am on ubuntu 20.04

thanks

[nathanchance]

What is your invocation of build-llvm.py and what is the exact error message that you get when you try to run it? build-binutils.py is not a prerequisite of build-llvm.py, they are two standalone scripts.

Your build-binutils.py error occurs because -m ARCH is passing -march=ARCH -mtune=ARCH to the compiler, which obviously isn't valid. Pick a valid value for those flags.

[xxjoe2]

thanks for your prompt reply

no error but i didn't look at the log file

first i built it with
/tmp/tc-build$ CC=clang CFLAGS= CXX=clang++ CXXFLAGS= ./build-llvm.py -t AArch64 ARM --quiet-cmake --lto thin --no-ccache --use-good-revision -i /tmp/clang

no luck, so i tried to use -p option

/tmp/tc-build$ CC=clang CFLAGS= CXX=clang++ CXXFLAGS= ./build-llvm.py -s -t AArch64 ARM X86 --quiet-cmake --lto thin -i /tmp/clang -n -p clang clang-tools-extra lldb lld polly -D LLVM_PARALLEL_COMPILE_JOBS=2 LLVM_PARALLEL_LINK_JOBS=2

but still no bfd or gold files

[nathanchance]

Oh, I misunderstood your issue from the original post.

The reason you don't have ld.gold is because ld.gold does not work for the upstream kernel, which is what this project cares most about:

https://git.kernel.org/linus/75959d44f9dc8e44410667009724e4e238515502

If you want/need it for other reasons, add --enable-gold to self.configure_flags in BinutilsBuilder in tc_build/binutils.py, which is what I was getting after in #231.

[xxjoe2]

ok. now i know where to make the changes. thanks

i edited tc_build/binutils.py and addded --enable-gold as below
'--quiet', '--with-system-zlib', '--enable-gold', ]

then i executed it again with
CC=clang CFLAGS= CXX=clang++ CXXFLAGS= ./build-llvm.py -s -t AArch64 ARM --quiet-cmake --lto thin -i /tmp/clang -n -p clang clang-tools-extra lldb lld polly -D LLVM_PARALLEL_COMPILE_JOBS=4 LLVM_PARALLEL_LINK_JOBS=4

but this time i got segmentation fault
-- Symbols (liblldb): exporting all symbols from the lldb namespace -- Configuring done -- Generating done -- Build files have been written to: /tmp/tc-build/build/llvm/final ninja: Entering directory/tmp/tc-build/build/llvm/final'
[3324/5675] Linking CXX executable bin/clang-refactor
FAILED: bin/clang-refactor
: && /tmp/tc-build/build/llvm/bootstrap/bin/clang++ -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -w -fdiagnostics-color -ffunction-sections -fdata-sections -flto=thin -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -fuse-ld=/tmp/tc-build/build/llvm/bootstrap/bin/ld.lld -Wl,--color-diagnostics -flto=thin -Wl,--gc-sections tools/clang/tools/clang-refactor/CMakeFiles/clang-refactor.dir/ClangRefactor.cpp.o tools/clang/tools/clang-refactor/CMakeFiles/clang-refactor.dir/TestSupport.cpp.o -o bin/clang-refactor -Wl,-rpath,"$ORIGIN/../lib:" lib/libLLVMOption.a lib/libLLVMSupport.a -lpthread lib/libclangAST.a lib/libclangBasic.a lib/libclangFormat.a lib/libclangFrontend.a lib/libclangLex.a lib/libclangRewrite.a lib/libclangSerialization.a lib/libclangTooling.a lib/libclangToolingCore.a lib/libclangToolingRefactoring.a lib/libclangIndex.a lib/libclangFormat.a lib/libclangToolingInclusions.a lib/libclangFrontend.a lib/libclangDriver.a lib/libLLVMWindowsDriver.a lib/libLLVMOption.a lib/libclangParse.a lib/libclangSerialization.a lib/libclangSema.a lib/libclangEdit.a lib/libclangAnalysis.a lib/libclangASTMatchers.a lib/libclangAST.a lib/libLLVMFrontendOpenMP.a lib/libLLVMScalarOpts.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMFrontendOffloading.a lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a lib/libLLVMProfileData.a lib/libLLVMSymbolize.a lib/libLLVMDebugInfoPDB.a lib/libLLVMDebugInfoMSF.a lib/libLLVMDebugInfoBTF.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMObject.a lib/libLLVMMCParser.a lib/libLLVMMC.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMTextAPI.a lib/libclangSupport.a lib/libLLVMFrontendHLSL.a lib/libLLVMBitReader.a lib/libclangToolingCore.a lib/libclangRewrite.a lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a lib/libLLVMBinaryFormat.a lib/libLLVMTargetParser.a lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMSupport.a -lrt -ldl -lpthread -lm /usr/lib/x86_64-linux-gnu/libz.so lib/libLLVMDemangle.a && :
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: error: linker command failed due to signal (use -v to see invocation)
[3331/5675] Linking CXX executable bin/clang-rename
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "./build-llvm.py", line 709, in
final.build()
File "/tmp/tc-build/tc_build/llvm.py", line 147, in build
self.run_cmd(ninja_cmd)
File "/tmp/tc-build/tc_build/builder.py", line 39, in run_cmd
return subprocess.run(cmd, capture_output=capture_output, check=True, cwd=cwd)
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-C', PosixPath('/tmp/tc-build/build/llvm/final'), 'all']' returned non-zero exit status 1.
`
how can i fix this pls?

note: it uses ubuntu system cc & ld. does it matter? or do i have to remove -s optiom and re-download the files?

`

== Checking CC and LD ==

CC: /usr/lib/llvm-17/bin/clang
CXX: /usr/bin/clang++
LD: /usr/lib/llvm-17/bin/ld.lld

`

[nathanchance]

What is your end goal here? Again, build-binutils.py and build-llvm.py are two different scripts that build two different projects. build-binutils.py builds binutils: the GNU assembler (GAS), linker (ld.bfd, ld.gold), archiver, etc. build-llvm.py builds LLVM (clang, ld.lld, llvm-ar, etc). You are talking about wanting ld.gold and you edited tc_build/binutils.py but then you ran build-llvm.py?

The segmentation fault could be an upstream problem or due to a lack of memory since you are using ThinLTO.

[xxjoe2]

i want to build a clang coz all existing pre-builts didnt work for me, but i need bfd and gold

so, perhaps i misread your reply or you were talking about fixing the binutils problem?
"If you want/need it for other reasons, add --enable-gold to self.configure_flags in BinutilsBuilder in tc_build/binutils.py"?

[nathanchance]

i want to build a clang coz all existing pre-builts didnt work for me, but i need bfd and gold

Okay so you'll need to run both scripts in that case.

For what it's worth, LTO does very little for runtime performance and you pay a relatively significant build time cost for it (I am in the process of collecting benchmarks right now that I'll publish later). You'd be better off switching it out for --pgo kernel-defconfig or --pgo kernel-defconfig-slim, depending on how much build time you can spare.

Alternatively, I host prebuilt LLVM toolchains (built using build-llvm.py) on kernel.org that have already been built with these optimizations, you may consider seeing if those would work for you.

so, perhaps i misread your reply or you were talking about fixing the binutils problem? "If you want/need it for other reasons, add --enable-gold to self.configure_flags in BinutilsBuilder in tc_build/binutils.py"?

Yes, that was the (potential) solution to

i couldn't find the ld.gold nor ld.bfd files.

and this

Your build-binutils.py error occurs because -m ARCH is passing -march=ARCH -mtune=ARCH to the compiler, which obviously isn't valid. Pick a valid value for those flags.

was the solution to

also i tried to run build_binutils.py but it returned errors as below.

from your original post. Hope all that helps!

[xxjoe2]

ok. so what about the --enable-gold option? where should it put it?

[nathanchance]

diff --git a/tc_build/binutils.py b/tc_build/binutils.py
index 09eb98c..d28013f 100644
--- a/tc_build/binutils.py
+++ b/tc_build/binutils.py
@@ -21,6 +21,7 @@ class BinutilsBuilder(Builder):
             '--disable-nls',
             '--disable-werror',
             '--enable-deterministic-archives',
+            '--enable-gold',
             '--enable-new-dtags',
             '--enable-plugins',
             '--enable-threads',

[xxjoe2]

ok thanks a lot. i will try again and your prebuilt
sorry for being a noob but switching to clang drives me crazy...

[xxjoe2]

oh sorry, 1 more thing
i downloaded your prebuilt but under bin/ the files are different. i want to use clang to build an android kernel, which usually have the clang triple like aarch64-linux-gnu. does it mean yours wont work for me or i have to create symlinks?

[nathanchance]

I am not sure I fully understand the question. The prebuilt is only LLVM/clang, you will still need to provide a copy of aarch64-linux-gnu binutils (like aarch64-linux-gnu-as, aarch64-linux-gnu-ar, etc), which is what build-binutils.py will generate (or usually, distribution versions of those are sufficient).

Perhaps showing your kernel build command would be helpful, along with your kernel source, so I know how to alter it for this toolchain.

[xxjoe2]

hmm, i think it's due to my misunderstanding. usually those prebuilts like google or NeutronClang, under the bin dir i can find the followings (sorry for posting screen capture but it's a long listing)

https://tmpfiles.org/dl/3120119/screenshot_20231109_020032_termius.jpg

and the dir structue looks like this
aarch64-linux-gnu
arm-linux-gnueabi
bin
etc
lib
share
x86_64-pc-linux-gnu

so it's not your problem but mine. i want to build one with clang and llvm (or what it should called) with those files under bin looks like the image above. i have googled but i still not very clear about them and thus i call it clang in general like others do

and for the kernel build command, i managed to compile the kernel with third party prebuilts (but didnt boot up) so it's fine. except the google android one, even i provided the env CLANG_TRIPLE it will prompt an error that CLANG_TRIPLE is not needed, but if i remove this env, it asks me for the triple with the below ....

`
CLANG_HOME=/opt1/android/toolchain/ndk/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64

export PATH=$CLANG_HOME/bin:$PATH

make API=30 CC=clang CROSS_COMPILE=aarch64-linux-android$API- CROSS_COMPILE_ARM32=armv7a-linux-androideabi$API- CROSS_COMPILE_COMPAT=armv7a-linux-androideabi$API- LLVM=1 CLANG_TRIPLE1=aarch64-linux-android$API- ARCH=arm64 AR=llvm-ar NM=llvm-nm OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump STRIP=llvm-strip CLANG_FLAGS="--target=aarch64" LLVM_IAS=1
`

[nathanchance]

Okay, yeah, that's the disconnect :)

The aarch64-linux-gnu and plain as/ld/objcopy/etc tools in that bin/ folder are generated from build-binutils.py and are known as GNU binutils. clang and the llvm/ll tools (along with other irrelevant tools for building the kernel) in that bin/ folder are LLVM.

The prebuilts on kernel.org only provide LLVM, as our project is focused on using LLVM exclusively to build the kernel. On some older kernel releases, such as some that Android uses, you have to mix and match the two sets of tools (due to various bugs and incompatibilities that we fixed along the way), hence why those other prebuilts choose to package both in one tarball/repo.

So you can extract the LLVM prebuilt from kernel.org (let's say you extracted it to $HOME/toolchain so you would have $HOME/toolchain/bin, etc) and use that same extraction folder (NOT the bin folder) as the --install-folder argument to build-binutils.py to get aarch64-linux-gnu tools alongside the LLVM ones in the same manner as those other toolchains.

Hope that helps clear some things up!

[xxjoe2]

absolutely

let me try again

thanks a lot man

[xxjoe2]

hi

just wanna tell you, as what i expected and what you said, the built is succeed finally and seems i can see all the files i wanted. big thanks

and if you have any info about the google clang problem i mentioned pls let me know, coz those who have built the kernel successfully were using google's

i will keep this issue as open for 1 or 2 days as to try to build my kernel. if everything is fine, i will close it

thanks for your time and help once again

[nathanchance]

and if you have any info about the google clang problem i mentioned pls let me know, coz those who have built the kernel successfully were using google's

What version of AOSP LLVM were you using and what was the exact error that you got? You may need to scroll back in your build logs depending on when the error occurs.

[xxjoe2]

sorry for late reply since i have to try your clang (it works but the kernel still wont boot) and prepare previous env for the errors

i had problems when compiling vdso with some pre-built clangs as well as google's. there are 2 different errors which i believe it's the linker problem or something is missing

problem 1:
the command i use for google's:
CLANG_HOME=/opt1/android/toolchain/ndk/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64 export PATH=$CLANG_HOME/bin:$PATH make API=30 CC=clang CROSS_COMPILE=aarch64-linux-android$API- CROSS_COMPILE_ARM32=armv7a-linux-androideabi$API- CROSS_COMPILE_COMPAT=armv7a-linux-androideabi$API- LLVM=1 CLANG_TRIPLE1=aarch64-linux-android$API- ARCH=arm64 AR=llvm-ar NM=llvm-nm OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump STRIP=llvm-strip CLANG_FLAGS="--target=aarch64" LLVM_IAS=1

error:
make -f ./scripts/Makefile.build obj=arch/arm64/kernel/vdso include/generated/vdso-offsets.h -EL -maarch64linux -O3 -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv --build-id -n -T arch/arm64/kernel/vdso/vdso.lds arch/arm64/kernel/vdso/vgettimeofday.o arch/arm64/kernel/vdso/note.o arch/arm64/kernel/vdso/sigreturn.o -o arch/arm64/kernel/vdso/vdso.so.dbg /bin/sh: 1: -EL: not found make[1]: *** [arch/arm64/kernel/vdso/Makefile:51: arch/arm64/kernel/vdso/vdso.so.dbg] Error 127 make: *** [arch/arm64/Makefile:233: vdso_prepare] Error 2
this particular build from google was recommended by others who had built the kernel successfully. one told me the bfd should handle it properly but not for me

problem 2 (ignore the llvm17 path for now):
make -f ./scripts/Makefile.build obj=arch/arm64/kernel/vdso /usr/lib/llvm-17/bin/llvm-objcopy -S arch/arm64/kernel/vdso/vdso.so.dbg arch/arm64/kernel/vdso/vdso.so /usr/lib/llvm-17/bin/clang -Wp,-MD,arch/arm64/kernel/vdso/.vdso.o.d -nostdinc -isystem /usr/lib/llvm-17/lib/clang/17/include -I./arch/arm64/include -I./arch/arm64/include/generated/uapi -I./arch/arm64/include/generated -I./include -I./drivers/kernelsu/include -I./arch/arm64/include/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Qunused-arguments -mlittle-endian -Qunused-arguments -D__ASSEMBLY__ --target=aarch64-linux-gnu -DCC_USE_CLANG --prefix=/usr/bin/aarch64-linux-gnu- --gcc-toolchain=/usr -Werror=unknown-warning-option -fno-PIE -DCONFIG_AS_LSE=1 -DCONFIG_VDSO32=1 -c -o arch/arm64/kernel/vdso/vdso.o arch/arm64/kernel/vdso/vdso.S /usr/lib/llvm-17/bin/ld.lld -EL -maarch64elf -O3 -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv --build-id -n -T -r -o arch/arm64/kernel/vdso/built-in.o arch/arm64/kernel/vdso/vdso.o ld.lld: error: cannot find linker script -r make[2]: *** [scripts/Makefile.build:507: arch/arm64/kernel/vdso/built-in.o] Error 1 make[1]: *** [scripts/Makefile.build:647: arch/arm64/kernel/vdso] Error 2 make: *** [Makefile:1239: arch/arm64/kernel] Error 2

the above error occured with ubuntu repo's clang as well as certain pre-builts

i couldn't fix neither of them so i switched to other builds and found some of them worked, including the one built with your script

in case it could be time consuming then just leave it coz i don wanna waste your time

thanks

[nathanchance]

problem 1: the command i use for google's: CLANG_HOME=/opt1/android/toolchain/ndk/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64 export PATH=$CLANG_HOME/bin:$PATH make API=30 CC=clang CROSS_COMPILE=aarch64-linux-android$API- CROSS_COMPILE_ARM32=armv7a-linux-androideabi$API- CROSS_COMPILE_COMPAT=armv7a-linux-androideabi$API- LLVM=1 CLANG_TRIPLE1=aarch64-linux-android$API- ARCH=arm64 AR=llvm-ar NM=llvm-nm OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump STRIP=llvm-strip CLANG_FLAGS="--target=aarch64" LLVM_IAS=1

This command is not correct because you are passing API to make but the use of it in the command $API is being expanded by your shell. The error you are getting from that command indicates that your value of CROSS_COMPILE or CROSS_COMPILE_COMPAT/CROSS_COMPILE_ARM32 is incorrect. This should be something like:

$ CLANG_HOME=/opt1/android/toolchain/ndk/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64
$ API=30
$ export PATH=$CLANG_HOME/bin:$PATH
$ make \
    AR=llvm-ar \
    ARCH=arm64 \
    CC=clang \
    CLANG_TRIPLE=aarch64-linux-gnu- \
    CROSS_COMPILE=aarch64-linux-android$API- \
    CROSS_COMPILE_ARM32=armv7a-linux-androideabi$API- \
    CROSS_COMPILE_COMPAT=armv7a-linux-androideabi$API- \
    LLVM=1 \
    LLVM_IAS=1 \
    NM=llvm-nm \
    OBJCOPY=llvm-objcopy \
    OBJDUMP=llvm-objdump \
    STRIP=llvm-strip

A few notes/questions about that command:

• CLANG_TRIPLE's value was incorrect, it should be aarch64-linux-gnu. The value CROSS_COMPILE is normally used for --target but Android's target triple does not work for the kernel. CLANG_TRIPLE changes that, although newer Android branches have moved away from that variable. You would be better of avoiding use of the NDK when building the kernel and using the GAS prebuilts from AOSP with AOSP LLVM. On that note...
• Is this command supposed to be used with multiple different kernels/versions? I only ask because LLVM=1 is the same thing as AR=llvm-ar, CC=clang, LD=ld.lld, etc, so if your kernel has supposed for that, you can clean up the command a lot.
• CLANG_FLAGS is unnecessary, it should be set up by the build system.

[xxjoe2]

that's the default command on github, with some changes like LLVM=1 for changes in Makefiles by the developer

oh, yeah, i was wrong to place API there, it's my fault. for the CLANG_TRIPLE, i was wondering why it asked me for the triple if i didnt supply it with error
Makefile:573: *** "Clang with Android --target detected. Did you specify CLANG_TRIPLE?". Stop.
and if i supplied it, it was denied by the Makefile and i had to remove the checking
so now i understand why, thanks for your clarifications

however, when i followed your command and tried to build it again with google's, the same error occured
make -f ./scripts/Makefile.build obj=arch/arm64/kernel/vdso include/generated/vdso-offsets.h -EL -maarch64elf -O3 -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv --build-id -n -T arch/arm64/kernel/vdso/vdso.lds arch/arm64/kernel/vdso/vgettimeofday.o arch/arm64/kernel/vdso/note.o arch/arm64/kernel/vdso/sigreturn.o -o arch/arm64/kernel/vdso/vdso.so.dbg /bin/sh: 1: -EL: not found make[1]: *** [arch/arm64/kernel/vdso/Makefile:51: arch/arm64/kernel/vdso/vdso.so.dbg] Error 127 make: *** [arch/arm64/Makefile:233: vdso_prepare] Error 2

so, may be just leave it for now

edit
ok i got it. the one from google's doesnt support lto, while it seems lto is critical in compiling vdso thing

[xxjoe2]

alright. it's the build script problem that led to unbootable kernel. so it should be fine with your tc-build scripts.

thanks again for your explanations, time etc. you may close this of you dont have any comments or suggestions

thanks a lot

[nathanchance]

No problem, glad you were able to get it sorted out!