Can't build BPF program on Fedora 40 and Debian 13 (Linux 6.10)

[lnicola]

Just a heads-up, since someone else might run into this:

=== [0x.tools] xcapture-bpf 2.0.3 BETA by Tanel Poder. Fedora Linux 40 6.10.6 x86_64
===  Loading BPF...
In file included from /virtual/main.c:31:
In file included from include/uapi/linux/ptrace.h:183:
In file included from arch/x86/include/asm/ptrace.h:175:
In file included from arch/x86/include/asm/paravirt_types.h:12:
In file included from arch/x86/include/asm/nospec-branch.h:15:
arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
   47 |                 return this_cpu_read_const(const_pcpu_hot.current_task);
      |                        ^
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:30: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
arch/x86/include/asm/percpu.h:93:28: note: expanded from macro '__my_cpu_ptr'
   93 | #define __my_cpu_ptr(ptr)       (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr)
      |                                  ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:31:
In file included from include/uapi/linux/ptrace.h:183:
In file included from arch/x86/include/asm/ptrace.h:175:
In file included from arch/x86/include/asm/paravirt_types.h:12:
In file included from arch/x86/include/asm/nospec-branch.h:15:
arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:9: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:32:
In file included from include/linux/sched.h:13:
arch/x86/include/asm/processor.h:543:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
  543 |                 return this_cpu_read_const(const_pcpu_hot.top_of_stack);
      |                        ^
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:30: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
arch/x86/include/asm/percpu.h:93:28: note: expanded from macro '__my_cpu_ptr'
   93 | #define __my_cpu_ptr(ptr)       (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr)
      |                                  ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:32:
In file included from include/linux/sched.h:13:
arch/x86/include/asm/processor.h:543:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:9: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:34:
In file included from include/linux/syscalls.h:93:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:62:
In file included from include/linux/security.h:35:
include/linux/bpf.h:348:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_wq'
  348 |                 return sizeof(struct bpf_wq);
      |                        ^     ~~~~~~~~~~~~~~~
include/linux/bpf.h:348:24: note: forward declaration of 'struct bpf_wq'
  348 |                 return sizeof(struct bpf_wq);
      |                                      ^
include/linux/bpf.h:377:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_wq'
  377 |                 return __alignof__(struct bpf_wq);
      |                        ^          ~~~~~~~~~~~~~~~
include/linux/bpf.h:377:29: note: forward declaration of 'struct bpf_wq'
  377 |                 return __alignof__(struct bpf_wq);
      |                                           ^
4 warnings and 2 errors generated.
Traceback (most recent call last):
  File "/home/grayshade/Projects/0xtools/bin/xcapture-bpf", line 474, in <module>
    b = BPF(text= ifdef + bpf_text)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/bcc/__init__.py", line 479, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

You can ignore the warnings (biolatency shows them too), but I don't understand the problem with the missing struct. It's defined in /usr/src/kernels/6.10.6-200.fc40.x86_64/include/uapi/linux/bpf.h:

struct bpf_wq {                                                                                                                                                                                                                       
  __u64 __opaque[2];                                                                                                                                                                                                                  
} __attribute__((aligned(8)));                                                                                                                                                                                                        

Which /usr/src/kernels/6.10.6-200.fc40.x86_64/include/linux/bpf.h includes:

#include <uapi/linux/bpf.h>

// ...

static inline u32 btf_field_type_size(enum btf_field_type type)
{
        switch (type) {
        case BPF_SPIN_LOCK:
                return sizeof(struct bpf_spin_lock);
        case BPF_TIMER:
                return sizeof(struct bpf_timer);
        case BPF_WORKQUEUE:
                return sizeof(struct bpf_wq);
        case BPF_KPTR_UNREF:
        case BPF_KPTR_REF:
        case BPF_KPTR_PERCPU:
                return sizeof(u64);
        case BPF_LIST_HEAD:
                return sizeof(struct bpf_list_head);
        case BPF_LIST_NODE:
                return sizeof(struct bpf_list_node);
        case BPF_RB_ROOT:
                return sizeof(struct bpf_rb_root);
        case BPF_RB_NODE:
                return sizeof(struct bpf_rb_node);
        case BPF_REFCOUNT:
                return sizeof(struct bpf_refcount);
        default:
                WARN_ON_ONCE(1);
                return 0;
        }
}

---

TL;DR: this fails to build on 6.10:

from bcc import BPF

bpf_text = """
#include <uapi/linux/bpf.h>
#include <linux/syscalls.h>
"""
BPF(text=bpf_text)

[tanelpoder]

Interesting, I'll test it on Debian 13 and see what's up. Does Debian 13 have 6.10 kernel as a default or did you upgrade the kernel separately?

There's a chance that since BCC is somewhat deprecated (in favor to libbpf approach), maybe there's some header/kernel incompatibility, like what I saw on Ubuntu 22.04.

My next step is to build v3 based on libbpf (using BTF/CO-RE), this should avoid future kernel/header incompatibility issues, in addition to all the other benefits.

[lnicola]

Does Debian 13 have 6.10 kernel as a default or did you upgrade the kernel separately?

apt update && apt upgrade will give you 6.10.6.

[tanelpoder]

It took a while, but I finally attempted to test on Debian 13. I didn't realize that it's not a GA release yet... there's a good chance that you'd need to separately download & build the latest bcc toolset (instead of the bundled package) to make it work.

I saw various issues on Ubuntu 22.04 too (24.04 works ok) - given that bcc is pretty much deprecated in favor or proper pre-compiled binaries that rely on BTF and CO-RE (using libbpf), I stopped working on v2 and started working on v3 that does everything right according to the modern state of eBPF. The highly in-flux prototype of v3 (using libbpf + BTF + CO-RE) is in the "next" directory:

• https://github.com/tanelpoder/0xtools/tree/master/next

Once that's out, it should not have any issues with missing or incompatible headers (even if people are using non-standard kernels that usually come with the distro, etc).

I did demo some absolute basics of the new approach (BPF task_iterator, etc) at my talk with Liz Rice a few months ago (the last 10 mins of the video or so):

• https://www.youtube.com/watch?v=DFYLkYwgaGE

I'm leaving this issue open, in case anyone else with a similar issue ends up coming here. Thanks!