eBPF CO-RE - Portable Tools Setup & Testing

Embarking on eBPF development often feels frustrating: you are promised the power of “Compile Once, Run Everywhere”, but you are delivered a nightmare of header redefinition errors and cryptic “Error loading vmlinux BTF” messages the moment you move a binary from your dev VM to a real host. This friction usually stems from a subtle mismatch between your build environment’s outdated libraries and your target’s modern kernel, breaking the portability promises of BPF.

In this blog post, we will walk through a complete method to bypass system package limitations and build a static, self-contained toolchain. But we won’t stop at compilation. We will also build a custom, automated testing harness that spins up multiple Linux kernel versions in seconds, ensuring your tool truly runs everywhere before you ever touch a production server.

If you are learning eBPF, you have likely hit the two bosses of the early game:

• Header Issues: Compiler errors screaming about redefinition of struct bpf_insn or __u32.
• Portability Issues: Your code compiles in your VM but fails instantly on your host machine with Error loading vmlinux BTF.

This blog post can be considered as a guide to avoid such errors and to learn how to set up a robust, Compile Once - Run Everywhere (CO-RE) environment that actually works.

The Architecture

• Dev Environment: A stable Linux VM (e.g., Ubuntu via Vagrant).
• Target Environment: A modern Linux Host (e.g., Kali Linux, Arch, or Fedora).
• The Goal: Compile static binaries inside the VM that run on any modern Linux kernel without installing dependencies on the target.

Step 1: The Dependency Strategy

Don’t rely on your OS package manager (like apt install libbpf-dev) for the core library. The version in standard repositories is often too old to understand the BTF (debug info) of newer kernels (like Linux 6.x), leading to load errors.

We will include libbpf, a C library to load, verify, and interact with eBPF programs and maps in the Linux kernel.

Project Structure

Create a clean directory for your project.

/my-ebpf-project
├── program.c       # Userspace loader
├── program.bpf.c   # Kernel code
├── Makefile        # The build automation
└── libbpf/         # (We will clone this)

Fetch and Build Libbpf

Run these commands inside your project folder. This builds the library locally without polluting your system.

# Clone the official library
git clone https://github.com/libbpf/libbpf.git

# Create a local build directory
mkdir -p libbpf/build

# Build the library AND install headers to the build folder
make -C libbpf/src BUILD_DIR=../build OBJDIR=../build

# Build the static library archive (.a)
make -C src BUILD_DIR=../build OBJDIR=../build DESTDIR=../build install

Step 2: The “Magic” Makefile

This Makefile does three critical things:

1. Generates vmlinux.h: Creates a header file containing every type definition in your running kernel.
2. Links Statically: Bundles libbpf, libelf, and zlib into the final binary so you can copy-paste it to another machine.
3. Points to Local libbpf: Ignores the system libbpf and uses the fresh one we just built.

Create the Makefile:

# Compiler settings
BPF_CLANG=clang

# Kernel side flags
BPF_CFLAGS=-g -O2 -target bpf

# Userspace side flags
# Note: -std=gnu99 to prevent C23 symbol issues (like __isoc23_strtoull)
USER_CFLAGS=-g -O2 -std=gnu99

# --- LIBBPF CONFIG ---
# Point to the library we built locally
LIBBPF_DIR = ./libbpf/build
LIBBPF_OBJ = $(LIBBPF_DIR)/libbpf.a

# Include the headers we installed into libbpf/build/usr/include
# We also include uapi for kernel definitions if needed
INCLUDES = -I$(LIBBPF_DIR)/usr/include -I./libbpf/include/uapi

# Link Statically: Bundles libbpf into the binary
STATIC_LDFLAGS = -static -Wl,--whole-archive $(LIBBPF_OBJ) -Wl,--no-whole-archive -lelf -lz

# Project Name
NAME=program
BPFOBJ=$(NAME).bpf.o
SKELETON=$(NAME).skel.h
EXEC=$(NAME)

# --- TARGETS ---

# Build User-Space Executable
$(EXEC): $(SKELETON) $(NAME).c
	$(BPF_CLANG) $(USER_CFLAGS) $(INCLUDES) $(NAME).c $(STATIC_LDFLAGS) -o $(EXEC)

# Build BPF Kernel Object
$(BPFOBJ): $(NAME).bpf.c vmlinux.h
	$(BPF_CLANG) $(BPF_CFLAGS) $(INCLUDES) -c $(NAME).bpf.c -o $(BPFOBJ)

# Generate vmlinux.h (The "All-in-One" Kernel Header)
vmlinux.h:
	bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

# Generate Skeleton Header (Bridge between Kernel and User)
$(SKELETON): $(BPFOBJ)
	bpftool gen skeleton $(BPFOBJ) > $(SKELETON)

clean:
	- rm -f *.o *.skel.h vmlinux.h $(EXEC)

Step 3: Minimal Code

A common error when developing eBPF programs is Type Redefinition. This happens when you include both vmlinux.h (which defines everything) and standard system headers (like <linux/bpf.h>).

The Rule: In your kernel code (program.bpf.c), include vmlinux.h and nothing else that defines system types.

Minimal program.bpf.c

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

SEC("tracepoint/syscalls/sys_enter_execve")
int handle_exec(void *ctx)
{
    char msg[] = "Hello from CO-RE!";
    bpf_printk("%s\n", msg);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Minimal program.c

#include <stdio.h>
#include <unistd.h>
#include <bpf/libbpf.h>
#include "program.skel.h" // This file is auto-generated by the Makefile

int main()
{
    // Open and Load the BPF program into the kernel
    struct program_bpf *skel = program_bpf__open_and_load();
    if (!skel) {
	    fprintf(stderr, "Failed to load and verify BPF skeleton\n");
	    return 1;
	}
	
	// Attach the program to the tracepoint
	if (program_bpf__attach(skel)) {
		fprintf(stderr, "Failed to attach BPF program\n");
		return 1;
	}
	
	printf("Success! I am listening.\n");
	printf("Run: sudo cat /sys/kernel/tracing/trace_pipe\n");
	printf("Then run 'ls' in another terminal to see output.\n");
	
	// Keep the process alive
	while (1) {
		sleep(1);
	}
	
	return 0;
}

Step 4: Build and Deploy

Here comes the moment of truth:

1. Build inside your VM by running the make command.
2. Copy the resulting program binary to your target machine (Kali, Debian, etc.).
3. Launch the program on the host machine by running the sudo ./program command.

Congratulations, you have achieved true CO-RE!

Why this works? By linking the upstream libbpf statically, your binary carries its own “loader”. When you run it on a host with a modern kernel, the bundled library knows how to read the newer BTF formats, while the system-installed library on your old VM would have crashed.

Step 5: Automated Testing on Multiple Kernels

The remaining part of this blog post is heavily based on this one about Creating your own micro-Linux.

We’ll build a minimal, BPF-capable Linux system from scratch. It covers compiling multiple Kernels, building BusyBox, creating the Initramfs, and running the environments in QEMU.

For that purpose, it is advised to dedicate a new directory for the testing. Inside that root directory, create a new one called shared where you’ll add the eBPF program binaries you build in the previous step.

mkdir automated-tests
cd automated-tests
mkdir shared
cp ../program shared/

You now have the following directory structure:

automated-tests/
    └── shared/
        └── program

Prerequisites

# Libraries
sudo apt install -y build-essential libncurses-dev libelf-dev libssl-dev

# BTF Generation
sudo apt install pahole

# QEMU
sudo apt install qemu-system-x86

A Script to Build the Linux Kernels

This section compiles a list of modern Linux kernels with the specific configurations required for BPF (Berkeley Packet Filter) and BTF (BPF Type Format) support. When we will run it later, it will generate the bzImage-* files for all the versions we want to test.

Create a build-linux-kernels.sh script then make it executable with chmod +x:

#!/bin/bash

# List of kernels to test (LTS versions + latest)
VERSIONS="5.15.148 6.7.4"

for VER in $VERSIONS; do
  echo ">>> Processing $VER"
  # Download & Extract 
  wget -nc https://cdn.kernel.org/pub/linux/kernel/v${VER%%.*}.x/linux-$VER.tar.xz 
  tar -xf linux-$VER.tar.xz 
  cd linux-$VER

  # Generate a default configuration
  make defconfig

  # Configure Debug Info (Prerequisite for BTF)
  ./scripts/config --disable CONFIG_DEBUG_INFO_NONE
  ./scripts/config --disable CONFIG_DEBUG_INFO_REDUCED
  ./scripts/config --disable CONFIG_DEBUG_INFO_SPLIT
  ./scripts/config --enable CONFIG_DEBUG_INFO
  ./scripts/config --enable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
  
  # Enable BTF (CO-RE)
  ./scripts/config --enable CONFIG_DEBUG_INFO_BTF
  ./scripts/config --enable CONFIG_DEBUG_INFO_BTF_MODULES

  # Enable BPF Subsystem
  ./scripts/config --enable CONFIG_BPF
  ./scripts/config --enable CONFIG_BPF_SYSCALL
  ./scripts/config --enable CONFIG_BPF_JIT

  # Enable Tracing (Ftrace)
  ./scripts/config --enable CONFIG_FTRACE
  ./scripts/config --enable CONFIG_FTRACE_SYSCALLS
  ./scripts/config --enable CONFIG_BPF_EVENTS

  # Allow Mounting on Older Kernels (9P Protocol Support)
  ./scripts/config --enable CONFIG_NET_9P
  ./scripts/config --enable CONFIG_9P_FS
  ./scripts/config --enable CONFIG_NET_9P_VIRTIO
  ./scripts/config --enable CONFIG_VIRTIO
  ./scripts/config --enable CONFIG_VIRTIO_PCI
  ./scripts/config --enable CONFIG_VIRTIO_MENU

  # Allow Tracepoints on Older Kernels
  # Required for BPF to attach to Tracepoints
  ./scripts/config --enable CONFIG_PERF_EVENTS
  # Required for legacy path support (debug & tracing)
  ./scripts/config --enable CONFIG_DEBUG_FS

  # Update the config for this version
  make olddefconfig

  # Build
  make -j$(nproc) bzImage
  cp arch/x86_64/boot/bzImage ../bzImage-$VER

  # Return to root
  cd ..

  # Cleanup
  rm -rf linux-$VER/
  rm linux-$VER.tar.xz
done

Note: If you need to clean up a build environment entirely, use make mrproper. This deletes the .config file and all generated files.

A Script to Build BusyBox & Create the Initramfs

BusyBox combines tiny versions of many common UNIX utilities into a single small executable. We configure it as a static binary so it doesn’t require external shared libraries. The Initramfs (Initial RAM Filesystem) is the small filesystem loaded into memory at boot. It contains the init script, which is the very first process started by the kernel. This will generate the initramfs.cpio.gz file.

Create a build-busybox-initramfs.sh script then make it executable with chmod +x:

#!/bin/bash

# Download BusyBox
wget https://github.com/mirror/busybox/archive/refs/tags/1_36_0.tar.gz
tar -xvzf 1_36_0.tar.gz
cd busybox-1_36_0

# Configure for Static Linking
make defconfig
# Force static linking (no shared libs required)
sed -i 's/^# CONFIG_STATIC is not set/CONFIG_STATIC=y/' .config
# Disable TC (Traffic Control) if it causes build issues, or keep if needed
sed -i 's/^CONFIG_TC=y/# CONFIG_TC is not set/' .config

# Compile and Install
make oldconfig
make -j$(nproc)
make install

# Return to root
cd ..

# Create directory structure
mkdir initramfs
mkdir -p initramfs/bin initramfs/sbin initramfs/etc initramfs/proc \
         initramfs/sys initramfs/dev initramfs/usr/bin initramfs/usr/sbin

# Copy BusyBox installation
cp -a busybox-1_36_0/_install/* ./initramfs

# Create the init script
cat > initramfs/init << 'EOF'
#!/bin/sh

# Mount essential pseudo-filesystems
mount -t devtmpfs devtmpfs /dev
mount -t proc none /proc
mount -t sysfs none /sys

# Mount Shared Folder
mkdir -p /mnt/shared
mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt/shared

# Mount Tracing Folder
mkdir -p /sys/kernel/tracing
mount -t tracefs nodev /sys/kernel/tracing

# Mount DebugFS (for older Kernles)
mkdir -p /sys/kernel/debug
mount -t debugfs debugfs /sys/kernel/debug

# Test Automation
# If test.sh exists in the shared folder, run it and shut down.
if [ -f "/mnt/shared/test.sh" ]; then
  echo "--- Running Automated Test ---"
  sh /mnt/shared/test.sh >> /mnt/shared/test.log 2>&1
  echo "--- Test Finished. Shutting down. ---"
  poweroff -f
fi

# Interactive Fallback
echo "Welcome to Micro Linux (Interactive Mode)"
exec /bin/sh
EOF

# Make init executable
chmod +x initramfs/init

# Package the Initramfs (cpio + gzip)
cd initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../initramfs.cpio.gz

# Return to root
cd ..

# Cleanup
rm -rf busybox-1_36_0
rm 1_36_0.tar.gz
rm -rf initramfs

QEMU Flags

Later, we will run the kernels using QEMU, passing in our custom bzImage-* files and initramfs.cpio.gz. We also enable file sharing to easily transfer binaries into the VM.

qemu-system-x86_64 \
    -kernel bzImage \
    -initrd initramfs.cpio.gz \
    -nographic \
    -append "console=ttyS0" \
    -virtfs local,path=./shared,mount_tag=host0,security_model=mapped,id=host0

Understanding the Flags:

• -kernel: The Linux kernel image we compiled.
• -initrd: The Initial RAM Disk we created.
• -nographic + -append "console=ttyS0": Redirects output to the terminal rather than a GUI window.
• -virtfs: Sets up a shared folder using the 9p protocol. mount_tag=host0 is the identifier we will use to mount it inside the guest.

A Script for Automated Tests with QEMU

If you place a script named test.sh in your shared folder, the VM will boot, execute the script, and immediately power off. This allows you to loop through multiple kernels rapidly.

Create a shared/test.sh script then make it executable with chmod +x:

#!/bin/sh

echo "Testing Kernel: $(uname -r)"

echo "Waiting for Tracepoints..."
while [ ! -f /sys/kernel/tracing/events/syscalls/sys_enter_execve/id ]; do
    sleep 0.1
done

echo "Starting BPF Program in background..."
/mnt/shared/program &
BPF_PID=$!

# Give it time to attach
sleep 2

echo "Triggering event..."
cat /sys/kernel/tracing/trace_pipe &

# Give it time to print output
sleep 2

echo "Killing BPF program and exiting..."
kill $BPF_PID

Also create a run-qemu-tests.sh script then make it executable with chmod +x:

#!/bin/sh

# Assumes you have multiple kernels named bzImage-6.6.16, bzImage-5.15.148, etc.
for KERNEL in bzImage*; do 
  echo ">>> Booting $KERNEL" 
  qemu-system-x86_64 -kernel $KERNEL \
    -initrd initramfs.cpio.gz -nographic -append "console=ttyS0" \
    -virtfs local,path=./shared,mount_tag=host0,security_model=mapped,id=host0
done

Building & Testing

You now have the following directory structure:

automated-tests/
    ├── build-busybox-initramfs.sh
    ├── build-linux-kernels.sh
    ├── run-qemu-tests.sh
    └── shared/
        ├── program
        └── test.sh
        

Run the following commands to generate the bzImage-* kernel images and the initramfs.cpio.gz initial RAM filesystem.

# Build the Linux Kernels (might be VERY long! Note that you can reduce the number of kernels in the 'for' loop)
./build-linux-kernels.sh

# Build BusyBox & Create the Initramfs
./build-busybox-initramfs.sh

Now you can run the automated tests:

# Automated Tests with QEMU
./run-qemu-tests.sh

An output log will be generated inside shared/test.log and you will be able to compare how your eBPF program runs against the kernel versions you selected. In another terminal, run to see the logs in live (after test.log has been created):

tail -f shared/test.log

And you should see that output:

Testing Kernel: 5.10.209
Waiting for Tracepoints...
Starting BPF Program in background...
Triggering event...
              sh-76      [000] dN..     5.808107: bpf_trace_printk: Hello from CO-RE!

Killing BPF program and exiting...
Testing Kernel: 5.15.148
Waiting for Tracepoints...
Starting BPF Program in background...
Triggering event...
              sh-81      [000] dN..1     5.607006: bpf_trace_printk: Hello from CO-RE!

Killing BPF program and exiting...
Testing Kernel: 6.1.77
Waiting for Tracepoints...
Starting BPF Program in background...
Triggering event...
             cat-83      [000] d..31     6.334987: bpf_trace_printk: Hello from CO-RE!

Killing BPF program and exiting...
Testing Kernel: 6.7.4
Waiting for Tracepoints...
Starting BPF Program in background...
Triggering event...
              sh-61      [000] .N.21     6.663061: bpf_trace_printk: Hello from CO-RE!

Killing BPF program and exiting...

Conclusion

By following this guide, you have effectively decoupled your development environment from the constraints of your target machine. Instead of fighting with system package managers or worrying about kernel version mismatches, you now have a self-contained build pipeline that bundles a modern, BPF-aware loader directly into your binary.

Furthermore, with the addition of the QEMU automation script, you have moved beyond simple compilation. You now have a testing harness that can boot multiple kernel versions and verify your application in seconds. This provides the ultimate confidence check, proving your code works across years of Linux releases before you ever touch a production server.

This blog post is licensed under CC BY-SA 4.0