kernel/linux.git/include/linux/bpf.h, branch v7.3-rc1

bpf: Add source and instruction diagnostic context

2026-08-15T18:11:16+00:00

Teach verifier diagnostics to annotate an instruction with BTF source line information and nearby BPF instructions. The renderer keeps source text in a fixed-width lane and prints instructions in a stable right-hand gutter. Wrap annotation text under the source line so long error labels remain readable while the source and instruction lanes keep their fixed layout. Keeping source and instruction context in one commit preserves the visual layout contract that later diagnostic reports rely on. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-3-memxor@gmail.com Signed-off-by: Eduard Zingerman

bpf: Populate mmap-able array map memory lazily

2026-08-14T22:31:30+00:00

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all). Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path. This makes mmap() O(1), and munmap() proportional to the number of pages that were actually faulted in rather than to the size of the map. The handler is reached through a new optional ->map_mmap_fault callback. Maps that provide it get a vm_operations_struct with a .fault handler; maps that populate their mapping eagerly keep the one they had. Both share the same open/close callbacks, so the existing VMA accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized rather than each map installing its own vm_operations_struct. Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE. Kernel-side access to the map (via the vmalloc address) is unaffected. Time for one mmap()+munmap() of an 8MiB mmap-able array map: before after no MAP_POPULATE, no access 226us 1.1us no MAP_POPULATE, access all pages 236us 1341us MAP_POPULATE, no access 312us 493us MAP_POPULATE, access all pages 318us 519us Mapping without touching the data, which is what this change targets, gets ~160x cheaper. Faulting in the whole mapping one page at a time is more expensive than the eager remap_vmalloc_range() loop, so users that do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE is not free before this change either: it adds ~85us (226us => 312us) for no benefit, as the mapping is already fully populated. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Song Liu Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814155623.111565-1-song@kernel.org

bpf: Make bpf_trampoline_multi_detach return void

2026-08-13T00:52:23+00:00

bpf_trampoline_multi_detach() always returns 0 and the sole caller ignores the return value. Change it to return void and drop the WARN_ON_ONCE at the call site. Signed-off-by: Hui Zhu Acked-by: Leon Hwang Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/12beba657f5c9e86a016a097750209287a2f262a.1786412280.git.zhuhui@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Support arena and arena__nullable on struct_ops arguments

2026-08-08T10:03:26+00:00

A struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in. Add the __arena and __arena__nullable stub argument suffixes to make this convenient. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). The trampoline converts the value while saving the arguments into the BPF ctx, ctx[slot] = (u32)(kaddr - kern_vm_start), so the program never sees a kernel address and nothing rewrites the ctx after the fact. The converted value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address. __arena converts unconditionally and the kernel caller must not pass NULL. __arena__nullable preserves NULL, tested on the full 64-bit kernel pointer, and surfaces to the verifier as PTR_TO_ARENA (but not as a PTR_TO_ARENA | PTR_MAYBE_NULL). The reason is that PTR_TO_ARENA in the program's type state already encompasses NULL-ness, so it is not meaningful to force a NULL check for the program. The composite suffix intentionally ends in __nullable. Classify __arena__nullable before the generic suffix so scalar arena pointees do not take the generic nullable BTF pointer path. This patch adds the generic side. prepare_arg_info() records arena and nullable argument flags in the struct_ops function model, and bpf_tramp_arena_base() returns the arena base for a single-program struct_ops indirect trampoline. Only that trampoline converts: its program's arena is fixed at generation time. Generic trampolines can mix programs with different arenas and reject arena context arguments defensively, which is unreachable today as only struct_ops programs carry them. Architectures that do not implement the conversion are gated out at verification time with bpf_jit_supports_arena_args(). Co-developed-by: Kumar Kartikeya Dwivedi Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-6-memxor@gmail.com Signed-off-by: Eduard Zingerman

bpf: Support arena and arena__nullable kfunc argument suffixes

2026-08-08T10:03:26+00:00

Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation. Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments. __arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call. The double separator makes the annotations composable: __arena__nullable also ends in __nullable and naturally follows the common nullable argument path. Plain __arena follows that path too for verifier type checking because both forms accept a constant zero; the function-model flag still determines whether the JIT preserves NULL or rebases it to the arena base. This patch adds the verifier side: the suffixes are recognized in check_kfunc_args() and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state in insn_aux_data. JITs declare support with bpf_jit_supports_arena_args() and verification fails with -ENOTSUPP elsewhere. Co-developed-by: Kumar Kartikeya Dwivedi Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-5-memxor@gmail.com Signed-off-by: Eduard Zingerman

bpf, riscv: Add and use bpf_atomic_is_load_acq() helper

2026-08-07T12:57:13+00:00

A load-acquire is the only BPF_STX class instruction that reads from src_reg into dst_reg, that is, it has the operand roles of a BPF_LDX. JIT code which tells loads from stores apart by instruction class alone has to special case it, for example when deciding which register holds the faulting address and which one to clear from an exception handler. riscv64 already does so, open coded as a bare insn->imm test. Add a bpf_atomic_is_load_acq() helper and convert riscv64 over to it, so that the x86-64 and arm64 JITs can use the same helper in subsequent patches. Unlike bpf_atomic_is_load_store(), which presumes that its argument is already known to be a BPF_ATOMIC instruction, the new helper is called from code which still sees all instruction classes, so it checks class and mode itself. Also, move bpf_atomic_is_load_store() to filter.h next to BPF_ATOMIC_OP, so that both helpers stay together. No functional change intended. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260806201047.333389-2-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Remove unused BTF_FMODEL_STRUCT_ARG

2026-08-03T12:55:22+00:00

Commit 814cba835ef6 ("bpf, x86: Fix trampoline stack size for 128-bit arguments") changed the x86 trampoline to compute the number of registers from arg_size for every argument, which removed the last user of BTF_FMODEL_STRUCT_ARG. No other architecture or verifier code looks at the flag, so remove the macro and the code in __get_type_fmodel_flags() which sets it. Keep BTF_FMODEL_SIGNED_ARG at BIT(1) rather than renumbering it to BIT(0), so BIT(0) is available for a future flag. No functional change. Signed-off-by: Yonghong Song Signed-off-by: Daniel Borkmann Acked-by: Leon Hwang Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20260803052726.2821447-1-yonghong.song@linux.dev

bpf: Generate kfunc argument prototype at add-call time

2026-08-02T22:31:26+00:00

Kfunc argument checking re-derives each argument's kfunc_ptr_arg_type from BTF on every verification of a call in check_kfunc_args(). Now that get_kfunc_arg_type() is a function of the kfunc's BTF alone, it no longer inspects register state. The classification can be computed once when the call is added and cached. This is a step toward describing kfuncs with a bpf_func_proto and sharing the helper argument-checking path. Generate the classification at bpf_add_kfunc_call() time: - Extend struct bpf_func_proto to be able to describe a kfunc: widen arg_type[] and the arg_btf_id[]/arg_size[] union from 5 to MAX_BPF_FUNC_ARGS, since a kfunc may take up to 12 arguments (5 in registers, 7 on the stack). - Embed a bpf_func_proto in struct bpf_kfunc_desc, populated by gen_kfunc_arg_proto() which runs get_kfunc_arg_type() for each argument and stores the result in proto.arg_type[]. Grow the descriptor table's descs[] as a flexible array to not waste memory. - check_kfunc_args() reads the cached classification from meta->fn The KF_ARG_PTR_TO_CTX classification depends on the resolved program type, and for BPF_PROG_TYPE_EXT that is the target program's type, which resolve_prog_type() reads from prog->aux->saved_dst_prog_type. That field is normally recorded later during verification in check_attach_btf_id(), after bpf_add_kfunc_call() has run. Record saved_dst_prog_type and saved_dst_attach_type from dst_prog at program load time in bpf_prog_load() so the resolved type is available at add-call time without reordering check_attach_btf_id(). This keeps e.g. an freplace of an XDP program calling bpf_xdp_metadata_rx_hash() classifying its struct xdp_md * argument as context. The classification result is unchanged; it is only computed earlier and cached. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-19-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Rename ARG_CONST_SIZE{,_OR_ZERO} to ARG_MEM_SIZE{,_OR_ZERO}

2026-08-02T22:29:14+00:00

ARG_CONST_SIZE does not require a constant: check_mem_size_reg() accepts any bounded scalar and verifies the memory access against its maximum (reg_umax). Rename ARG_CONST_SIZE and ARG_CONST_SIZE_OR_ZERO to ARG_MEM_SIZE and ARG_MEM_SIZE_OR_ZERO to reflect that. ARG_CONST_ALLOC_ SIZE_OR_ZERO, which does require a constant, is left unchanged. Pure rename, no functional change. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-10-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Fix WARNING in bpf_tracing_link_release

2026-07-24T20:34:44+00:00

The trampoline could be corrupted by the blindly 'tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX' in verifier. 1. A fexit attached to a tail_call_reachable prog. 'tr->flags' became 'BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_TAIL_CALL_CTX'. And, the trampoline would poke the target prog's nop insn using jmp insn instead of call insn. 2. Another fexit loaded with the same tail_call_reachable prog target. 'tr->flags' became 'BPF_TRAMP_F_TAIL_CALL_CTX'. 3. Close the first fexit link. Due to no BPF_TRAMP_F_CALL_ORIG in 'tr->flags', the trampoline will fail to restore the prog's nop insn using call insn. [ 3.410719] WARNING: kernel/bpf/syscall.c:3551 at bpf_tracing_link_release+0x53/0x60, CPU#1: test_progs/98 ... [ 3.428793] bpf_link_free+0x58/0x130 [ 3.429293] bpf_link_release+0x23/0x30 Fix the warning by updating 'tr->flags' with '|=' and lock. Fixes: 2b5dcb31a19a ("bpf, x64: Fix tailcall infinite loop") Signed-off-by: Leon Hwang Reviewed-by: Pu Lehui Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20260722151909.69142-2-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi