bpf: implement lookup-free direct value access for maps

This generic extension to BPF maps allows for directly loading
an address residing inside a BPF map value as a single BPF
ldimm64 instruction!

The idea is similar to what BPF_PSEUDO_MAP_FD does today, which
is a special src_reg flag for ldimm64 instruction that indicates
that inside the first part of the double insns's imm field is a
file descriptor which the verifier then replaces as a full 64bit
address of the map into both imm parts. For the newly added
BPF_PSEUDO_MAP_VALUE src_reg flag, the idea is the following:
the first part of the double insns's imm field is again a file
descriptor corresponding to the map, and the second part of the
imm field is an offset into the value. The verifier will then
replace both imm parts with an address that points into the BPF
map value at the given value offset for maps that support this
operation. Currently supported is array map with single entry.
It is possible to support more than just single map element by
reusing both 16bit off fields of the insns as a map index, so
full array map lookup could be expressed that way. It hasn't
been implemented here due to lack of concrete use case, but
could easily be done so in future in a compatible way, since
both off fields right now have to be 0 and would correctly
denote a map index 0.

The BPF_PSEUDO_MAP_VALUE is a distinct flag as otherwise with
BPF_PSEUDO_MAP_FD we could not differ offset 0 between load of
map pointer versus load of map's value at offset 0, and changing
BPF_PSEUDO_MAP_FD's encoding into off by one to differ between
regular map pointer and map value pointer would add unnecessary
complexity and increases barrier for debugability thus less
suitable. Using the second part of the imm field as an offset
into the value does /not/ come with limitations since maximum
possible value size is in u32 universe anyway.

This optimization allows for efficiently retrieving an address
to a map value memory area without having to issue a helper call
which needs to prepare registers according to calling convention,
etc, without needing the extra NULL test, and without having to
add the offset in an additional instruction to the value base
pointer. The verifier then treats the destination register as
PTR_TO_MAP_VALUE with constant reg->off from the user passed
offset from the second imm field, and guarantees that this is
within bounds of the map value. Any subsequent operations are
normally treated as typical map value handling without anything
extra needed from verification side.

The two map operations for direct value access have been added to
array map for now. In future other types could be supported as
well depending on the use case. The main use case for this commit
is to allow for BPF loader support for global variables that
reside in .data/.rodata/.bss sections such that we can directly
load the address of them with minimal additional infrastructure
required. Loader support has been added in subsequent commits for
libbpf library.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

1 files changed, 21 insertions, 7 deletions

diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 1d65e56594db..828518bb947b 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2072,13 +2072,26 @@ static int bpf_map_get_fd_by_id(const union bpf_attr *attr)
 }
 
 static const struct bpf_map *bpf_map_from_imm(const struct bpf_prog *prog,
-                                              unsigned long addr)
+                                              unsigned long addr, u32 *off,
+                                              u32 *type)
 {
+        const struct bpf_map *map;
         int i;
 
-        for (i = 0; i < prog->aux->used_map_cnt; i++)
-                if (prog->aux->used_maps[i] == (void *)addr)
-                        return prog->aux->used_maps[i];
+        for (i = 0, *off = 0; i < prog->aux->used_map_cnt; i++) {
+                map = prog->aux->used_maps[i];
+                if (map == (void *)addr) {
+                        *type = BPF_PSEUDO_MAP_FD;
+                        return map;
+                }
+                if (!map->ops->map_direct_value_meta)
+                        continue;
+                if (!map->ops->map_direct_value_meta(map, addr, off)) {
+                        *type = BPF_PSEUDO_MAP_VALUE;
+                        return map;
+                }
+        }
+
         return NULL;
 }
 
@@ -2086,6 +2099,7 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog)
 {
         const struct bpf_map *map;
         struct bpf_insn *insns;
+        u32 off, type;
         u64 imm;
         int i;
 
@@ -2113,11 +2127,11 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog)
                         continue;
 
                 imm = ((u64)insns[i + 1].imm << 32) | (u32)insns[i].imm;
-                map = bpf_map_from_imm(prog, imm);
+                map = bpf_map_from_imm(prog, imm, &off, &type);
                 if (map) {
-                        insns[i].src_reg = BPF_PSEUDO_MAP_FD;
+                        insns[i].src_reg = type;
                         insns[i].imm = map->id;
-                        insns[i + 1].imm = 0;
+                        insns[i + 1].imm = off;
                         continue;
                 }
         }