XDP-hints as BTF early design discussion phase

XDP-hints as BTF early design discussion phase

[Jesper Dangaard Brouer]

Trying to get started with XDP-hints again.

The fundamental idea is that XDP-hints metadata struct's are defined by 
the kernel, preferably by the kernel module, and described via BTF. As 
BTF layout is defined by kernel, this means userspace (AF_XDP) and 
BPF-progs must adapt to layout (used by running kernel). This imply that 
kernel is free to change layout.

The BTF ID is exposed (to BPF-prog and AF_XDP) on per packet basis, to 
give kernel more freedom to change layout runtime. This push 
responsibility to userspace/BPF-prog of handling different layouts, 
which seems natural. For the kernel this solves many issues around 
concurrency and NIC config changes that affects BTF info available (e.g. 
when BTF layout is allowed to change).

End-goal is to make it easier for kernel drivers can invent new layouts 
to suite new hardware features. Thus, we prefer a solution where 
XDP-hints metadata struct's are defined in the kernel module code.


(Idea below ... please let us know what you think, wrong direction?)

Exploring kernel module code defining the XDP-hints metadata struct.

Kernel module BTFs are now[1][2] exposed through sysfs as 
/sys/kernel/btf/<module-name>. Thus, userspace can use libbpf 
btf__load_module_btf() and others BTF APIs. Started playing here[3].

Credit to Toke, who had an idea that drivers could "say" what struct's 
are available, by defining a union with a known name e.g. 
metadata_hints_avail' and have supported metadata struct's included in 
that union. Then we don't need new APIs for exporting these BTF-metadata 
struct's. To find struct names, we BTF walk this union.


-Jesper

  [1] 
https://lore.kernel.org/bpf/20201110011932.3201430-5-andrii@kernel.org/
  [2] 36e68442d1af ("bpf: Load and verify kernel module BTFs") (Author: 
Andrii Nakryiko)
  [3] 
https://github.com/xdp-project/bpf-examples/blob/master/BTF-playground/

Re: XDP-hints as BTF early design discussion phase

[Andrii Nakryiko]

On Tue, Sep 14, 2021 at 5:23 AM Larysa Zaremba <larysa.zaremba@intel.com> wrote:
>
> From: Jesper Dangaard Brouer <jbrouer@redhat.com>
> Date: Mon, 13 Sep 2021 13:35:04 +0200
>
> > Trying to get started with XDP-hints again.
>
> > The fundamental idea is that XDP-hints metadata struct's are defined by
> > the kernel, preferably by the kernel module, and described via BTF. As
> > BTF layout is defined by kernel, this means userspace (AF_XDP) and
> > BPF-progs must adapt to layout (used by running kernel). This imply that
> > kernel is free to change layout.
>
> > The BTF ID is exposed (to BPF-prog and AF_XDP) on per packet basis, to
> > give kernel more freedom to change layout runtime. This push
> > responsibility to userspace/BPF-prog of handling different layouts,
> > which seems natural. For the kernel this solves many issues around
> > concurrency and NIC config changes that affects BTF info available (e.g.
> > when BTF layout is allowed to change).
>
> Aside from generic XDP hints defined directly in kernel code,
> it should be possible to define custom hints layout inside network driver.
> Kernel module BTF support[1][2] provides the means to do so.
>
> As long as our XDP program is attached to interfaces serviced by a single driver
> and BPF program uses hints layouts only from this particular driver + generic,
> there should be no problem, we can easily obtain a BTF type_id in BPF code
> and compare it to the one sent by driver,
> because BTF type_ids are unique in such case (was discussed earlier).
>
> However, if we want a single program to use layouts from multiple drivers,
> we can’t do it so easily, because in such case same type_id can point
> to absolutely different types. Obviously, probability of both those types being
> hints structures is extremely low,
> but using type_id only would be at least inconsistent.
> This generates the following issues:
>
> 1.      We have to send both BTF ID and type_id in generic hints struct from driver.
>     This takes up precious space inside generic structure.
>     The first solution we’ve came up with is not using full 64 bits, but instead
>     splitting 32 bits we already planned for type_id into for example,
>     12 for BTF_ID and 20 for type_id. Feel free to suggest a better solution.
> 2.      BTF ID needs to be compared in BPF code too,
>     but it’s not that easily obtained as type_id
>     (through  bpf_core_type_id_kernel which expands to __builtin_btf_type_id).
>     This patch[3] suggests that at least at some point there was an intension
>     to return both BTF_ID and type_id in a single 64 bit integer.
>     However, I’ve run a test and bpf_core_type_id_kernel does not seem
>     to return anything on the most significant 32 bits.
>     libbpf source also seems to patch only type_ids into program.
>     Therefore I have a question to BPF CO-RE developers,
>     do I understand the situation correctly?
>     Is there any other existing way to obtain a BTF ID (not type_id)
>     from type name inside BPF code?

I've even posted a patch before that was resolving local BTF types to
the module BTF FD/ID (that was one of the points of contention) and
BTF type ID. See [0] for context. I think we never reached any
conclusion and this never landed.

  [0] https://patchwork.kernel.org/project/netdevbpf/patch/20201205025140.443115-1-andrii@kernel.org/

>
> So I would like to ask for some suggestions in this situation.
>
>
> > End-goal is to make it easier for kernel drivers can invent new layouts
> > to suite new hardware features. Thus, we prefer a solution where
> > XDP-hints metadata struct's are defined in the kernel module code.
>
>
> > (Idea below ... please let us know what you think, wrong direction?)
>
> > Exploring kernel module code defining the XDP-hints metadata struct.
>
> > Kernel module BTFs are now[1][2] exposed through sysfs as
> > /sys/kernel/btf/<module-name>. Thus, userspace can use libbpf
> > btf__load_module_btf() and others BTF APIs. Started playing here[3].
>
> > Credit to Toke, who had an idea that drivers could "say" what struct's
> > are available, by defining a union with a known name e.g.
> > metadata_hints_avail' and have supported metadata struct's included in
> > that union. Then we don't need new APIs for exporting these BTF-metadata
> > struct's. To find struct names, we BTF walk this union.
>
>
> > -Jesper
>
> --Larysa
>
> >   [1]
> > https://lore.kernel.org/bpf/20201110011932.3201430-5-andrii@kernel.org/
> >   [2] 36e68442d1af ("bpf: Load and verify kernel module BTFs") (Author:
> > Andrii Nakryiko)
> >   [3]
> > https://github.com/xdp-project/bpf-examples/blob/master/BTF-playground/
>
> [1]
> https://lwn.net/Articles/836249/
> [2]
> https://www.spinics.net/lists/netdev/msg702049.html
> [3]
> https://reviews.llvm.org/D91489
>
> ________________________________
> Intel Technology Poland sp. z o.o.
> ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
>
> Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.
> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.