Re: XDP-hints: Howto support multiple BTF types per packet basis?

["Toke Høiland-Jørgensen" <toke@redhat.com>]

Michal Swiatkowski <michal.swiatkowski@linux.intel.com> writes:

>> I would expect that the program would decide ahead-of-time which BTF IDs
>> it supports, by something like including the relevant structs from
>> vmlinux.h. And then we need the BTF ID encoded into the packet metadata
>> as well, so that it is possible to check at run-time which driver the
>> packet came from (since a packet can be redirected, so you may end up
>> having to deal with multiple formats in the same XDP program).
>> 
>> Which would allow you to write code like:
>> 
>> if (ctx->has_driver_meta) {
>>   /* this should be at a well-known position, like first (or last) in meta area */
>>   __u32 *meta_btf_id = ctx->data_meta;
>>   
>>   if (*meta_btf_id == BTF_ID_MLX5) {
>>     struct meta_mlx5 *meta = ctx->data_meta;
>>     /* do something with meta */
>>   } else if (meta_btf_id == BTF_ID_I40E) {
>>     struct meta_i40e *meta = ctx->data_meta;
>>     /* do something with meta */
>>   } /* etc */
>> }
>> 
>> and libbpf could do relocations based on the different meta structs,
>> even removing the code for the ones that don't exist on the running
>> kernel.
>
> This looks nice. In this case we need defintions of struct meta_mlx5 and
> struct meta_i40e at build time. How are we going to deliver this to bpf
> core app? This will be available in /sys/kernel/btf/mlx5 and
> /sys/kernel/btf/i40e (if drivers are loaded). Should we dump this to
> vmlinux.h? Or a developer of the xdp program should add this definition
> to his code?

Well, if the driver just defines the struct, the BTF for it will be
automatically part of the driver module BTF. BPF program developers
would need to include this in their programs somehow (similar to how
you'll need to get the type definitions from vmlinux.h today to use
CO-RE); how they do this is up to them. Since this is a compile-time
thing it will probably depend on the project (for instance, BCC includes
a copy of vmlinux.h in their source tree, but you can also just pick out
the structs you need).

> Maybe create another /sys/kernel/btf/hints with vmlinux and hints from
> all drivers which support hints?

It may be useful to have a way for the kernel to export all the hints
currently loaded, so libbpf can just use that when relocating. The
problem of course being that this will only include drivers that are
actually loaded, so users need to make sure to load all their network
drivers before loading any XDP programs. I think it would be better if
the loader could discover all modules *available* on the system, but I'm
not sure if there's a good way to do that.

> Previously in this thread someone mentioned this ___ use case in libbpf
> and proposed creating something like mega xdp hints structure with all
> available fields across all drivers. As I understand this could solve
> the problem about defining correct structure at build time. But how will
> it work when there will be more than one structures with the same name
> before ___? I mean:
> struct xdp_hints___mega defined only in core app
> struct xdp_hints___mlx5 available when mlx5 driver is loaded
> struct xdp_hints___i40e available when i40e driver is loaded
>
> When there will be only one driver loaded should libbpf do correct
> reallocation of fields? What will happen when both of the drivers are
> loaded?

I think we definitely need to make this easy for developers so they
don't have to go and manually track down the driver structs and write
the disambiguation code etc. I.e., the example code I included above
that checks the frame BTF ID and does the loading based on it should be
auto-generated. We already have some precedence for auto-generated code
in vmlinux.h and the bpftool skeletons. So maybe we could have a command
like 'bpftool gen_xdp_meta <fields>' which would go and lookup all the
available driver structs and generate a code helper function that will
extract the driver structs and generate the loader code? So that if,
say, you're interested in rxhash and tstamp you could do:

bpftool gen_xdp_meta rxhash tstamp > my_meta.h

which would then produce my_meta.h with content like:

struct my_meta { /* contains fields specified on the command line */
  u32 rxhash;
  u32 tstamp;
}

struct meta_mlx5 {/*generated from kernel BTF */};
struct meta_i40e {/*generated from kernel BTF */};

static inline int get_xdp_meta(struct xdp_md *ctx, struct my_meta *meta)
{
 if (ctx->has_driver_meta) {
   /* this should be at a well-known position, like first (or last) in meta area */
   __u32 *meta_btf_id = ctx->data_meta;
   
   if (*meta_btf_id == BTF_ID_MLX5) {
     struct meta_mlx5 *meta = ctx->data_meta;
     my_meta->rxhash = meta->rxhash;
     my_meta->tstamp = meta->tstamp;
     return 0;
   } else if (meta_btf_id == BTF_ID_I40E) {
     struct meta_i40e *meta = ctx->data_meta;
     my_meta->rxhash = meta->rxhash;
     my_meta->tstamp = meta->tstamp;
     return 0;
   } /* etc */
 }
 return -ENOENT;
}


-Toke