Re: XDP-hints: Howto support multiple BTF types per packet basis?

["Toke Høiland-Jørgensen" <toke@redhat.com>]

John Fastabend <john.fastabend@gmail.com> writes:

>> > > union and independent set of BTFs are two different things, I'll let
>> > > you guys figure out which one you need, but I replied how it could
>> > > look like in CO-RE world
>> >
>> > I think a union is sufficient and more aligned with how the
>> > hardware would actually work.
>> 
>> Sure. And I think those are two orthogonal concerns. You can start
>> with a single struct mynic_metadata with union inside it, and later
>> add the ability to swap mynic_metadata with another
>> mynic_metadata___v2 that will have a similar union but with a
>> different layout.
>
> Right and then you just have normal upgrade/downgrade problems with
> any struct.
>
> Seems like a workable path to me. But, need to circle back to the
> what we want to do with it part that Jesper replied to.

So while this seems to be a viable path for getting libbpf to do all the
relocations (and thanks for hashing that out, I did not have a good grip
of the details), doing it all in userspace means that there is no way
for the XDP program to react to changes once it has been loaded. So this
leaves us with a selection of non-very-attractive options, IMO. I.e.,
we would have to:

- have to block any modifications to the hardware config that would
  change the metadata format; this will probably result in irate users

- require XDP programs to deal with all possible metadata permutations
  supported by that driver (by exporting them all via a BTF union or
  similar); this means a potential for combinatorial explosion of config
  options and as NICs become programmable themselves I'm not even sure
  if it's possible for the driver to know ahead of time

- throw up our hands and just let the user deal with it (i.e., to
  nothing and so require XDP programs to be reloaded if the NIC config
  changes); this is not very friendly and is likely to lead to subtle
  bugs if an XDP program parses the metadata assuming it is in a
  different format than it is

Given that hardware config changes are not just done by ethtool, but
also by things like running `tcpdump -j`, I really think we have to
assume that they can be quite dynamic; which IMO means we have to solve
this as part of the initial design. And I have a hard time seeing how
this is possible without involving the kernel somehow.

Unless I'm missing something? WDYT?

-Toke