[xdp-hints] Re: [RFC bpf-next 0/5] xdp: hints via kfuncs

[John Fastabend <john.fastabend@gmail.com>]

Jakub Kicinski wrote:
> On Fri, 28 Oct 2022 16:16:17 -0700 John Fastabend wrote:
> > > > And it's actually harder to abstract away inter HW generation
> > > > differences if the user space code has to handle all of it.  
> > 
> > I don't see how its any harder in practice though?
> 
> You need to find out what HW/FW/config you're running, right?
> And all you have is a pointer to a blob of unknown type.

Yep. I guess I'm in the position of already having to do this
somewhat to collect stats from the device. Although its maybe
a bit more involved here by vendors that are versioning the
descriptors.

Also nit, its not unknown type we know the full type by BTF.

> 
> Take timestamps for example, some NICs support adjusting the PHC 
> or doing SW corrections (with different versions of hw/fw/server
> platforms being capable of both/one/neither).

Its worse actually.

Having started to do this timestamping it is not at all consistent
across nics so I think we are stuck having to know hw specifics
here regardless. Also some nics will timestamp all RX pkts, some
specific pkts, some require configuration to decide which mode
to run in and so on. You then end up with a matrix of features
supported by hw/fw/sw and desired state and I can't see any way
around this.

> 
> Sure you can extract all this info with tracing and careful
> inspection via uAPI. But I don't think that's _easier_.
> And the vendors can't run the results thru their validation 
> (for whatever that's worth).

I think you hit our point of view differences below. See I
don't want to depend on the vendor. I want access to the
fields otherwise I'm stuck working with vendors on their
time frames. You have the other perspective of supporting the
NIC and ability to update kernels where as I still live with
4.18/4.19 kernels (even 4.14 sometimes). So what we land now
needs to work in 5 years still.

> 
> > > I've had the same concern:
> > > 
> > > Until we have some userspace library that abstracts all these details,
> > > it's not really convenient to use. IIUC, with a kptr, I'd get a blob
> > > of data and I need to go through the code and see what particular type
> > > it represents for my particular device and how the data I need is
> > > represented there. There are also these "if this is device v1 -> use
> > > v1 descriptor format; if it's a v2->use this another struct; etc"
> > > complexities that we'll be pushing onto the users. With kfuncs, we put
> > > this burden on the driver developers, but I agree that the drawback
> > > here is that we actually have to wait for the implementations to catch
> > > up.  
> > 
> > I agree with everything there, you will get a blob of data and then
> > will need to know what field you want to read using BTF. But, we
> > already do this for BPF programs all over the place so its not a big
> > lift for us. All other BPF tracing/observability requires the same
> > logic. I think users of BPF in general perhaps XDP/tc are the only
> > place left to write BPF programs without thinking about BTF and
> > kernel data structures.
> > 
> > But, with proposed kptr the complexity lives in userspace and can be
> > fixed, added, updated without having to bother with kernel updates, etc.
> > From my point of view of supporting Cilium its a win and much preferred
> > to having to deal with driver owners on all cloud vendors, distributions,
> > and so on.
> > 
> > If vendor updates firmware with new fields I get those immediately.
> 
> Conversely it's a valid concern that those who *do* actually update
> their kernel regularly will have more things to worry about.

I'm not sure if a kptr_func is any harder to write than a user space
relocation for that func? In tetragon and cilium we've done userspace
rewrites for some time. Happy to generalize that infra into kernel
repo if that helps. IMO having a libhw.h in kernel tree ./tools/bpf
directory would work.

> 
> > > Jakub mentions FW and I haven't even thought about that; so yeah, bpf
> > > programs might have to take a lot of other state into consideration
> > > when parsing the descriptors; all those details do seem like they
> > > belong to the driver code.  
> > 
> > I would prefer to avoid being stuck on requiring driver writers to
> > be involved. With just a kptr I can support the device and any
> > firwmare versions without requiring help.
> 
> 1) where are you getting all those HW / FW specs :S

Most are public docs of course vendors have internal docs with more
details but what can you do. Also source code has the structs.

> 2) maybe *you* can but you're not exactly not an ex-driver developer :S

Sure :) but we put a libhw.h file in kernel and test with selftests
(which will be hard without hardware) and then not everyone needs
to be a driver internals expert.

> 
> > > Feel free to send it early with just a handful of drivers implemented;
> > > I'm more interested about bpf/af_xdp/user api story; if we have some
> > > nice sample/test case that shows how the metadata can be used, that
> > > might push us closer to the agreement on the best way to proceed.  
> > 
> > I'll try to do a intel and mlx implementation to get a cross section.
> > I have a good collection of nics here so should be able to show a
> > couple firmware versions. It could be fine I think to have the raw
> > kptr access and then also kfuncs for some things perhaps.
> > 
> > > > I'd prefer if we left the door open for new vendors. Punting descriptor
> > > > parsing to user space will indeed result in what you just said - major
> > > > vendors are supported and that's it.  
> > 
> > I'm not sure about why it would make it harder for new vendors? I think
> > the opposite, 
> 
> TBH I'm only replying to the email because of the above part :)
> I thought this would be self evident, but I guess our perspectives 
> are different.

Yep.

> 
> Perhaps you look at it from the perspective of SW running on someone
> else's cloud, an being able to move to another cloud, without having 
> to worry if feature X is available in xdp or just skb.

Exactly. I have SW running in a data center or cloud for a security
team or ops team and they don't own the platform usually. Anyways
the platform team is going to stay on a LTS kernel for at least a
year or two most likely. I maintain the SW and some 3rd party
nic vendor may not even know about my SW (cilium/tetragon).

> 
> I look at it from the perspective of maintaining a cloud, with people
> writing random XDP applications. If I swap a NIC from an incumbent to a
> (superior) startup, and cloud users are messing with raw descriptor -
> I'd need to go find every XDP program out there and make sure it
> understands the new descriptors.

I get it. Its interesting that you wouldn't tell the XDP programmers
to deal with it which is my case. My $.02 is a userspace lib could
abstract this easier than a kernel func and also add new features
without rolling new kernels.

> 
> There is a BPF foundation or whatnot now - what about starting a
> certification program for cloud providers and making it clear what
> features must be supported to be compatible with XDP 1.0, XDP 2.0 etc?

Maybe but still stuck on kernel versions.

> 
> > it would be easier because I don't need vendor support at all.
> 
> Can you support the enfabrica NIC on day 1? :) To an extent, its just
> shifting the responsibility from the HW vendor to the middleware vendor.

Yep. With important detail that I can run new features on old kernels
even if vendor didn't add rxhash or timestamp out the gate.

Nothing stops the hw vendor from contributing to this in kernel
source bpf lib with the features though.

> 
> > Thinking it over seems there could be room for both.
> 
> Are you thinking more or less Stan's proposal but with one of 
> the callbacks being "give me the raw thing"? Probably as a ro dynptr?
> Possible, but I don't think we need to hold off Stan's work.

Yeah that was my thinking. Both could coexist. OTOH doing it in
BPF program lib side seems cleaner to me from a kernel maitenance
and hardware support. We've had trouble getting drivers to support
XDP features so adding more requirements of entry seems problematic
to me when we can avoid it.