[xdp-hints] Re: Basic/Dumb question WAS(Re: Re: XDP-hints via local BTF info

["Toke Høiland-Jørgensen" <toke@redhat.com>]

Tom Herbert <tom@sipanda.io> writes:

> On Mon, Nov 22, 2021 at 5:59 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>>
>> Jamal Hadi Salim <jhs@mojatatu.com> writes:
>>
>> > And it goes something like this:
>> >
>> > Why does the metadata have to go in the DMA descriptors?
>> > Our experience with the XDP metadata is: you start accessing
>> > that there is a performance penalty (extra cache miss(es)).
>> >
>> > Why is the metadata not encapped as part of the data? We
>> > dont have MTU issues on receive since that is entirely a local matter;
>> > meaning the hardware can expand the packet as much as it wants within
>> > the boundaries of alloced DMA buffer space and XDP and any other
>> > subsystem (TC for example) can take advantage of the metadata.
>>
>> Once the hardware learns how to do that, this is absolutely the
>> direction we want to go in. But for existing stuff, the metadata is in
>> the descriptor and needs to be translated somehow...
>>
> Toke,
>
> By existing stuff I think you mean legacy stuff :-)

Well, whatever term you want to use for "the stuff that's in hardware
today" :)

> I tend to agree with Jamal. Descriptor space is exceedingly limited
> and it's unclear to me that the benefits of getting a few bits of
> *hints* from the device outweigh the costs to develop XDP hints and
> perpetually maintain the facility. That is to say it doesn't seem to
> address the two fundamental limitations of the venerable 50 year old
> device driver model: 1) the narrow waist of PCIe bus in expressing
> ancillary information 2) the black box nature of devices that prevent
> the stack from having any visibility into what it's actually doing
> (and hence we can only get hints at best and not real operational data
> for primary protocol processing).
>
> I believe CXL and programmable devices are the emerging architectural
> solution. This will enable split plane acceleration. For instance, if
> we do this right, a NIC would be able to perform all the stateless
> processing of a TCP/IP packet such that when the host gets the packet
> it could immediately jump to the TCP receive processing function ala
> TXDP-- or even more than that the device could process the whole
> received TCP datapath and the host just puts the received data on the
> socket (necessary to solve the TLS offload OOO problem). All this
> requires a very tight integration between the stack and the device to
> the extent that the lines are blurred and the boundary of the software
> stack is effectively pushed into the device (but definitely not TOE!).

This is certainly something that xdp-hints should be able to address.
And it does: the XDP metadata area is right before the packet data, and
while today that bit of memory is not covered by DMA, there is
absolutely no reason why it couldn't be for devices that support putting
arbitrary stuff in there. In which case the driver can just export a BTF
description of what the hardware writes into that area, and XDP can
access it directly using xdp-hints.

All the talk about translating from hardware descriptors applies to
existing hardware (or "legacy" if you want to call it that), but if the
hardware can just write the metadata directly, driver translation is
obviously not needed.

-Toke