[xdp-hints] Re: Basic/Dumb question WAS(Re: Re: XDP-hints via local BTF info

[Tom Herbert <tom@sipanda.io>]

On Mon, Nov 22, 2021 at 5:59 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Jamal Hadi Salim <jhs@mojatatu.com> writes:
>
> > And it goes something like this:
> >
> > Why does the metadata have to go in the DMA descriptors?
> > Our experience with the XDP metadata is: you start accessing
> > that there is a performance penalty (extra cache miss(es)).
> >
> > Why is the metadata not encapped as part of the data? We
> > dont have MTU issues on receive since that is entirely a local matter;
> > meaning the hardware can expand the packet as much as it wants within
> > the boundaries of alloced DMA buffer space and XDP and any other
> > subsystem (TC for example) can take advantage of the metadata.
>
> Once the hardware learns how to do that, this is absolutely the
> direction we want to go in. But for existing stuff, the metadata is in
> the descriptor and needs to be translated somehow...
>
Toke,

By existing stuff I think you mean legacy stuff :-)

I tend to agree with Jamal. Descriptor space is exceedingly limited
and it's unclear to me that the benefits of getting a few bits of
*hints* from the device outweigh the costs to develop XDP hints and
perpetually maintain the facility. That is to say it doesn't seem to
address the two fundamental limitations of the venerable 50 year old
device driver model: 1) the narrow waist of PCIe bus in expressing
ancillary information 2) the black box nature of devices that prevent
the stack from having any visibility into what it's actually doing
(and hence we can only get hints at best and not real operational data
for primary protocol processing).

I believe CXL and programmable devices are the emerging architectural
solution. This will enable split plane acceleration. For instance, if
we do this right, a NIC would be able to perform all the stateless
processing of a TCP/IP packet such that when the host gets the packet
it could immediately jump to the TCP receive processing function ala
TXDP-- or even more than that the device could process the whole
received TCP datapath and the host just puts the received data on the
socket (necessary to solve the TLS offload OOO problem). All this
requires a very tight integration between the stack and the device to
the extent that the lines are blurred and the boundary of the software
stack is effectively pushed into the device (but definitely not TOE!).

Tom

> -Toke
>