XDP hardware hints discussion mail archive
 help / color / mirror / Atom feed
From: Jesper Dangaard Brouer <jbrouer@redhat.com>
To: Martin KaFai Lau <martin.lau@linux.dev>,
	Jesper Dangaard Brouer <jbrouer@redhat.com>,
	Magnus Karlsson <magnus.karlsson@gmail.com>
Cc: brouer@redhat.com, Stanislav Fomichev <sdf@google.com>,
	bpf@vger.kernel.org, netdev@vger.kernel.org,
	xdp-hints@xdp-project.net, larysa.zaremba@intel.com,
	memxor@gmail.com, Lorenzo Bianconi <lorenzo@kernel.org>,
	mtahhan@redhat.com,
	Alexei Starovoitov <alexei.starovoitov@gmail.com>,
	Daniel Borkmann <borkmann@iogearbox.net>,
	Andrii Nakryiko <andrii.nakryiko@gmail.com>,
	dave@dtucker.co.uk, Magnus Karlsson <magnus.karlsson@intel.com>,
	bjorn@kernel.org, Jakub Kicinski <kuba@kernel.org>
Subject: [xdp-hints] Re: [PATCH RFCv2 bpf-next 00/18] XDP-hints: XDP gaining access to HW offload hints via BTF
Date: Tue, 11 Oct 2022 13:57:36 +0200	[thread overview]
Message-ID: <ad360933-953a-7a99-5057-4d452a9a6005@redhat.com> (raw)
In-Reply-To: <c8a712d8-dc97-8df5-6421-a5ccb1357b67@linux.dev>



On 11/10/2022 08.29, Martin KaFai Lau wrote:
> On 10/6/22 8:29 AM, Jesper Dangaard Brouer wrote:
>>
>> On 06/10/2022 11.14, Magnus Karlsson wrote:
>>> On Wed, Oct 5, 2022 at 9:27 PM Martin KaFai Lau 
>>> <martin.lau@linux.dev> wrote:
>>>>
>>>> On 10/4/22 7:15 PM, Stanislav Fomichev wrote:
>>>>> On Tue, Oct 4, 2022 at 6:24 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>>>>>
>>>>>> On Tue, 4 Oct 2022 18:02:56 -0700 Stanislav Fomichev wrote:
>>>>>>> +1, sounds like a good alternative (got your reply while typing)
>>>>>>> I'm not too versed in the rx_desc/rx_queue area, but seems like worst
>>>>>>> case that bpf_xdp_get_hwtstamp can probably receive a xdp_md ctx and
>>>>>>> parse it out from the pre-populated metadata?
>>>>>>
>>>>>> I'd think so, worst case the driver can put xdp_md into a struct
>>>>>> and container_of() to get to its own stack with whatever fields
>>>>>> it needs.
>>>>>
>>>>> Ack, seems like something worth exploring then.
>>>>>
>>>>> The only issue I see with that is that we'd probably have to extend
>>>>> the loading api to pass target xdp device so we can pre-generate
>>>>> per-device bytecode for those kfuncs?
>>>>
>>>> There is an existing attr->prog_ifindex for dev offload purpose.  
>>>> May be we can
>>>> re-purpose/re-use some of the offload API.  How this kfunc can be 
>>>> presented also
>>>> needs some thoughts, could be a new ndo_xxx.... not sure.
>>>>> And this potentially will block attaching the same program
>>>>   > to different drivers/devices?
>>>>> Or, Martin, did you maybe have something better in mind?
>>>>
>>>> If the kfunc/helper is inline, then it will have to be per device.  
>>>> Unless the
>>>> bpf prog chooses not to inline which could be an option but I am 
>>>> also not sure
>>>> how often the user wants to 'attach' a loaded xdp prog to a 
>>>> different device.
>>>> To some extend, the CO-RE hints-loading-code will have to be per 
>>>> device also, no?
>>>>
>>>> Why I asked the kfunc/helper approach is because, from the set, it 
>>>> seems the
>>>> hints has already been available at the driver.  The specific 
>>>> knowledge that the
>>>> xdp prog missing is how to get the hints from the rx_desc/rx_queue.  
>>>> The
>>>> straight forward way to me is to make them (rx_desc/rx_queue) 
>>>> available to xdp
>>>> prog and have kfunc/helper to extract the hints from them only if 
>>>> the xdp prog
>>>> needs it.  The xdp prog can selectively get what hints it needs and 
>>>> then
>>>> optionally store them into the meta area in any layout.
>>>
>>> This sounds like a really good idea to me, well worth exploring. To
>>> only have to pay, performance wise, for the metadata you actually use
>>> is very important. I did some experiments [1] on the previous patch
>>> set of Jesper's and there is substantial overhead added for each
>>> metadata enabled (and fetched from the NIC). This is especially
>>> important for AF_XDP in zero-copy mode where most packets are directed
>>> to user-space (if not, you should be using the regular driver that is
>>> optimized for passing packets to the stack or redirecting to other
>>> devices). In this case, the user knows exactly what metadata it wants
>>> and where in the metadata area it should be located in order to offer
>>> the best performance for the application in question. But as you say,
>>> your suggestion could potentially offer a good performance upside to
>>> the regular XDP path too.
> 
> Yeah, since we are on this flexible hint layout, after reading the 
> replies in other threads, now I am also not sure why we need a 
> xdp_hints_common and probably I am missing something also.  It seems to 
> be most useful in __xdp_build_skb_from_frame. However, the xdp prog can 
> also fill in the xdp_hints_common by itself only when needed instead of 
> having the driver always filling it in.
> 

I *want* the XDP-hints to be populated even when no XDP-prog is running.
The xdp_frame *is* the mini-SKB concept. These XDP-hints are about
adding HW offload hints to this mini-SKB, to allow it grow into a
full-SKB with these offloads.

I could add this purely as a netstack feature, via extending xdp_frame
area with a common struct. For XDP-prog access I could extend xdp_md
with fields that gets UAPI rewrite mapped to access these fields. For
the AF_XDP users this data becomes harder to access, but an XDP-prog
could (spend cycles) moving these offloads into the metadata area, but
why not place them there is the first place.

I think the main point is that I don't see the XDP-prog as the primary
consumer of these hints.
One reason/use-case for letting XDP-prog access these hints prior to
creating a full-SKB is to help fixing up (or providing) offload hints.
The mvneta driver patch highlight this as HW have limited hints, which
an XDP-prog can provide prior to calling netstack.

In this patchset I'm trying to balance the different users. And via BTF
I'm trying hard not to create more UAPI (e.g. more fixed fields avail in
xdp_md that we cannot get rid of). And trying to add driver flexibility
on-top of the common struct.  This flexibility seems to be stalling the
patchset as we haven't found the perfect way to express this (yet) given
BTF layout is per driver.


>>
>> Okay, lets revisit this again.  And let me explain why I believe this
>> isn't going to fly.
>>
>> I was also my initial though, lets just give XDP BPF-prog direct access
>> to the NIC rx_descriptor, or another BPF-prog populate XDP-hints prior
>> to calling XDP-prog.  Going down this path (previously) I learned three
>> things:
>>
>> (1) Understanding/decoding rx_descriptor requires access to the
>> programmers datasheet, because it is very compacted and the mean of the
>> bits depend on other bits and plus current configuration status of the 
>> HW.
>>
>> (2) HW have bugs and for certain chip revisions driver will skip some
>> offload hints.  Thus, chip revisions need to be exported to BPF-progs
>> and handled appropriately.
>>
>> (3) Sometimes the info is actually not available in the rx_descriptor.
>> Often for HW timestamps, the timestamp need to be read from a HW
>> register.  How do we expose this to the BPF-prog?
> 
> hmm.... may be I am missing those hw specific details here.  How would 
> the driver handle the above cases and fill in the xdp_hints in the 
> meta?  Can the same code be called by the xdp prog?
>

As I mentioned above, I want the XDP-hints to be populated even when no 
XDP-prog is running. I don't want the dependency on loading an XDP-prog 
to get the hints populated, as e.g. netstack is one of the users.


>>
>> Notice that this patchset doesn't block this idea, as it is orthogonal.
>> After we have established a way to express xdp_hints layouts via BTF,
>> then we can still add a pre-XDP BPF-prog that populates the XDP-hints,
>> and squeeze out more performance by skipping some of the offloads that
>> your-specific-XDP-prog are not interested in.
>>
>> --Jesper
>>
> 


  reply	other threads:[~2022-10-11 11:57 UTC|newest]

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-07 15:45 [xdp-hints] " Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 01/18] libbpf: factor out BTF loading from load_module_btfs() Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 02/18] libbpf: try to load vmlinux BTF from the kernel first Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 03/18] libbpf: patch module BTF obj+type ID into BPF insns Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 04/18] net: create xdp_hints_common and set functions Jesper Dangaard Brouer
2022-09-09 10:49   ` [xdp-hints] " Burakov, Anatoly
2022-09-09 14:13     ` Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 05/18] net: add net_device feature flag for XDP-hints Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 06/18] xdp: controlling XDP-hints from BPF-prog via helper Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 07/18] i40e: Refactor i40e_ptp_rx_hwtstamp Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 08/18] i40e: refactor i40e_rx_checksum with helper Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 09/18] bpf: export btf functions for modules Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 10/18] btf: Add helper for kernel modules to lookup full BTF ID Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 11/18] i40e: add XDP-hints handling Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 12/18] net: use XDP-hints in xdp_frame to SKB conversion Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 13/18] mvneta: add XDP-hints support Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 14/18] i40e: Add xdp_hints_union Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 15/18] ixgbe: enable xdp-hints Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 16/18] ixgbe: add rx timestamp xdp hints support Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 17/18] xsk: AF_XDP xdp-hints support in desc options Jesper Dangaard Brouer
2022-09-08  8:06   ` [xdp-hints] " Magnus Karlsson
2022-09-08 10:10     ` Maryam Tahhan
2022-09-08 15:04       ` Jesper Dangaard Brouer
2022-09-09  6:43         ` Magnus Karlsson
2022-09-09  8:12           ` Maryam Tahhan
2022-09-09  9:42             ` Jesper Dangaard Brouer
2022-09-09 10:14               ` Magnus Karlsson
2022-09-09 12:35                 ` Jesper Dangaard Brouer
2022-09-09 12:44                   ` Magnus Karlsson
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 18/18] ixgbe: AF_XDP xdp-hints processing in ixgbe_clean_rx_irq_zc Jesper Dangaard Brouer
2022-09-08  9:30 ` [xdp-hints] Re: [PATCH RFCv2 bpf-next 00/18] XDP-hints: XDP gaining access to HW offload hints via BTF Alexander Lobakin
2022-09-09 13:48   ` Jesper Dangaard Brouer
2022-10-03 23:55 ` sdf
2022-10-04  9:29   ` Jesper Dangaard Brouer
2022-10-04 18:26     ` Stanislav Fomichev
2022-10-05  0:25       ` Martin KaFai Lau
2022-10-05  0:59         ` Jakub Kicinski
2022-10-05  1:02           ` Stanislav Fomichev
2022-10-05  1:24             ` Jakub Kicinski
2022-10-05  2:15               ` Stanislav Fomichev
2022-10-05 19:26                 ` Martin KaFai Lau
2022-10-06  9:14                   ` Magnus Karlsson
2022-10-06 15:29                     ` Jesper Dangaard Brouer
2022-10-11  6:29                       ` Martin KaFai Lau
2022-10-11 11:57                         ` Jesper Dangaard Brouer [this message]
2022-10-05 10:06             ` Toke Høiland-Jørgensen
2022-10-05 18:47               ` sdf
2022-10-06  8:19                 ` Maryam Tahhan
2022-10-06 17:22                   ` sdf
2022-10-05 14:19             ` Jesper Dangaard Brouer
2022-10-06 14:59               ` Jakub Kicinski
2022-10-05 13:43         ` Jesper Dangaard Brouer
2022-10-05 16:29       ` Jesper Dangaard Brouer
2022-10-05 18:43         ` sdf
2022-10-06 17:47           ` Jesper Dangaard Brouer
2022-10-07 15:05             ` David Ahern
2022-10-05 13:14     ` Burakov, Anatoly

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.xdp-project.net/postorius/lists/xdp-hints.xdp-project.net/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ad360933-953a-7a99-5057-4d452a9a6005@redhat.com \
    --to=jbrouer@redhat.com \
    --cc=alexei.starovoitov@gmail.com \
    --cc=andrii.nakryiko@gmail.com \
    --cc=bjorn@kernel.org \
    --cc=borkmann@iogearbox.net \
    --cc=bpf@vger.kernel.org \
    --cc=brouer@redhat.com \
    --cc=dave@dtucker.co.uk \
    --cc=kuba@kernel.org \
    --cc=larysa.zaremba@intel.com \
    --cc=lorenzo@kernel.org \
    --cc=magnus.karlsson@gmail.com \
    --cc=magnus.karlsson@intel.com \
    --cc=martin.lau@linux.dev \
    --cc=memxor@gmail.com \
    --cc=mtahhan@redhat.com \
    --cc=netdev@vger.kernel.org \
    --cc=sdf@google.com \
    --cc=xdp-hints@xdp-project.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox