XDP hardware hints discussion mail archive
 help / color / mirror / Atom feed
From: Martin KaFai Lau <martin.lau@linux.dev>
To: Jesper Dangaard Brouer <jbrouer@redhat.com>,
	Magnus Karlsson <magnus.karlsson@gmail.com>
Cc: brouer@redhat.com, Stanislav Fomichev <sdf@google.com>,
	bpf@vger.kernel.org, netdev@vger.kernel.org,
	xdp-hints@xdp-project.net, larysa.zaremba@intel.com,
	memxor@gmail.com, Lorenzo Bianconi <lorenzo@kernel.org>,
	mtahhan@redhat.com,
	Alexei Starovoitov <alexei.starovoitov@gmail.com>,
	Daniel Borkmann <borkmann@iogearbox.net>,
	Andrii Nakryiko <andrii.nakryiko@gmail.com>,
	dave@dtucker.co.uk, Magnus Karlsson <magnus.karlsson@intel.com>,
	bjorn@kernel.org, Jakub Kicinski <kuba@kernel.org>
Subject: [xdp-hints] Re: [PATCH RFCv2 bpf-next 00/18] XDP-hints: XDP gaining access to HW offload hints via BTF
Date: Mon, 10 Oct 2022 23:29:07 -0700	[thread overview]
Message-ID: <c8a712d8-dc97-8df5-6421-a5ccb1357b67@linux.dev> (raw)
In-Reply-To: <cc7e4a27-93ab-e9de-55e0-10029948d738@redhat.com>

On 10/6/22 8:29 AM, Jesper Dangaard Brouer wrote:
> 
> On 06/10/2022 11.14, Magnus Karlsson wrote:
>> On Wed, Oct 5, 2022 at 9:27 PM Martin KaFai Lau <martin.lau@linux.dev> wrote:
>>>
>>> On 10/4/22 7:15 PM, Stanislav Fomichev wrote:
>>>> On Tue, Oct 4, 2022 at 6:24 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>>>>
>>>>> On Tue, 4 Oct 2022 18:02:56 -0700 Stanislav Fomichev wrote:
>>>>>> +1, sounds like a good alternative (got your reply while typing)
>>>>>> I'm not too versed in the rx_desc/rx_queue area, but seems like worst
>>>>>> case that bpf_xdp_get_hwtstamp can probably receive a xdp_md ctx and
>>>>>> parse it out from the pre-populated metadata?
>>>>>
>>>>> I'd think so, worst case the driver can put xdp_md into a struct
>>>>> and container_of() to get to its own stack with whatever fields
>>>>> it needs.
>>>>
>>>> Ack, seems like something worth exploring then.
>>>>
>>>> The only issue I see with that is that we'd probably have to extend
>>>> the loading api to pass target xdp device so we can pre-generate
>>>> per-device bytecode for those kfuncs?
>>>
>>> There is an existing attr->prog_ifindex for dev offload purpose.  May be we can
>>> re-purpose/re-use some of the offload API.  How this kfunc can be presented also
>>> needs some thoughts, could be a new ndo_xxx.... not sure.
>>>> And this potentially will block attaching the same program
>>>   > to different drivers/devices?
>>>> Or, Martin, did you maybe have something better in mind?
>>>
>>> If the kfunc/helper is inline, then it will have to be per device.  Unless the
>>> bpf prog chooses not to inline which could be an option but I am also not sure
>>> how often the user wants to 'attach' a loaded xdp prog to a different device.
>>> To some extend, the CO-RE hints-loading-code will have to be per device also, 
>>> no?
>>>
>>> Why I asked the kfunc/helper approach is because, from the set, it seems the
>>> hints has already been available at the driver.  The specific knowledge that the
>>> xdp prog missing is how to get the hints from the rx_desc/rx_queue.  The
>>> straight forward way to me is to make them (rx_desc/rx_queue) available to xdp
>>> prog and have kfunc/helper to extract the hints from them only if the xdp prog
>>> needs it.  The xdp prog can selectively get what hints it needs and then
>>> optionally store them into the meta area in any layout.
>>
>> This sounds like a really good idea to me, well worth exploring. To
>> only have to pay, performance wise, for the metadata you actually use
>> is very important. I did some experiments [1] on the previous patch
>> set of Jesper's and there is substantial overhead added for each
>> metadata enabled (and fetched from the NIC). This is especially
>> important for AF_XDP in zero-copy mode where most packets are directed
>> to user-space (if not, you should be using the regular driver that is
>> optimized for passing packets to the stack or redirecting to other
>> devices). In this case, the user knows exactly what metadata it wants
>> and where in the metadata area it should be located in order to offer
>> the best performance for the application in question. But as you say,
>> your suggestion could potentially offer a good performance upside to
>> the regular XDP path too.

Yeah, since we are on this flexible hint layout, after reading the replies in 
other threads, now I am also not sure why we need a xdp_hints_common and 
probably I am missing something also.  It seems to be most useful in 
__xdp_build_skb_from_frame.  However, the xdp prog can also fill in the 
xdp_hints_common by itself only when needed instead of having the driver always 
filling it in.

> 
> Okay, lets revisit this again.  And let me explain why I believe this
> isn't going to fly.
> 
> I was also my initial though, lets just give XDP BPF-prog direct access
> to the NIC rx_descriptor, or another BPF-prog populate XDP-hints prior
> to calling XDP-prog.  Going down this path (previously) I learned three
> things:
> 
> (1) Understanding/decoding rx_descriptor requires access to the
> programmers datasheet, because it is very compacted and the mean of the
> bits depend on other bits and plus current configuration status of the HW.
> 
> (2) HW have bugs and for certain chip revisions driver will skip some
> offload hints.  Thus, chip revisions need to be exported to BPF-progs
> and handled appropriately.
> 
> (3) Sometimes the info is actually not available in the rx_descriptor.
> Often for HW timestamps, the timestamp need to be read from a HW
> register.  How do we expose this to the BPF-prog?

hmm.... may be I am missing those hw specific details here.  How would the 
driver handle the above cases and fill in the xdp_hints in the meta?  Can the 
same code be called by the xdp prog?

> 
>> [1] 
>> https://lore.kernel.org/bpf/CAJ8uoz1XVqVCpkKo18qbkh6jq_Lejk24OwEWCB9cWhokYLEBDQ@mail.gmail.com/
> 
> 
> Notice that this patchset doesn't block this idea, as it is orthogonal.
> After we have established a way to express xdp_hints layouts via BTF,
> then we can still add a pre-XDP BPF-prog that populates the XDP-hints,
> and squeeze out more performance by skipping some of the offloads that
> your-specific-XDP-prog are not interested in.
> 
> --Jesper
> 


  reply	other threads:[~2022-10-11  6:29 UTC|newest]

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-07 15:45 [xdp-hints] " Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 01/18] libbpf: factor out BTF loading from load_module_btfs() Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 02/18] libbpf: try to load vmlinux BTF from the kernel first Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 03/18] libbpf: patch module BTF obj+type ID into BPF insns Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 04/18] net: create xdp_hints_common and set functions Jesper Dangaard Brouer
2022-09-09 10:49   ` [xdp-hints] " Burakov, Anatoly
2022-09-09 14:13     ` Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 05/18] net: add net_device feature flag for XDP-hints Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 06/18] xdp: controlling XDP-hints from BPF-prog via helper Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 07/18] i40e: Refactor i40e_ptp_rx_hwtstamp Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 08/18] i40e: refactor i40e_rx_checksum with helper Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 09/18] bpf: export btf functions for modules Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 10/18] btf: Add helper for kernel modules to lookup full BTF ID Jesper Dangaard Brouer
2022-09-07 15:45 ` [xdp-hints] [PATCH RFCv2 bpf-next 11/18] i40e: add XDP-hints handling Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 12/18] net: use XDP-hints in xdp_frame to SKB conversion Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 13/18] mvneta: add XDP-hints support Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 14/18] i40e: Add xdp_hints_union Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 15/18] ixgbe: enable xdp-hints Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 16/18] ixgbe: add rx timestamp xdp hints support Jesper Dangaard Brouer
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 17/18] xsk: AF_XDP xdp-hints support in desc options Jesper Dangaard Brouer
2022-09-08  8:06   ` [xdp-hints] " Magnus Karlsson
2022-09-08 10:10     ` Maryam Tahhan
2022-09-08 15:04       ` Jesper Dangaard Brouer
2022-09-09  6:43         ` Magnus Karlsson
2022-09-09  8:12           ` Maryam Tahhan
2022-09-09  9:42             ` Jesper Dangaard Brouer
2022-09-09 10:14               ` Magnus Karlsson
2022-09-09 12:35                 ` Jesper Dangaard Brouer
2022-09-09 12:44                   ` Magnus Karlsson
2022-09-07 15:46 ` [xdp-hints] [PATCH RFCv2 bpf-next 18/18] ixgbe: AF_XDP xdp-hints processing in ixgbe_clean_rx_irq_zc Jesper Dangaard Brouer
2022-09-08  9:30 ` [xdp-hints] Re: [PATCH RFCv2 bpf-next 00/18] XDP-hints: XDP gaining access to HW offload hints via BTF Alexander Lobakin
2022-09-09 13:48   ` Jesper Dangaard Brouer
2022-10-03 23:55 ` sdf
2022-10-04  9:29   ` Jesper Dangaard Brouer
2022-10-04 18:26     ` Stanislav Fomichev
2022-10-05  0:25       ` Martin KaFai Lau
2022-10-05  0:59         ` Jakub Kicinski
2022-10-05  1:02           ` Stanislav Fomichev
2022-10-05  1:24             ` Jakub Kicinski
2022-10-05  2:15               ` Stanislav Fomichev
2022-10-05 19:26                 ` Martin KaFai Lau
2022-10-06  9:14                   ` Magnus Karlsson
2022-10-06 15:29                     ` Jesper Dangaard Brouer
2022-10-11  6:29                       ` Martin KaFai Lau [this message]
2022-10-11 11:57                         ` Jesper Dangaard Brouer
2022-10-05 10:06             ` Toke Høiland-Jørgensen
2022-10-05 18:47               ` sdf
2022-10-06  8:19                 ` Maryam Tahhan
2022-10-06 17:22                   ` sdf
2022-10-05 14:19             ` Jesper Dangaard Brouer
2022-10-06 14:59               ` Jakub Kicinski
2022-10-05 13:43         ` Jesper Dangaard Brouer
2022-10-05 16:29       ` Jesper Dangaard Brouer
2022-10-05 18:43         ` sdf
2022-10-06 17:47           ` Jesper Dangaard Brouer
2022-10-07 15:05             ` David Ahern
2022-10-05 13:14     ` Burakov, Anatoly

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.xdp-project.net/postorius/lists/xdp-hints.xdp-project.net/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c8a712d8-dc97-8df5-6421-a5ccb1357b67@linux.dev \
    --to=martin.lau@linux.dev \
    --cc=alexei.starovoitov@gmail.com \
    --cc=andrii.nakryiko@gmail.com \
    --cc=bjorn@kernel.org \
    --cc=borkmann@iogearbox.net \
    --cc=bpf@vger.kernel.org \
    --cc=brouer@redhat.com \
    --cc=dave@dtucker.co.uk \
    --cc=jbrouer@redhat.com \
    --cc=kuba@kernel.org \
    --cc=larysa.zaremba@intel.com \
    --cc=lorenzo@kernel.org \
    --cc=magnus.karlsson@gmail.com \
    --cc=magnus.karlsson@intel.com \
    --cc=memxor@gmail.com \
    --cc=mtahhan@redhat.com \
    --cc=netdev@vger.kernel.org \
    --cc=sdf@google.com \
    --cc=xdp-hints@xdp-project.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox