[xdp-hints] XDP-hints presentation at LLC2022

[xdp-hints] XDP-hints presentation at LLC2022

[Jesper Dangaard Brouer]

Hi XDP-hints community,

Gave this talk[1] on Friday (13 May) "XDP hints via BPF" at Lund Linux
Con.  In the talk we are proposing several *possible* solutions, with
"pros" and "cons".  We are open to new ideas, proposals and critique.

I will play with implementing some of these ideas and to see if they
pan-out.  I want to acknowledge that Alexander Lobakin (+ co-authors)
also have started coding on solutions here[3].  Lets see if we can
collaborate and find a solution we can all agree on.

--Jesper


[1] Slide PDF: 
https://github.com/xdp-project/xdp-project/blob/master/conference/LLC2022/xdp_hints_hw_metadata-final.pdf

[2] Slide source: 
https://github.com/xdp-project/xdp-project/tree/master/conference/LLC2022

[3] https://github.com/alobakin/linux/commits/xdp_hints

[xdp-hints] Re: XDP-hints presentation at LLC2022

[Alexander Lobakin]

From: Jesper Dangaard Brouer <jbrouer@redhat.com>
Date: Mon, 16 May 2022 15:58:06 +0200

> Hi XDP-hints community,

Hey!

> 
> Gave this talk[1] on Friday (13 May) "XDP hints via BPF" at Lund Linux
> Con.  In the talk we are proposing several *possible* solutions, with
> "pros" and "cons".  We are open to new ideas, proposals and critique.
> 
> I will play with implementing some of these ideas and to see if they
> pan-out.  I want to acknowledge that Alexander Lobakin (+ co-authors)
> also have started coding on solutions here[3].  Lets see if we can
> collaborate and find a solution we can all agree on.

Nice catch about "it makes it possible to make NIC driver L2
only" -- it's not present in the current code, but we're thinking
of this for some time already :)
Re collaborate -- I'm expecting the first RFC to hit internal review
in 2-3 weeks, hope it will get posted to LKML soon after that. GH
tree should be finalized this week, so you could start playing.

> 
> --Jesper
> 
> 
> [1] Slide PDF: 
> https://github.com/xdp-project/xdp-project/blob/master/conference/LLC2022/xdp_hints_hw_metadata-final.pdf
> 
> [2] Slide source: 
> https://github.com/xdp-project/xdp-project/tree/master/conference/LLC2022
> 
> [3] https://github.com/alobakin/linux/commits/xdp_hints

Thanks,
Al

[xdp-hints] Re: XDP-hints presentation at LLC2022

[Bezdeka, Florian]

Hi all,

On Mon, 2022-05-16 at 16:41 +0200, Alexander Lobakin wrote:
> From: Jesper Dangaard Brouer <jbrouer@redhat.com>
> Date: Mon, 16 May 2022 15:58:06 +0200
> 
> > Hi XDP-hints community,
> 
> Hey!
> 
> > 
> > Gave this talk[1] on Friday (13 May) "XDP hints via BPF" at Lund Linux
> > Con.  In the talk we are proposing several *possible* solutions, with
> > "pros" and "cons".  We are open to new ideas, proposals and critique.
> > 
> > I will play with implementing some of these ideas and to see if they
> > pan-out.  I want to acknowledge that Alexander Lobakin (+ co-authors)
> > also have started coding on solutions here[3].  Lets see if we can
> > collaborate and find a solution we can all agree on.
> 
> Nice catch about "it makes it possible to make NIC driver L2
> only" -- it's not present in the current code, but we're thinking
> of this for some time already :)
> Re collaborate -- I'm expecting the first RFC to hit internal review
> in 2-3 weeks, hope it will get posted to LKML soon after that. GH
> tree should be finalized this week, so you could start playing.

Is there something already available that would allow us to start
playing with xdp hints?

The main focus for us would be forwarding meta information like TX/RX
hardware timestamps from/into the AF_XDP application. The slides
mention that already, so it seems to be "in scope".

In addition I would like to learn which drivers you are primarily
targeting. Especially if drivers for Intel i210 (igb) and i225 (igc)
cards are on your radar.

Best regards,
Florian

> 
> > 
> > --Jesper
> > 
> > 
> > [1] Slide PDF: 
> > https://github.com/xdp-project/xdp-project/blob/master/conference/LLC2022/xdp_hints_hw_metadata-final.pdf
> > 
> > [2] Slide source: 
> > https://github.com/xdp-project/xdp-project/tree/master/conference/LLC2022
> > 
> > [3] https://github.com/alobakin/linux/commits/xdp_hints
> 
> Thanks,
> Al

[xdp-hints] Re: XDP-hints presentation at LLC2022

[Jesper Dangaard Brouer]

On 14/06/2022 12.40, Bezdeka, Florian wrote:
> 
> On Mon, 2022-05-16 at 16:41 +0200, Alexander Lobakin wrote:
>> From: Jesper Dangaard Brouer <jbrouer@redhat.com>
>> Date: Mon, 16 May 2022 15:58:06 +0200
>>
>>> Hi XDP-hints community,
>>
>> Hey!

Happy to see you Florian joining the XDP-hints community mailing-list.

>>>
>>> Gave this talk[1] on Friday (13 May) "XDP hints via BPF" at Lund Linux
>>> Con.  In the talk we are proposing several *possible* solutions, with
>>> "pros" and "cons".  We are open to new ideas, proposals and critique.
>>>
>>> I will play with implementing some of these ideas and to see if they
>>> pan-out.  I want to acknowledge that Alexander Lobakin (+ co-authors)
>>> also have started coding on solutions here[3].  Lets see if we can
>>> collaborate and find a solution we can all agree on.
>>
>> Nice catch about "it makes it possible to make NIC driver L2
>> only" -- it's not present in the current code, but we're thinking
>> of this for some time already :)
>> Re collaborate -- I'm expecting the first RFC to hit internal review
>> in 2-3 weeks, hope it will get posted to LKML soon after that. GH
>> tree should be finalized this week, so you could start playing.
> 
> Is there something already available that would allow us to start
> playing with xdp hints?

I have some working kernel-side code (that Lorenzo is doing internal
review of), but I've been sick so I've not had time to cleanup and send
on the (bpf-kernel) list. I'll try to get an RFC out tomorrow.

Is your use-case AF_XDP related?

For AF_XDP we/you also need to decode the BTF-format in userspace, to
interpret what layout the kernel or XDP BPF-prog are providing in the
metadata area XDP-hints.

We have a simple BTF-format decoder in this bpf-example[0] what was
developed in collaboration with a large Wind-Turbine producers use-case.
  The example doesn't rely on any kernel changes, as we only depend on
the XDP BPF-prog to provide BTF-info and software RX-timestamps. Thus,
this is a "software only" solution, and we need my mentioned kernel
changes to get access to the hardware (XDP) hints. This is something you
can "start playing with" right now.

  [0] 
https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-interaction


> The main focus for us would be forwarding meta information like TX/RX
> hardware timestamps from/into the AF_XDP application. The slides
> mention that already, so it seems to be "in scope".
> 
> In addition I would like to learn which drivers you are primarily
> targeting. Especially if drivers for Intel i210 (igb) and i225 (igc)
> cards are on your radar.

Yes, Intel i210 (driver igb) and i225 (driver igc) are very much on my
radar and in my testlab.  The mentioned embedded project were
investigating these as part of the hardware evaluation.

We need to land RX-side of XDP-hints *first*, but the longer term plan
is to support TX-side hints.  Specifically I've looked into hardware
LaunchTime features supported by chips i210 and i225.  I've documented
the low-level register differences here[4], which shows we need some
translation layer as driver level (as e.g. bits and granularity differ
between chips for same "perceived" feature).

  [4] 
https://github.com/xdp-project/xdp-project/blob/master/areas/tsn/code01_follow_qdisc_TSN_offload.org


>>>
>>> [1] Slide PDF:
>>> https://github.com/xdp-project/xdp-project/blob/master/conference/LLC2022/xdp_hints_hw_metadata-final.pdf
>>>
>>> [2] Slide source:
>>> https://github.com/xdp-project/xdp-project/tree/master/conference/LLC2022
>>>
>>> [3] https://github.com/alobakin/linux/commits/xdp_hints

--Jesper

[xdp-hints] Re: XDP-hints presentation at LLC2022

[Alexander Lobakin]

From: Jesper Dangaard Brouer <jbrouer@redhat.com>
Date: Wed, 15 Jun 2022 09:45:49 +0200

> On 14/06/2022 12.40, Bezdeka, Florian wrote:
> > 
> > On Mon, 2022-05-16 at 16:41 +0200, Alexander Lobakin wrote:
> >> From: Jesper Dangaard Brouer <jbrouer@redhat.com>
> >> Date: Mon, 16 May 2022 15:58:06 +0200
> >>
> >>> Hi XDP-hints community,
> >>
> >> Hey!
> 
> Happy to see you Florian joining the XDP-hints community mailing-list.
> 
> >>>
> >>> Gave this talk[1] on Friday (13 May) "XDP hints via BPF" at Lund Linux
> >>> Con.  In the talk we are proposing several *possible* solutions, with
> >>> "pros" and "cons".  We are open to new ideas, proposals and critique.
> >>>
> >>> I will play with implementing some of these ideas and to see if they
> >>> pan-out.  I want to acknowledge that Alexander Lobakin (+ co-authors)
> >>> also have started coding on solutions here[3].  Lets see if we can
> >>> collaborate and find a solution we can all agree on.
> >>
> >> Nice catch about "it makes it possible to make NIC driver L2
> >> only" -- it's not present in the current code, but we're thinking
> >> of this for some time already :)
> >> Re collaborate -- I'm expecting the first RFC to hit internal review
> >> in 2-3 weeks, hope it will get posted to LKML soon after that. GH
> >> tree should be finalized this week, so you could start playing.
> > 
> > Is there something already available that would allow us to start
> > playing with xdp hints?

(to Florian)

Yes, we have[0] complete tree with core and Rx parts, with
Generic/skb path supporting Hints as well as the ice driver,
and cpumap consuming Hints.
Please watch out a bit if you use GCC < 10 as it suffers from one
silly bug, which a worked around already locally, but didn't pushed
yet.

> 
> I have some working kernel-side code (that Lorenzo is doing internal
> review of), but I've been sick so I've not had time to cleanup and send
> on the (bpf-kernel) list. I'll try to get an RFC out tomorrow.

(to Jesper)

I'm not sure what's the point of sending it since you're aware of
our tree and that it's almost done, you were watching it and IIRC
we sorta agreed in the past that anything should be sent as follow
ups :) It's currently on internal review as 50+ patches are not
something to review in one hour (it will be split onto several
series series later on, but I want to give a full picture at first).

> 
> Is your use-case AF_XDP related?
> 
> For AF_XDP we/you also need to decode the BTF-format in userspace, to
> interpret what layout the kernel or XDP BPF-prog are providing in the
> metadata area XDP-hints.
> 
> We have a simple BTF-format decoder in this bpf-example[0] what was
> developed in collaboration with a large Wind-Turbine producers use-case.
>   The example doesn't rely on any kernel changes, as we only depend on
> the XDP BPF-prog to provide BTF-info and software RX-timestamps. Thus,
> this is a "software only" solution, and we need my mentioned kernel
> changes to get access to the hardware (XDP) hints. This is something you
> can "start playing with" right now.
> 
>   [0] 
> https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-interaction
> 
> 
> > The main focus for us would be forwarding meta information like TX/RX
> > hardware timestamps from/into the AF_XDP application. The slides
> > mention that already, so it seems to be "in scope".
> > 
> > In addition I would like to learn which drivers you are primarily
> > targeting. Especially if drivers for Intel i210 (igb) and i225 (igc)
> > cards are on your radar.
> 
> Yes, Intel i210 (driver igb) and i225 (driver igc) are very much on my
> radar and in my testlab.  The mentioned embedded project were
> investigating these as part of the hardware evaluation.
> 
> We need to land RX-side of XDP-hints *first*, but the longer term plan
> is to support TX-side hints.  Specifically I've looked into hardware
> LaunchTime features supported by chips i210 and i225.  I've documented
> the low-level register differences here[4], which shows we need some
> translation layer as driver level (as e.g. bits and granularity differ
> between chips for same "perceived" feature).
> 
>   [4] 
> https://github.com/xdp-project/xdp-project/blob/master/areas/tsn/code01_follow_qdisc_TSN_offload.org
> 
> 
> >>>
> >>> [1] Slide PDF:
> >>> https://github.com/xdp-project/xdp-project/blob/master/conference/LLC2022/xdp_hints_hw_metadata-final.pdf
> >>>
> >>> [2] Slide source:
> >>> https://github.com/xdp-project/xdp-project/tree/master/conference/LLC2022
> >>>
> >>> [3] https://github.com/alobakin/linux/commits/xdp_hints
> 
> --Jesper

[0] https://github.com/alobakin/linux/commits/xdp_hints

Thanks,
Olek

[xdp-hints] Re: XDP-hints presentation at LLC2022

[Bezdeka, Florian]

On Wed, 2022-06-15 at 09:45 +0200, Jesper Dangaard Brouer wrote:
> On 14/06/2022 12.40, Bezdeka, Florian wrote:
> > 
> > On Mon, 2022-05-16 at 16:41 +0200, Alexander Lobakin wrote:
> > > From: Jesper Dangaard Brouer <jbrouer@redhat.com>
> > > Date: Mon, 16 May 2022 15:58:06 +0200
> > > 
> > > > Hi XDP-hints community,
> > > 
> > > Hey!
> 
> Happy to see you Florian joining the XDP-hints community mailing-list.
> 
> > > > 
> > > > Gave this talk[1] on Friday (13 May) "XDP hints via BPF" at Lund Linux
> > > > Con.  In the talk we are proposing several *possible* solutions, with
> > > > "pros" and "cons".  We are open to new ideas, proposals and critique.
> > > > 
> > > > I will play with implementing some of these ideas and to see if they
> > > > pan-out.  I want to acknowledge that Alexander Lobakin (+ co-authors)
> > > > also have started coding on solutions here[3].  Lets see if we can
> > > > collaborate and find a solution we can all agree on.
> > > 
> > > Nice catch about "it makes it possible to make NIC driver L2
> > > only" -- it's not present in the current code, but we're thinking
> > > of this for some time already :)
> > > Re collaborate -- I'm expecting the first RFC to hit internal review
> > > in 2-3 weeks, hope it will get posted to LKML soon after that. GH
> > > tree should be finalized this week, so you could start playing.
> > 
> > Is there something already available that would allow us to start
> > playing with xdp hints?
> 
> I have some working kernel-side code (that Lorenzo is doing internal
> review of), but I've been sick so I've not had time to cleanup and send
> on the (bpf-kernel) list. I'll try to get an RFC out tomorrow.
> 
> Is your use-case AF_XDP related?

Yes. Our main use case would be using hardware features like ETF from
AF_XDP applications.

> 
> For AF_XDP we/you also need to decode the BTF-format in userspace, to
> interpret what layout the kernel or XDP BPF-prog are providing in the
> metadata area XDP-hints.

I saw the recently posted RFC series but I did not understand all of
the addressed topics yet. Who defines this layout? 

It would be important for us that the underlying hardware is abstracted
away. So changing the NIC (and with that the driver) should never
require updating the AF_XDP based application.

The referenced slides [1] mention that each driver "can" choose its own
BTF layout and that the driver can change the layout per packet. As the
BTF format is self-describing I assume that different drivers will use
different formats and the application must be able to deal with all of
them. Is that right?

> 
> We have a simple BTF-format decoder in this bpf-example[0] what was
> developed in collaboration with a large Wind-Turbine producers use-case.
>   The example doesn't rely on any kernel changes, as we only depend on
> the XDP BPF-prog to provide BTF-info and software RX-timestamps. Thus,
> this is a "software only" solution, and we need my mentioned kernel
> changes to get access to the hardware (XDP) hints. This is something you
> can "start playing with" right now.

We would also need RX-timestamps - especially for debugging purposes -
so this sounds like a match. Software timestamps should be enough for
some (simulation) setups where special hardware features are not
available.

In addition to the software timestamps I'm quite sure that we will need
the hardware timestamps as well. As such hardware features are normally
device / driver specific, we would need a driver independent API/
memory layout. 

> 
>   [0] 
> https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-interaction
> 
> 
> > The main focus for us would be forwarding meta information like TX/RX
> > hardware timestamps from/into the AF_XDP application. The slides
> > mention that already, so it seems to be "in scope".
> > 
> > In addition I would like to learn which drivers you are primarily
> > targeting. Especially if drivers for Intel i210 (igb) and i225 (igc)
> > cards are on your radar.
> 
> Yes, Intel i210 (driver igb) and i225 (driver igc) are very much on my
> radar and in my testlab.  The mentioned embedded project were
> investigating these as part of the hardware evaluation.

Same here with focus on i225 due to zero copy support.

> 
> We need to land RX-side of XDP-hints *first*, but the longer term plan
> is to support TX-side hints.  Specifically I've looked into hardware
> LaunchTime features supported by chips i210 and i225.  I've documented
> the low-level register differences here[4], which shows we need some
> translation layer as driver level (as e.g. bits and granularity differ
> between chips for same "perceived" feature).
> 
>   [4] 
> https://github.com/xdp-project/xdp-project/blob/master/areas/tsn/code01_follow_qdisc_TSN_offload.org

Looking at this one tells me that we might work on the same topics...

According to your investigations the Qdisc layer makes sure that launch
timestamps are valid. As we do not have such a layer when using AF_XDP,
we would need a feedback channel back to the application that the
supplied timestamp was invalid. You listed that as "design question".
Any ideas so far?

> 
> 
> > > > 
> > > > [1] Slide PDF:
> > > > https://github.com/xdp-project/xdp-project/blob/master/conference/LLC2022/xdp_hints_hw_metadata-final.pdf
> > > > 
> > > > [2] Slide source:
> > > > https://github.com/xdp-project/xdp-project/tree/master/conference/LLC2022
> > > > 
> > > > [3] https://github.com/alobakin/linux/commits/xdp_hints
> 
> --Jesper
>