From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from flow7-smtp.messagingengine.com (flow7-smtp.messagingengine.com [103.168.172.142]) by mail.toke.dk (Postfix) with ESMTPS id 86A30A7EE42 for ; Thu, 08 Aug 2024 22:53:30 +0200 (CEST) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key; unprotected) header.d=dxuuu.xyz header.i=@dxuuu.xyz header.a=rsa-sha256 header.s=fm2 header.b=jNfr2AXX; dkim=pass (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=P0NxuDLT Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailflow.nyi.internal (Postfix) with ESMTP id B35F5200B5F; Thu, 8 Aug 2024 16:53:29 -0400 (EDT) Received: from wimap24 ([10.202.2.84]) by compute3.internal (MEProxy); Thu, 08 Aug 2024 16:53:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dxuuu.xyz; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1723150409; x=1723157609; bh=a35vUxoQ9wKa+F4xH4FKyHhirnOC7Y9QMGL7QbeLZUU=; b= jNfr2AXXgsUzk0vESnqzmX6t6lN8oQstoMJbqcSGVlpnm414NjLEF/wsvzVYZj8Z s+yRdEasn4RybuZipbDE7uw0OQrlcfisuGjHUn0r9Z7YKylAOHQ5HrCSckeZeYar OalvM5T89IbA7yRt/9nxNw83oNwf0cShREjRY8DjY8VuBLpoXtR7wTnGrxqeCkDK zyilf47NcEQ1j7Sfev/Vxm5AZ5Suja+EuSVg4NUuo2ongtvti6J/vaExO076ulF1 qHRaPu/FaiHG3LBfS7kL/OjWwFKPC9+wfPfoKCPvNUjOdIu6Z91w1epbMSnypOBE /T+A8pNg0SVLWLLEseqPlQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1723150409; x= 1723157609; bh=a35vUxoQ9wKa+F4xH4FKyHhirnOC7Y9QMGL7QbeLZUU=; b=P 0NxuDLT6Z4R82PXx5kTM6VFXWhCILEiG8wfCjL5X325HRKBAy+ZGUGHX8eVGumFi rTk5m8RQ/PjS9lxU+lgkFFsWWg5DmIzxIUO4clFpp4X03TM3JqcZmrcpnyBpshYk oHTKZCwJgGyi0qGlUDETZYBemG0L/06kOeBp4jxxb6aRmOOfJFHGDe2UMjlYJEFF lwQL0Tdw/rrfALw+8/8L94jBn+xBnK4YxipR+wQ/wNEfhwCuQ9bL8gSki8rB9R/I 1SZc0z1oBt7FoHYuL+qVumvZkqL0aXeXsH/fmebYhyNztdCfOn7FbBaS6f+FCXTP WAldrQKEOgcVsxNIPieww== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledvgdduheehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnegfrhhlucfvnfffucdlfeehmdenucfjughrpefoggffhffvvefk jghfufgtgfesthhqredtredtjeenucfhrhhomhepfdffrghnihgvlhcuighufdcuoegugi husegugihuuhhurdighiiiqeenucggtffrrghtthgvrhhnpeeikeduveeiffekveffieeh hfdtffdtveetjeefieevveffveevudetkeffffelleenucffohhmrghinhepghhithhhuh gsrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhho mhepugiguhesugiguhhuuhdrgiihiidpnhgspghrtghpthhtohepvdejpdhmohguvgepsh hmthhpohhuthdprhgtphhtthhopegurghvvghmsegurghvvghmlhhofhhtrdhnvghtpdhr tghpthhtohepjhhohhhnrdhfrghsthgrsggvnhgusehgmhgrihhlrdgtohhmpdhrtghpth htohepjhhonhgrthhhrghnrdhlvghmohhnsehgmhgrihhlrdgtohhmpdhrtghpthhtohep vgguuhhmrgiivghtsehgohhoghhlvgdrtghomhdprhgtphhtthhopeifihhllhgvmhgsse hgohhoghhlvgdrtghomhdprhgtphhtthhopegrlhgvkhhsrghnuggvrhdrlhhosggrkhhi nhesihhnthgvlhdrtghomhdprhgtphhtthhopegrlhgvgigrnhgurhdrlhhosggrkhhinh esihhnthgvlhdrtghomhdprhgtphhtthhopehjvghsshgvrdgsrhgrnhguvggsuhhrghes ihhnthgvlhdrtghomhdprhgtphhtthhopehlrghrhihsrgdriigrrhgvmhgsrgesihhnth gvlhdrtghomh X-ME-Proxy: Feedback-ID: i6a694271:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 7AEB625E0064; Thu, 8 Aug 2024 16:53:29 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface MIME-Version: 1.0 Date: Thu, 08 Aug 2024 16:52:51 -0400 From: "Daniel Xu" To: "Alexander Lobakin" , "Lorenzo Bianconi" Message-Id: <308fd4f1-83a9-4b74-a482-216c8211a028@app.fastmail.com> In-Reply-To: <54aab7ec-80e9-44fd-8249-fe0cabda0393@intel.com> References: <20220628194812.1453059-1-alexandr.lobakin@intel.com> <20220628194812.1453059-33-alexandr.lobakin@intel.com> <54aab7ec-80e9-44fd-8249-fe0cabda0393@intel.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 32YRTKDAHJJ2OCEDAB2WCRXWEUXGV5Q6 X-Message-ID-Hash: 32YRTKDAHJJ2OCEDAB2WCRXWEUXGV5Q6 X-MailFrom: dxu@dxuuu.xyz X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Alexander Lobakin , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Larysa Zaremba , Michal Swiatkowski , Jesper Dangaard Brouer , =?UTF-8?Q?Bj=C3=B6rn_T=C3=B6pel?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , "toke@redhat.com" , Lorenzo Bianconi , David Miller , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jesse Brandeburg , John Fastabend , Yajun Deng , Willem de Bruijn , "bpf@vger.kernel.org" , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, xdp-hints@xdp-project.net X-Mailman-Version: 3.3.9 Precedence: list Subject: [xdp-hints] Re: [PATCH RFC bpf-next 32/52] bpf, cpumap: switch to GRO from netif_receive_skb_list() List-Id: XDP hardware hints design discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, On Thu, Aug 8, 2024, at 7:57 AM, Alexander Lobakin wrote: > From: Lorenzo Bianconi > Date: Thu, 8 Aug 2024 06:54:06 +0200 > >>> Hi Alexander, >>> >>> On Tue, Jun 28, 2022, at 12:47 PM, Alexander Lobakin wrote: >>>> cpumap has its own BH context based on kthread. It has a sane batch >>>> size of 8 frames per one cycle. >>>> GRO can be used on its own, adjust cpumap calls to the >>>> upper stack to use GRO API instead of netif_receive_skb_list() which >>>> processes skbs by batches, but doesn't involve GRO layer at all. >>>> It is most beneficial when a NIC which frame come from is XDP >>>> generic metadata-enabled, but in plenty of tests GRO performs better >>>> than listed receiving even given that it has to calculate full frame >>>> checksums on CPU. >>>> As GRO passes the skbs to the upper stack in the batches of >>>> @gro_normal_batch, i.e. 8 by default, and @skb->dev point to the >>>> device where the frame comes from, it is enough to disable GRO >>>> netdev feature on it to completely restore the original behaviour: >>>> untouched frames will be being bulked and passed to the upper stack >>>> by 8, as it was with netif_receive_skb_list(). >>>> >>>> Signed-off-by: Alexander Lobakin >>>> --- >>>> kernel/bpf/cpumap.c | 43 ++++++++++++++++++++++++++++++++++++++---= -- >>>> 1 file changed, 38 insertions(+), 5 deletions(-) >>>> >>> >>> AFAICT the cpumap + GRO is a good standalone improvement. I think >>> cpumap is still missing this. > > The only concern for having GRO in cpumap without metadata from the NIC > descriptor was that when the checksum status is missing, GRO calculates > the checksum on CPU, which is not really fast. > But I remember sometimes GRO was faster despite that. Good to know, thanks. IIUC some kind of XDP hint support landed already? My use case could also use HW RSS hash to avoid a rehash in XDP prog. And HW RX timestamp to not break SO_TIMESTAMPING. These two are on one of my TODO lists. But I can=E2=80=99t get to them for at least a few weeks. So free to take it if you=E2=80=99d like. > >>> >>> I have a production use case for this now. We want to do some intell= igent >>> RX steering and I think GRO would help over list-ified receive in so= me cases. >>> We would prefer steer in HW (and thus get existing GRO support) but = not all >>> our NICs support it. So we need a software fallback. >>> >>> Are you still interested in merging the cpumap + GRO patches? > > For sure I can revive this part. I was planning to get back to this > branch and pick patches which were not related to XDP hints and send > them separately. > >>=20 >> Hi Daniel and Alex, >>=20 >> Recently I worked on a PoC to add GRO support to cpumap codebase: >> - https://github.com/LorenzoBianconi/bpf-next/commit/a4b8264d5000ecf0= 16da5a2dd9ac302deaf38b3e >> Here I added GRO support to cpumap through gro-cells. >> - https://github.com/LorenzoBianconi/bpf-next/commit/da6cb32a4674aa72= 401c7414c9a8a0775ef41a55 >> Here I added GRO support to cpumap trough napi-threaded APIs (with = a some >> changes to them). > > Hmm, when I was testing it, adding a whole NAPI to cpumap was sorta > overkill, that's why I separated GRO structure from &napi_struct. > > Let me maybe find some free time, I would then test all 3 solutions > (mine, gro_cells, threaded NAPI) and pick/send the best? Sounds good. Would be good to compare results. [=E2=80=A6] Thanks, Daniel