From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from flow7-smtp.messagingengine.com (flow7-smtp.messagingengine.com [103.168.172.142]) by mail.toke.dk (Postfix) with ESMTPS id 5AB77A7EE23 for ; Thu, 08 Aug 2024 22:44:33 +0200 (CEST) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key; unprotected) header.d=dxuuu.xyz header.i=@dxuuu.xyz header.a=rsa-sha256 header.s=fm2 header.b=kPZYYpC+; dkim=pass (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=n2TbSaWU Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailflow.nyi.internal (Postfix) with ESMTP id 9CA7F200CE1; Thu, 8 Aug 2024 16:44:31 -0400 (EDT) Received: from wimap24 ([10.202.2.84]) by compute3.internal (MEProxy); Thu, 08 Aug 2024 16:44:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dxuuu.xyz; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1723149871; x=1723157071; bh=v+W3a7SXw8EMXoGeWpGy5gfB1VuUlziJvetZYk3mEQA=; b= kPZYYpC+iJ7h6Ov2ysI0xNU0NyCRpPo1i6/bsOJw7R4KKe7U26YrDA9IFaTWPgqM qWWN7j6TnZkhTJKoNqNQVkdW5+OpXQKjNGBs4DnoOwU8Z/tQwDpHNTu75muNN4ci J5o3/7sP0yWfqOChFt+IiX3tpFhYyPqHCV0Jm2ioHXJPPcrMUXYh4/G+qcsvfPq9 ypTM0vsfJybRXZejLvr/TZ77Wy/DWXUqqY8y8K1CqhyZc3rrJ4plja80KSJ/kvSj 98Tq6m3lZ+g6ar72VDabOXdMHk5Cu3xl++toIJ4im4Rc3PdB+iOwJpZ4MNyeyxB3 o8VXqzO6XDF2jbi8w4YPeg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1723149871; x= 1723157071; bh=v+W3a7SXw8EMXoGeWpGy5gfB1VuUlziJvetZYk3mEQA=; b=n 2TbSaWU61Gw1f3Rq7b2gW8ppq2Fwd5ZGhEKU6O38kbm0xoDY5i1k5VUq6apgfkLq c+1id2J7txtfC2mUIB2V5IXfsOKB4rrcmYw5r4G+J31k2OKgHR83czNPVviLjBBY 4iwzUY++fc9hDkgCbKRtN3Shdk4A6durovSZTK5zYXfwymlP8wvW2/Nmzr9HuHZg RAGXVva177d5xLI2VTRVsVg4bPK911Ekj+3hcSlOj97x0zWPt2kP5z5f67HyAlax XUodSmzY8ml9jSoZqRFx1lFpDwYf5EbgxJsOMWJSkgFezK0Nzt6jlbaGPWtksSfT VzIuEaU4Oo0Md+AIMrMOg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledvgdduheefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnegfrhhlucfvnfffucdlfeehmdenucfjughrpefoggffhffvvefk jghfufgtgfesthhqredtredtjeenucfhrhhomhepfdffrghnihgvlhcuighufdcuoegugi husegugihuuhhurdighiiiqeenucggtffrrghtthgvrhhnpeeikeduveeiffekveffieeh hfdtffdtveetjeefieevveffveevudetkeffffelleenucffohhmrghinhepghhithhhuh gsrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhho mhepugiguhesugiguhhuuhdrgiihiidpnhgspghrtghpthhtohepvdeipdhmohguvgepsh hmthhpohhuthdprhgtphhtthhopegurghvvghmsegurghvvghmlhhofhhtrdhnvghtpdhr tghpthhtohepjhhohhhnrdhfrghsthgrsggvnhgusehgmhgrihhlrdgtohhmpdhrtghpth htohepjhhonhgrthhhrghnrdhlvghmohhnsehgmhgrihhlrdgtohhmpdhrtghpthhtohep vgguuhhmrgiivghtsehgohhoghhlvgdrtghomhdprhgtphhtthhopeifihhllhgvmhgsse hgohhoghhlvgdrtghomhdprhgtphhtthhopegrlhgvgigrnhgurhdrlhhosggrkhhinhes ihhnthgvlhdrtghomhdprhgtphhtthhopehjvghsshgvrdgsrhgrnhguvggsuhhrghesih hnthgvlhdrtghomhdprhgtphhtthhopehlrghrhihsrgdriigrrhgvmhgsrgesihhnthgv lhdrtghomhdprhgtphhtthhopehmrggtihgvjhdrfhhijhgrlhhkohifshhkihesihhnth gvlhdrtghomh X-ME-Proxy: Feedback-ID: i6a694271:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 4088825E0064; Thu, 8 Aug 2024 16:44:31 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface MIME-Version: 1.0 Date: Thu, 08 Aug 2024 16:44:08 -0400 From: "Daniel Xu" To: "Lorenzo Bianconi" Message-Id: <7e6c0c0d-886e-4144-a0f4-d0d6f0faa1e6@app.fastmail.com> In-Reply-To: References: <20220628194812.1453059-1-alexandr.lobakin@intel.com> <20220628194812.1453059-33-alexandr.lobakin@intel.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: O76L6V65KEVMZV26NFSQTSE5ACGOD4U4 X-Message-ID-Hash: O76L6V65KEVMZV26NFSQTSE5ACGOD4U4 X-MailFrom: dxu@dxuuu.xyz X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Alexander Lobakin , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Larysa Zaremba , Michal Swiatkowski , Jesper Dangaard Brouer , =?UTF-8?Q?Bj=C3=B6rn_T=C3=B6pel?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , "toke@redhat.com" , Lorenzo Bianconi , David Miller , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jesse Brandeburg , John Fastabend , Yajun Deng , Willem de Bruijn , "bpf@vger.kernel.org" , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, xdp-hints@xdp-project.net X-Mailman-Version: 3.3.9 Precedence: list Subject: [xdp-hints] Re: [PATCH RFC bpf-next 32/52] bpf, cpumap: switch to GRO from netif_receive_skb_list() List-Id: XDP hardware hints design discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Lorenzo, On Thu, Aug 8, 2024, at 12:54 AM, Lorenzo Bianconi wrote: >> Hi Alexander, >>=20 >> On Tue, Jun 28, 2022, at 12:47 PM, Alexander Lobakin wrote: >> > cpumap has its own BH context based on kthread. It has a sane batch >> > size of 8 frames per one cycle. >> > GRO can be used on its own, adjust cpumap calls to the >> > upper stack to use GRO API instead of netif_receive_skb_list() which >> > processes skbs by batches, but doesn't involve GRO layer at all. >> > It is most beneficial when a NIC which frame come from is XDP >> > generic metadata-enabled, but in plenty of tests GRO performs better >> > than listed receiving even given that it has to calculate full frame >> > checksums on CPU. >> > As GRO passes the skbs to the upper stack in the batches of >> > @gro_normal_batch, i.e. 8 by default, and @skb->dev point to the >> > device where the frame comes from, it is enough to disable GRO >> > netdev feature on it to completely restore the original behaviour: >> > untouched frames will be being bulked and passed to the upper stack >> > by 8, as it was with netif_receive_skb_list(). >> > >> > Signed-off-by: Alexander Lobakin >> > --- >> > kernel/bpf/cpumap.c | 43 ++++++++++++++++++++++++++++++++++++++---= -- >> > 1 file changed, 38 insertions(+), 5 deletions(-) >> > >>=20 >> AFAICT the cpumap + GRO is a good standalone improvement. I think >> cpumap is still missing this. >>=20 >> I have a production use case for this now. We want to do some intelli= gent >> RX steering and I think GRO would help over list-ified receive in som= e cases. >> We would prefer steer in HW (and thus get existing GRO support) but n= ot all >> our NICs support it. So we need a software fallback. >>=20 >> Are you still interested in merging the cpumap + GRO patches? > > Hi Daniel and Alex, > > Recently I worked on a PoC to add GRO support to cpumap codebase: > -=20 > https://github.com/LorenzoBianconi/bpf-next/commit/a4b8264d5000ecf016d= a5a2dd9ac302deaf38b3e > Here I added GRO support to cpumap through gro-cells. > -=20 > https://github.com/LorenzoBianconi/bpf-next/commit/da6cb32a4674aa72401= c7414c9a8a0775ef41a55 > Here I added GRO support to cpumap trough napi-threaded APIs (with a=20 > some > changes to them). Cool!=20 > > Please note I have not run any performance tests so far, just verified= it does > not crash (I was planning to resume this work soon). Please let me kno= w if it > works for you. I=E2=80=99ll try to run an A/B test on your two approaches as well as Al= ex=E2=80=99s. I=E2=80=99ve still got some testbeds with production traffic going thru them. Thanks, Daniel