From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) by mail.toke.dk (Postfix) with ESMTPS id A91B1A440E3 for ; Wed, 6 Dec 2023 11:07:12 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=PgNgAMWU Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-58ddc32f43fso914432eaf.1 for ; Wed, 06 Dec 2023 02:07:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701857228; x=1702462028; darn=xdp-project.net; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=W/vlq7pt9AeGEnRub01NvFpgsDNby2jk6O8E2xtdj9s=; b=PgNgAMWU+0q5P6bxWPwhD7jsFj9zoX14HnS76OPZXR2u+N+z6Gg9YG94TTRfIOPz0o nw0RjKPFVbA6+LHzPJlvkNCLPmgff3/+yhutS/upAFEs57HDVb+4mvmDjgrcEuBjAE4a 0rFjPU0AiOCYHV5LBCxBYqlOATQFLgZWlw+FWF/ILZ9o0ATa3owwEY/VHnWExmvG4OId AJ9lTXNEsvx+U7yfYbLtDN4J9jLMO5Pv4DXGq55+o2dzrq0li8EUgTKRKz+cTR0ji16J GZR35wiu2u9apBrknEwR4Dh1McfRwZJ6QL0ygVf0Ej0to8biuizifskQefRRe6HQzBaP Sihg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701857228; x=1702462028; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W/vlq7pt9AeGEnRub01NvFpgsDNby2jk6O8E2xtdj9s=; b=F42N9TJGMJh47qiB1hSxaJwwkbuNgVwVx0It9jy/VhTSa2NaezeaG1FeKGHkYM/0+z E7smuAktvrg24xOaVEF+9gCT+T+iV4ng8xcenUxiLwru9V0MKXsPtkyrZ1yk8LjDedEz n7MWVTnzO7xv8CLVsDAWw2BcQAkxn97HUogShCsjgDUY1jO3ap96SAD1CoGzasiOwWoS Dh1U1NtZI+yrO7PnRGUFHA7fC9uu8O8+lRdf/uDyB2LzZVnxjdMw9TfHFskpOFr3rmEk dKvxSUBgPlGffbOndKn/g3eesclDM/nM7PcOpoGP5VChX5rpvKiWsMVkbkBXXMi/KAYH Xe3A== X-Gm-Message-State: AOJu0Yz7auIsrrBFMYNL1Xp9qvOs1LZN4x7kGy9DpyRxMHEGIsbn8B3Y 0lZ58Od9CXygMjxQpwVnGvFefTQF9XnZJvwmDgA= X-Google-Smtp-Source: AGHT+IFM6+SkktBbli78RNGNd8KWWCUqf7S2N+h7frD1ZfTKF9ggdkQwqOjS+jWDA8naLrSe5Bcn1Lp1etLJdeyYiUI= X-Received: by 2002:a05:6830:39de:b0:6d9:b049:7e9b with SMTP id bt30-20020a05683039de00b006d9b0497e9bmr1160555otb.3.1701857228489; Wed, 06 Dec 2023 02:07:08 -0800 (PST) MIME-Version: 1.0 References: <20231203165129.1740512-1-yoong.siang.song@intel.com> <20231203165129.1740512-3-yoong.siang.song@intel.com> <43b01013-e78b-417e-b169-91909c7309b1@kernel.org> <656de830e8d70_2e983e294ca@willemb.c.googlers.com.notmuch> <5a0faf8cc9ec3ab0d5082c66b909c582c8f1eae6.camel@siemens.com> <656f66023f7bd_3dd6422942a@willemb.c.googlers.com.notmuch> In-Reply-To: From: Magnus Karlsson Date: Wed, 6 Dec 2023 11:06:57 +0100 Message-ID: To: Stanislav Fomichev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 6SBFGOY6EFVEAGAFPUGLNNDPZTV6VBWZ X-Message-ID-Hash: 6SBFGOY6EFVEAGAFPUGLNNDPZTV6VBWZ X-MailFrom: magnus.karlsson@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Willem de Bruijn , Florian Bezdeka , yoong.siang.song@intel.com, Jesper Dangaard Brouer , davem@davemloft.net, Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Bjorn Topel , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Lorenzo Bianconi , Tariq Toukan , Willem de Bruijn , Maxime Coquelin , Andrii Nakryiko , Mykola Lysenko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Hao Luo , Jiri Olsa , Shuah Khan , Alexandre Torgue , Jose Abreu , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-doc@vger.kernel.org" , "bpf@vger.kernel.org" , "xdp-hints@xdp-project.net" , "linux-stm32@st-md-mailman.stormreply.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kselftest@vger.kernel.org" X-Mailman-Version: 3.3.8 Precedence: list Subject: [xdp-hints] Re: [PATCH bpf-next v3 2/3] net: stmmac: add Launch Time support to XDP ZC List-Id: XDP hardware hints design discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Tue, 5 Dec 2023 at 20:39, Stanislav Fomichev wrote: > > On 12/05, Willem de Bruijn wrote: > > Stanislav Fomichev wrote: > > > On Tue, Dec 5, 2023 at 7:34=E2=80=AFAM Florian Bezdeka > > > wrote: > > > > > > > > On Tue, 2023-12-05 at 15:25 +0000, Song, Yoong Siang wrote: > > > > > On Monday, December 4, 2023 10:55 PM, Willem de Bruijn wrote: > > > > > > Jesper Dangaard Brouer wrote: > > > > > > > > > > > > > > > > > > > > > On 12/3/23 17:51, Song Yoong Siang wrote: > > > > > > > > This patch enables Launch Time (Time-Based Scheduling) supp= ort to XDP zero > > > > > > > > copy via XDP Tx metadata framework. > > > > > > > > > > > > > > > > Signed-off-by: Song Yoong Siang > > > > > > > > --- > > > > > > > > drivers/net/ethernet/stmicro/stmmac/stmmac.h | 2 ++ > > > > > > > > > > > > > > As requested before, I think we need to see another driver im= plementing > > > > > > > this. > > > > > > > > > > > > > > I propose driver igc and chip i225. > > > > > > > > > > Sure. I will include igc patches in next version. > > > > > > > > > > > > > > > > > > > The interesting thing for me is to see how the LaunchTime max= 1 second > > > > > > > into the future[1] is handled code wise. One suggestion is to= add a > > > > > > > section to Documentation/networking/xsk-tx-metadata.rst per d= river that > > > > > > > mentions/documents these different hardware limitations. It = is natural > > > > > > > that different types of hardware have limitations. This is a= close-to > > > > > > > hardware-level abstraction/API, and IMHO as long as we docume= nt the > > > > > > > limitations we can expose this API without too many limitatio= ns for more > > > > > > > capable hardware. > > > > > > > > > > Sure. I will try to add hardware limitations in documentation. > > > > > > > > > > > > > > > > > I would assume that the kfunc will fail when a value is passed = that > > > > > > cannot be programmed. > > > > > > > > > > > > > > > > In current design, the xsk_tx_metadata_request() dint got return = value. > > > > > So user won't know if their request is fail. > > > > > It is complex to inform user which request is failing. > > > > > Therefore, IMHO, it is good that we let driver handle the error s= ilently. > > > > > > > > > > > > > If the programmed value is invalid, the packet will be "dropped" / = will > > > > never make it to the wire, right? > > > > Programmable behavior is to either drop or cap to some boundary > > value, such as the farthest programmable time in the future: the > > horizon. In fq: > > > > /* Check if packet timestamp is too far in the future. = */ > > if (fq_packet_beyond_horizon(skb, q, now)) { > > if (q->horizon_drop) { > > q->stat_horizon_drops++; > > return qdisc_drop(skb, sch, to_= free); > > } > > q->stat_horizon_caps++; > > skb->tstamp =3D now + q->horizon; > > } > > fq_skb_cb(skb)->time_to_send =3D skb->tstamp; > > > > Drop is the more obviously correct mode. > > > > Programming with a clock source that the driver does not support will > > then be a persistent failure. > > > > Preferably, this driver capability can be queried beforehand (rather > > than only through reading error counters afterwards). > > > > Perhaps it should not be a driver task to convert from possibly > > multiple clock sources to the device native clock. Right now, we do > > use per-device timecounters for this, implemented in the driver. > > > > As for which clocks are relevant. For PTP, I suppose the device PHC, > > converted to nsec. For pacing offload, TCP uses CLOCK_MONOTONIC. > > Do we need to expose some generic netdev netlink apis to query/adjust > nic clock sources (or maybe there is something existing already)? > Then the userspace can be responsible for syncing/converting the > timestamps to the internal nic clocks. +1 to trying to avoid doing > this in the drivers. > > > > > That is clearly a situation that the user should be informed about.= For > > > > RT systems this normally means that something is really wrong regar= ding > > > > timing / cycle overflow. Such systems have to react on that situati= on. > > > > > > In general, af_xdp is a bit lacking in this 'notify the user that the= y > > > somehow messed up' area :-( > > > For example, pushing a tx descriptor with a wrong addr/len in zc mode > > > will not give any visible signal back (besides driver potentially > > > spilling something into dmesg as it was in the mlx case). > > > We can probably start with having some counters for these events? > > > > This is because the AF_XDP completion queue descriptor format is only > > a u64 address? > > Yeah. XDP_COPY mode has the descriptor validation which is exported via > recvmsg errno, but zerocopy path seems to be too deep in the stack > to report something back. And there is no place, as you mention, > in the completion ring to report the status. > > > Could error conditions be reported on tx completion in the metadata, > > using xsk_tx_metadata_complete? > > That would be one way to do it, yes. But then the error reporting depends > on the metadata opt-in. Having a separate ring to export the errors, > or having a v2 tx-completions layout with extra 'status' field would also > work. There are error counters for the non-metadata and offloading cases above that can be retrieved with the XDP_STATISTICS getsockopt(). From if_xdp.h: struct xdp_statistics { __u64 rx_dropped; /* Dropped for other reasons */ __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */ __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */ __u64 rx_ring_full; /* Dropped due to rx ring being full */ __u64 rx_fill_ring_empty_descs; /* Failed to retrieve item from fill ring */ __u64 tx_ring_empty_descs; /* Failed to retrieve item from tx ring = */ }; Albeit, these are aggregate statistics and do not say anything about which packet that caused it. Works well for things that are programming bugs that should not occur (such as rx_invalid_descs and tx_invalid_descs) and requires the programmer to debug and fix his or her program, but it does not work for requests that might fail even though the program is correct and need to be handled on a packet by packet basis. So something needs to be added for that as you both say. Would prefer if we could avoid a v2 completion descriptor format or another ring that needs to be checked all the time, so if we could live with providing the error status in the metadata field of the packet at completion time, that would be good. Though having the error status in the completion ring would be faster as that cache line is hot, while the metadata section of the packet is likely not at completion time. So that speaks for a v2 completion ring format. Just thinking out loud here. > But this seems like something that should be handled separately? Because > we'd have to teach all existing zc drivers to report those errors back > instead of dropping these descriptors.. >