From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mail.toke.dk (Postfix) with ESMTPS id 3956A9DE9FD for ; Mon, 16 Jan 2023 14:09:57 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=OjqJYM4x DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673874596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jc4OJeoBzIgA4pINC3NorQcQX/ENG9cQlOJuRsKHw4A=; b=OjqJYM4xLdoxXj+Zp98kLJLdkkr09yFf9K1Tk/mfDTFmc719h1ZbErCGjJtNABO47WkLU1 5SdAZEuH+et9BqymYXf3+TiXxzPn7Ab+Ga2svviRoV3d9K87sMPc65iNlKAGC7v/c+qf5d Slx51sWJToNWyhNiCMkACyNT+z0Dd0w= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-668-DkLohkyBOva3GwKY4_0cmQ-1; Mon, 16 Jan 2023 08:09:53 -0500 X-MC-Unique: DkLohkyBOva3GwKY4_0cmQ-1 Received: by mail-ej1-f71.google.com with SMTP id hp2-20020a1709073e0200b0084d47e3fe82so15701795ejc.8 for ; Mon, 16 Jan 2023 05:09:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:references:to :content-language:subject:cc:user-agent:mime-version:date:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jc4OJeoBzIgA4pINC3NorQcQX/ENG9cQlOJuRsKHw4A=; b=gstPTe74Eki8FEHUBdBT68MIe1Wfzk+HyC0y/TSqNCnPg+YcFFLe9jOkndjLlMLaqe ZD5lyktAJmaDC3soBsRpUOt3ycR0CjF9hRboyk69DPz5VTsBCWd9Q5LGtRvDj/dFngTQ bBVGdR5HE/qLyAFCpJvknkROIHzcg9JR0Q/kwaANofQp3zHNQ9o2pZ+cK/FTlbzAwBJj SzbDFkRGk3wyOxVCICtQn3dTLCsstc6MdNJ09H5TnXwqjCcJH4n+E9g3OlP/MVIyjoKW 55L3qlH8vk3THUFkndNMfHsOwRBQVDEF9RqD9d4hcI9Y383xFWZ03Q6jGrKp09aZfX96 vYdg== X-Gm-Message-State: AFqh2kqMW0NEy2D75m++99VBS1SG0s6U/Y/2cDhs2kvqGWpbUIjQYjqD S65QmpM+Z6CYuB7k/UX25Z3m4MNIkH04NFMb8OCCAvz6H3hFSviLm/cnyicVj2gL4lUBCyfFPQe Okcw2Blaunk8xHBhz4Pkf X-Received: by 2002:a05:6402:d71:b0:498:5cfe:da81 with SMTP id ec49-20020a0564020d7100b004985cfeda81mr33319417edb.3.1673874592443; Mon, 16 Jan 2023 05:09:52 -0800 (PST) X-Google-Smtp-Source: AMrXdXv2K1R7SKjG8Ag4i9+ZhuIVH/paecDngj97ILQwmPZLuLLhNRnEVSY1o3G3I7DvFi/2QD8zZw== X-Received: by 2002:a05:6402:d71:b0:498:5cfe:da81 with SMTP id ec49-20020a0564020d7100b004985cfeda81mr33319375edb.3.1673874592051; Mon, 16 Jan 2023 05:09:52 -0800 (PST) Received: from [192.168.41.200] (83-90-141-187-cable.dk.customer.tdc.net. [83.90.141.187]) by smtp.gmail.com with ESMTPSA id q18-20020a17090676d200b00857c2c29553sm7510010ejn.197.2023.01.16.05.09.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jan 2023 05:09:51 -0800 (PST) From: Jesper Dangaard Brouer X-Google-Original-From: Jesper Dangaard Brouer Message-ID: Date: Mon, 16 Jan 2023 14:09:48 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 To: Stanislav Fomichev , bpf@vger.kernel.org References: <20230112003230.3779451-1-sdf@google.com> <20230112003230.3779451-2-sdf@google.com> In-Reply-To: <20230112003230.3779451-2-sdf@google.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Message-ID-Hash: 7ZUKA7AN72JNGLWIAW3457XQE47OBZHA X-Message-ID-Hash: 7ZUKA7AN72JNGLWIAW3457XQE47OBZHA X-MailFrom: jbrouer@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: brouer@redhat.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, haoluo@google.com, jolsa@kernel.org, David Ahern , Jakub Kicinski , Willem de Bruijn , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org, David Vernet X-Mailman-Version: 3.3.7 Precedence: list Subject: [xdp-hints] Re: [PATCH bpf-next v7 01/17] bpf: Document XDP RX metadata List-Id: XDP hardware hints design discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 12/01/2023 01.32, Stanislav Fomichev wrote: > Document all current use-cases and assumptions. > > Cc: John Fastabend > Cc: David Ahern > Cc: Martin KaFai Lau > Cc: Jakub Kicinski > Cc: Willem de Bruijn > Cc: Jesper Dangaard Brouer > Cc: Anatoly Burakov > Cc: Alexander Lobakin > Cc: Magnus Karlsson > Cc: Maryam Tahhan > Cc: xdp-hints@xdp-project.net > Cc: netdev@vger.kernel.org > Acked-by: David Vernet > Signed-off-by: Stanislav Fomichev > --- > Documentation/networking/index.rst | 1 + > Documentation/networking/xdp-rx-metadata.rst | 108 +++++++++++++++++++ > 2 files changed, 109 insertions(+) > create mode 100644 Documentation/networking/xdp-rx-metadata.rst > > diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst > index 4f2d1f682a18..4ddcae33c336 100644 > --- a/Documentation/networking/index.rst > +++ b/Documentation/networking/index.rst > @@ -120,6 +120,7 @@ Refer to :ref:`netdev-FAQ` for a guide on netdev development process specifics. > xfrm_proc > xfrm_sync > xfrm_sysctl > + xdp-rx-metadata > > .. only:: subproject and html > > diff --git a/Documentation/networking/xdp-rx-metadata.rst b/Documentation/networking/xdp-rx-metadata.rst > new file mode 100644 > index 000000000000..b6c8c77937c4 > --- /dev/null > +++ b/Documentation/networking/xdp-rx-metadata.rst > @@ -0,0 +1,108 @@ > +=============== > +XDP RX Metadata > +=============== > + > +This document describes how an eXpress Data Path (XDP) program can access > +hardware metadata related to a packet using a set of helper functions, > +and how it can pass that metadata on to other consumers. > + > +General Design > +============== > + > +XDP has access to a set of kfuncs to manipulate the metadata in an XDP frame. > +Every device driver that wishes to expose additional packet metadata can > +implement these kfuncs. The set of kfuncs is declared in ``include/net/xdp.h`` > +via ``XDP_METADATA_KFUNC_xxx``. > + > +Currently, the following kfuncs are supported. In the future, as more > +metadata is supported, this set will grow: > + > +.. kernel-doc:: net/core/xdp.c > + :identifiers: bpf_xdp_metadata_rx_timestamp bpf_xdp_metadata_rx_hash > + > +An XDP program can use these kfuncs to read the metadata into stack > +variables for its own consumption. Or, to pass the metadata on to other > +consumers, an XDP program can store it into the metadata area carried > +ahead of the packet. > + > +Not all kfuncs have to be implemented by the device driver; when not > +implemented, the default ones that return ``-EOPNOTSUPP`` will be used. > + > +Within an XDP frame, the metadata layout is as follows:: Below diagram describes XDP buff (xdp_buff), but text says 'XDP frame'. So XDP frame isn't referring literally to xdp_frame, which I find slightly confusing. It is likely because I think too much about the code and the different objects, xdp_frame, xdp_buff, xdp_md (xdp ctx seen be bpf-prog). I tried to grep in the (recent added) bpf/xdp docs to see if there is a definition of a XDP "packet" or "frame". Nothing popped up, except that Documentation/bpf/map_cpumap.rst talks about raw ``xdp_frame`` objects. Perhaps we can improve this doc by calling out xdp_buff here, like: Within an XDP frame, the metadata layout (accessed via ``xdp_buff``) is as follows:: > + > + +----------+-----------------+------+ > + | headroom | custom metadata | data | > + +----------+-----------------+------+ > + ^ ^ > + | | > + xdp_buff->data_meta xdp_buff->data > + > +An XDP program can store individual metadata items into this ``data_meta`` > +area in whichever format it chooses. Later consumers of the metadata > +will have to agree on the format by some out of band contract (like for > +the AF_XDP use case, see below). > + > +AF_XDP > +====== > + > +:doc:`af_xdp` use-case implies that there is a contract between the BPF > +program that redirects XDP frames into the ``AF_XDP`` socket (``XSK``) and > +the final consumer. Thus the BPF program manually allocates a fixed number of > +bytes out of metadata via ``bpf_xdp_adjust_meta`` and calls a subset > +of kfuncs to populate it. The userspace ``XSK`` consumer computes > +``xsk_umem__get_data() - METADATA_SIZE`` to locate that metadata. > +Note, ``xsk_umem__get_data`` is defined in ``libxdp`` and > +``METADATA_SIZE`` is an application-specific constant. The main problem with AF_XDP and metadata is that, the AF_XDP descriptor doesn't contain any info about the length METADATA_SIZE. The text does says this, but in a very convoluted way. I think this challenge should be more clearly spelled out. (p.s. This was something that XDP-hints via BTF have a proposed solution for) > + > +Here is the ``AF_XDP`` consumer layout (note missing ``data_meta`` pointer):: The "note" also hint to this issue. > + > + +----------+-----------------+------+ > + | headroom | custom metadata | data | > + +----------+-----------------+------+ > + ^ > + | > + rx_desc->address > + > +XDP_PASS > +======== > + > +This is the path where the packets processed by the XDP program are passed > +into the kernel. The kernel creates the ``skb`` out of the ``xdp_buff`` > +contents. Currently, every driver has custom kernel code to parse > +the descriptors and populate ``skb`` metadata when doing this ``xdp_buff->skb`` > +conversion, and the XDP metadata is not used by the kernel when building > +``skbs``. However, TC-BPF programs can access the XDP metadata area using > +the ``data_meta`` pointer. > + > +In the future, we'd like to support a case where an XDP program > +can override some of the metadata used for building ``skbs``. Happy this is mentioned as future work. > + > +bpf_redirect_map > +================ > + > +``bpf_redirect_map`` can redirect the frame to a different device. > +Some devices (like virtual ethernet links) support running a second XDP > +program after the redirect. However, the final consumer doesn't have > +access to the original hardware descriptor and can't access any of > +the original metadata. The same applies to XDP programs installed > +into devmaps and cpumaps. > + > +This means that for redirected packets only custom metadata is > +currently supported, which has to be prepared by the initial XDP program > +before redirect. If the frame is eventually passed to the kernel, the > +``skb`` created from such a frame won't have any hardware metadata populated > +in its ``skb``. If such a packet is later redirected into an ``XSK``, > +that will also only have access to the custom metadata. > + Good that this is documented, but I hope we can fix/improve this as future work. > +bpf_tail_call > +============= > + > +Adding programs that access metadata kfuncs to the ``BPF_MAP_TYPE_PROG_ARRAY`` > +is currently not supported. > + > +Example > +======= > + > +See ``tools/testing/selftests/bpf/progs/xdp_metadata.c`` and > +``tools/testing/selftests/bpf/prog_tests/xdp_metadata.c`` for an example of > +BPF program that handles XDP metadata. --Jesper