From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by mail.toke.dk (Postfix) with ESMTPS id E27D38EB293 for ; Mon, 22 Nov 2021 16:31:40 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key) header.d=sipanda-io.20210112.gappssmtp.com header.i=@sipanda-io.20210112.gappssmtp.com header.b=i7oMHx9M Received: by mail-pg1-x531.google.com with SMTP id l190so5061763pge.7 for ; Mon, 22 Nov 2021 07:31:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sipanda-io.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=9koGPaKiKKMNuLL2BrTB4XWNhsdx6WwrtVdEPJ3eseY=; b=i7oMHx9M25xfqP5qPIMZuMJyQtcrr7zMUFhWHevHzOZKJdHU+w2+nYiqyttOLunkkc foh6KIH9wVXVz6osyqaGolKj/ES7zPg4LZeDg6mX1Yc7DcFDU5d87VBoamL8HL+bFGCK +9Ao/AAgvZMkAHV41I783VYfO4s7XRE7GKrwF4YKhxIBO02wAye6XWbGsERQkMgeryAR H/EPzdIj+LkSS9fFIWYXLZuWYdDXgtIoAqJn4wkLnrWxPLOWuE8TcwMGTVNfhXMasLPY o2YEEtcOblr1PBNJ7w8GTtkL/go51euVNHJ48BF2vV3b6VCtzlO70i0tGPeoC6U0LOhG VnOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=9koGPaKiKKMNuLL2BrTB4XWNhsdx6WwrtVdEPJ3eseY=; b=sVCqypqrxb2+4EbHvQh4sd/UjkUo1dxlVWQaBb98UqMsEvNbIwizdJVqMtKDvYXSNy IdzGE97CXmlWFwWtrvBldXIE5pl86zWlqUrRrQ8bYIziWS7tcslAPVkWK10reuqgukla H8qI4H4N4S62uj06SvkIM2HpJRfxOpMNm6MxznT1843axRNBdsJ/BQXi+ahlDsikSh21 GiVW5eubvMFhg5vzRxV7g6OM86DZvxrZWcRykfSuICQY0ZOJJMFl7q66UB7wqPf7pmHc EXyUKtA+VLgRVDMbh6eIdd6QG677vm5jHRBKIgubrBZA5xXOnf9KyvJqPFENuYfLUk9k u94A== X-Gm-Message-State: AOAM531ZUYDPHyjxpoVX9PAj8vwymruFuyxZMwOsSHLbvJ0lXRxNjFrw Yp40/WmxC6sExX2Ze0q5G2UeJo8YYjrpTqlJFz1jdA== X-Google-Smtp-Source: ABdhPJzpHLMMpjkE/zw5StbkTgibVLvtzU5Y18ifLuM6W0hL7mJx6rNsQzPLtQgtB1chWcLNbvN+EbzgtvdapNjo2BU= X-Received: by 2002:a05:6a00:2181:b0:44d:c18d:7af9 with SMTP id h1-20020a056a00218100b0044dc18d7af9mr86861481pfi.16.1637595098333; Mon, 22 Nov 2021 07:31:38 -0800 (PST) MIME-Version: 1.0 References: <875ysqflg1.fsf@toke.dk> <61966ec0722fe_2f3212080@john.notmuch> <871r3cdwng.fsf@toke.dk> <87r1b8cmvf.fsf@toke.dk> In-Reply-To: <87r1b8cmvf.fsf@toke.dk> From: Tom Herbert Date: Mon, 22 Nov 2021 07:31:28 -0800 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 325UA42CBEPIFURORSSQ42IUH62IC6DU X-Message-ID-Hash: 325UA42CBEPIFURORSSQ42IUH62IC6DU X-MailFrom: tom@sipanda.io X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Jamal Hadi Salim , John Fastabend , Jesper Dangaard Brouer , "Karlsson, Magnus" , "Desouza, Ederson" , brouer@redhat.com, "xdp-hints@xdp-project.net" , Eelco Chaudron , Andrii Nakryiko , "Fijalkowski, Maciej" , "Burakov, Anatoly" X-Mailman-Version: 3.3.4 Precedence: list Subject: [xdp-hints] Re: Basic/Dumb question WAS(Re: Re: XDP-hints via local BTF info List-Id: XDP hardware hints design discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Mon, Nov 22, 2021 at 5:59 AM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > > Jamal Hadi Salim writes: > > > And it goes something like this: > > > > Why does the metadata have to go in the DMA descriptors? > > Our experience with the XDP metadata is: you start accessing > > that there is a performance penalty (extra cache miss(es)). > > > > Why is the metadata not encapped as part of the data? We > > dont have MTU issues on receive since that is entirely a local matter; > > meaning the hardware can expand the packet as much as it wants within > > the boundaries of alloced DMA buffer space and XDP and any other > > subsystem (TC for example) can take advantage of the metadata. > > Once the hardware learns how to do that, this is absolutely the > direction we want to go in. But for existing stuff, the metadata is in > the descriptor and needs to be translated somehow... > Toke, By existing stuff I think you mean legacy stuff :-) I tend to agree with Jamal. Descriptor space is exceedingly limited and it's unclear to me that the benefits of getting a few bits of *hints* from the device outweigh the costs to develop XDP hints and perpetually maintain the facility. That is to say it doesn't seem to address the two fundamental limitations of the venerable 50 year old device driver model: 1) the narrow waist of PCIe bus in expressing ancillary information 2) the black box nature of devices that prevent the stack from having any visibility into what it's actually doing (and hence we can only get hints at best and not real operational data for primary protocol processing). I believe CXL and programmable devices are the emerging architectural solution. This will enable split plane acceleration. For instance, if we do this right, a NIC would be able to perform all the stateless processing of a TCP/IP packet such that when the host gets the packet it could immediately jump to the TCP receive processing function ala TXDP-- or even more than that the device could process the whole received TCP datapath and the host just puts the received data on the socket (necessary to solve the TLS offload OOO problem). All this requires a very tight integration between the stack and the device to the extent that the lines are blurred and the boundary of the software stack is effectively pushed into the device (but definitely not TOE!). Tom > -Toke >