From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by mail.toke.dk (Postfix) with ESMTPS id 0145A8520E8 for ; Fri, 28 May 2021 16:52:25 +0200 (CEST) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com header.i=@herbertland-com.20150623.gappssmtp.com header.b=StQQBjbN Received: by mail-ej1-x62e.google.com with SMTP id l1so5767208ejb.6 for ; Fri, 28 May 2021 07:52:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=rsLrPNP25GlRv26GVcYNUaSU+AFOtZqN+4ND73+5zd0=; b=StQQBjbNTP2gVIel6xGuW3M6D+hJrzHGOxOfnRzpPXy6WMdtZ5CP5aM700VXQLdHgc /ZbAwvXH8UoZ4o0zRwoHndcYht6twsaWTcXtWHJ7bKy+3Q147aLPjLgm2B6iZeQ0/cbA 8jIX0y84Zlsvyk1ruhfjEQgucF04/LyL3BPIP57zfEuHvyk+mmBvW+5QP5dmwgv4LpV9 Wb0OtATV3RFk4peAJZlKL6f/Mb6o+3okJHBm/Iwoqwt9v1ePPueBb+VkNPsCNKIAtrSk yqrpsfkUKGlQSL5AX+Me9a+pMLl7hG290iHjn5wXP6yGE76oPpiUKqf9tyI5Iw948rTs WNiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=rsLrPNP25GlRv26GVcYNUaSU+AFOtZqN+4ND73+5zd0=; b=a6yQuUU+vtBVBNnxyXkTN+5jFQ7h6vMW1wPe2DfXZUrBN9vBmPj34fcTUbfXa3VOQJ ZCDFQAdNpnu3ZiNub0lSVozRMMN62z3+XdbjfezCo/G3eaFXPZftJ4HytVBWg0IG4MaG o9c/JwO4PvRT5U5Zpox8pYmRcsaiQcZSf1XDQjKsssM1uBNmNTXRFD28qYEtlJXopgcK w8cIWyZAkq+KL/6K5EP6PgrUdX43l2e7Iw3Eywz7Tojx9nZ4SNr9E+oddUutsJSVQ25/ egPa3SYbEU+EebWyD135++q6QBs78j4Gn7p+JTkU3XtC+O5E6lzeb/qq8g7S+e4UVsSS QX6g== X-Gm-Message-State: AOAM532UgGSREHGDcJrlQbC3Nzg/zFbXFVnYx0cUmvUKHJwsKNVvf2Rf 0wyWYchYdZ19yoZ79IiWr0HCfajKvrpPLUtl1vjyJUoOxdw= X-Google-Smtp-Source: ABdhPJyE/yemmcvEGCEskMU5IlG9eqVhBJQdO8U2YNdDgmIzyoYV14Z8Eg2pcBLnpgFZn3Ox30ccYy9E8PeNUI7MJjE= X-Received: by 2002:a17:907:2ccb:: with SMTP id hg11mr2959256ejc.259.1622213539675; Fri, 28 May 2021 07:52:19 -0700 (PDT) MIME-Version: 1.0 From: Tom Herbert Date: Fri, 28 May 2021 07:52:08 -0700 Message-ID: Subject: Challenges of XDP hints to overcome To: xdp-hints@xdp-project.net Content-Type: text/plain; charset="UTF-8" Message-ID-Hash: 2AAM62TS7LEIKYS2URKH77KA3W5CCEMB X-Message-ID-Hash: 2AAM62TS7LEIKYS2URKH77KA3W5CCEMB X-MailFrom: tom@herbertland.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.4 Precedence: list List-Id: XDP hardware hints design discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: I believe there are some major challenges to overcome if XDP hints is to become ubiquitous: 1. The amount of meta information that can be practically conveyed per packet over the PCIe bus is limited. For instance there is no practical way to convey a complete set of metadata for a packet like we get on output from flow dissector with killing PCIe bandwidth. It seems like the best we can do is maybe convey a few items such as L3, L4 offsets. This roadblock is going to be hard to overcome without an architectural change (CXL for instance). 2. The stack needs to be able to make use of the hints. For instance, in Transport XDP, XDP layer could conceptually perform all the stateless processing for a TCP/IP packet and jump into TCP code at the point where connection lookup is done. This is feasible to do in software XDP, presumably some explicit program like in eBPF would need to be offloaded to make it work on hardware 3. Device is a black box to the stack. We really don't want to get into another situation like protocol specific checksum offloads where the device is doing something that may or may not be doing in a way the stack thinks it's appropriate. We need full transparency as to what the device is doing. IMO, a requirement of XDP hints should be that the device MUST be user programmable and the program for parsing and metadata extraction is under control and visible to the stack. Effectively, this extends the stack into the device (Split Plane Acceleration). More specifically, to eliminate any possibility of ambiguity and achieve "write once, run anywhere", we want the *exact* same program to be run by the stack in software and then is offloaded to the hardware-- this is the model of the PANDA Parser. Tom