From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mail.toke.dk (Postfix) with ESMTPS id 808A585A96A for ; Thu, 24 Jun 2021 14:23:10 +0200 (CEST) Authentication-Results: mail.toke.dk; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=iMEr+B12 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624537389; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5dZ1hcsgYKL9PXhUuTgw75fJdwkSMgO5X2pmzQJtfQc=; b=iMEr+B12owY2+K3XIyggPWXVvtdXXjZcWUmBicBomRBX9nX9SnhGODvslJu6WLWy2p6flT mUc8Wqmle6NgQkBCJB5yOVqiwOUMkpSr0nQwhuFP/lD5i43MDZfHfTVA8w/YSdREVLvmGT kM5EECsN78KoZadDQBIljgMVjbLlRfc= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-226-jMGZ_3mcPzitCQ5ICELLkA-1; Thu, 24 Jun 2021 08:23:06 -0400 X-MC-Unique: jMGZ_3mcPzitCQ5ICELLkA-1 Received: by mail-ed1-f70.google.com with SMTP id v12-20020aa7dbcc0000b029038fc8e57037so3272776edt.0 for ; Thu, 24 Jun 2021 05:23:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=bk5gTGN46GI98trCYcbMSxKLrEsnWSHlsMzFzO5YGXA=; b=Dxsnljk7xvVylQtfl/lv3Qq3UFP7WtsP0TAfaueI5B5yy60NTS0xXGIrI2dcSnIAii IqIVtTYzQNNIFmJdX9vh33OgslLE5G6vKSAwxpWHXNARf4cLbqVZlMOZ/wTEFWqI91lO dYrLQAL00yp8cBiaiunXF9T2xLY1FiIHojWsNJaUD7CngFLio1lAdWuRZw6dWfSOrMJw /nQjYQlef+0++wDLRNWvZD2pCLDknmXVno9Bqv4PjuEzy+aNaNYgnxAeEOehclol2TJw OM9lttywrJ109OAtRbYk9B+pYNdJmtACT7fQlhk1bVJMWfHFPyLReZAGrZwJ6Fn0hOp9 eORg== X-Gm-Message-State: AOAM533M99ddGxR0fBsdy4k5RGFRw3+AVsQTEyy6aZIRiq2tXxroA6g9 PROS4xXnQZhlJhAkoW2d7PIjAPcBOW7sGSfdV3655epQdtQnhc97bLSv/HEYXG0YlEm9A1k6qP9 bVQjdN+uljaPxke0Ut0Z2 X-Received: by 2002:a17:906:58d4:: with SMTP id e20mr5052975ejs.461.1624537385423; Thu, 24 Jun 2021 05:23:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz08jEssUwwbwFCiDH9FO1gAKK9tAfx+k53Sum6xG28XdcFvOaIUQ/u4XDSgbFIiU+2m/sScA== X-Received: by 2002:a17:906:58d4:: with SMTP id e20mr5052930ejs.461.1624537385017; Thu, 24 Jun 2021 05:23:05 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id w2sm1131378ejn.118.2021.06.24.05.23.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Jun 2021 05:23:04 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 3D8A2180731; Thu, 24 Jun 2021 14:23:02 +0200 (CEST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= To: Michal Swiatkowski Subject: Re: XDP-hints: Howto support multiple BTF types per packet basis? In-Reply-To: References: <60b08442b18d5_1cf8208a0@john-XPS-13-9370.notmuch> <87fsy7gqv7.fsf@toke.dk> <60b0ffb63a21a_1cf82089e@john-XPS-13-9370.notmuch> <20210528180214.3b427837@carbon> <60b12897d2e3f_1cf820896@john-XPS-13-9370.notmuch> <8735u3dv2l.fsf@toke.dk> <60b6cf5b6505e_38d6d208d8@john-XPS-13-9370.notmuch> <20210602091837.65ec197a@kicinski-fedora-PC1C0HJN.hsd1.ca.comcast.net> <874kdqqfnm.fsf@toke.dk> X-Clacks-Overhead: GNU Terry Pratchett Date: Thu, 24 Jun 2021 14:23:02 +0200 Message-ID: <87mtrfmoyh.fsf@toke.dk> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=toke@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 6HMXV3JZFTH5NCJ3LZWJOL5IX6CR5DVS X-Message-ID-Hash: 6HMXV3JZFTH5NCJ3LZWJOL5IX6CR5DVS X-MailFrom: toke@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Jakub Kicinski , John Fastabend , Jesper Dangaard Brouer , Andrii Nakryiko , BPF-dev-list , Magnus Karlsson , William Tu , xdp-hints@xdp-project.net X-Mailman-Version: 3.3.4 Precedence: list List-Id: XDP hardware hints design discussion Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Michal Swiatkowski writes: > On Tue, Jun 22, 2021 at 01:53:33PM +0200, Toke H=C3=B8iland-J=C3=B8rgense= n wrote: >> Michal Swiatkowski writes: >>=20 >> > On Wed, Jun 02, 2021 at 09:18:37AM -0700, Jakub Kicinski wrote: >> >> On Tue, 01 Jun 2021 17:22:51 -0700 John Fastabend wrote: >> >> > > If we do this, the BPF program obviously needs to know which fiel= ds are >> >> > > valid and which are not. AFAICT you're proposing that this should= be >> >> > > done out-of-band (i.e., by the system administrator manually ensu= ring >> >> > > BPF program config fits system config)? I think there are a coupl= e of >> >> > > problems with this: >> >> > >=20 >> >> > > - It requires the system admin to coordinate device config with a= ll of >> >> > > their installed XDP applications. This is error-prone, especial= ly as >> >> > > the number of applications grows (say if different containers h= ave >> >> > > different XDP programs installed on their virtual devices). =20 >> >> >=20 >> >> > A complete "system" will need to be choerent. If I forward into a v= eth >> >> > device the orchestration component needs to ensure program sending >> >> > bits there is using the same format the program installed there exp= ects. >> >> >=20 >> >> > If I tailcall/fentry into another program that program the callee a= nd >> >> > caller need to agree on the metadata protocol. >> >> >=20 >> >> > I don't see any way around this. Someone has to manage the network. >> >>=20 >> >> FWIW I'd like to +1 Toke's concerns. >> >>=20 >> >> In large deployments there won't be a single arbiter. Saying there=20 >> >> is seems to contradict BPF maintainers' previous stand which lead=20 >> >> to addition of bpf_links for XDP. >> >>=20 >> >> In practical terms person rolling out an NTP config change may not=20 >> >> be aware that in some part of the network some BPF program expects >> >> descriptor not to contain time stamps. Besides features may depend=20 >> >> or conflict so the effects of feature changes may not be obvious=20 >> >> across multiple drivers in a heterogeneous environment. >> >>=20 >> >> IMO guarding from obvious mis-configuration provides obvious value. >> > >> > Hi, >> > >> > Thanks for a lot of usefull information about CO-RE. I have read >> > recommended articles, but still don't understand everything, so sorry = if >> > my questions are silly. >> > >> > As introduction, I wrote small XDP example using CO-RE (autogenerated >> > vmlinux.h and getting rid of skeleton etc.) based on runqslower >> > implementation. Offset reallocation of hints works great, I built CO-R= E >> > application, added new field to hints struct, changed struct layout an= d >> > without rebuilding application everything still works fine. Is it wort= h >> > to add XDP sample using CO-RE in kernel or this isn't good place for >> > this kind of sample? >> > >> > First question not stricte related to hints. How to get rid of #define >> > and macro when I am using generated vmlinux.h? For example I wanted to >> > use htons macro and ethtype definition. They are located in headers th= at >> > also contains few struct definition. Because of that I have redefiniti= on >> > error when I am trying to include them (redefinition in vmlinux.h and >> > this included file). What can I do with this besides coping definition= s >> > to bpf code? >>=20 >> One way is to only include the structs you actually need from vmlinux.h. >> You can even prune struct members, since CO-RE works just fine with >> partial struct definitions as long as the member names match. >>=20 >> Jesper has an example on how to handle this here: >> https://github.com/netoptimizer/bpf-examples/blob/ktrace01-CO-RE.public/= headers/vmlinux_local.h >>=20 > > I see, thanks, I will take a look at other examples. > >> > I defined hints struct in driver code, is it right place for that? All >> > vendors will define their own hints struct or the idea is to have one >> > big hints struct with flags informing about availability of each field= s? >> > >> > For me defining it in driver code was easier because I can have used >> > module btf to generate vmlinux.h with hints struct inside. However thi= s >> > break portability if other vendors will have different struct name etc= , >> > am I right? >>=20 >> I would expect the easiest is for drivers to just define their own >> structs and maybe have some infrastructure in the core to let userspace >> discover the right BTF IDs to use for a particular netdev. However, as >> you say it's not going to work if every driver just invents their own >> field names, so we'll need to coordinate somehow. We could do this by >> convention, though, it'll need manual intervention to make sure the >> semantics of identically-named fields match anyway. >>=20 >> Cf the earlier discussion with how many BTF IDs each driver might >> define, I think we *also* need a way to have flags that specify which >> fields of a given BTF ID are currently used; and having some common >> infrastructure for that would be good... >>=20 > > Sounds good.=20 > > Sorry, but I feel that I don't fully understand the idea. Correct me if > I am wrong: > > In building CO-RE application step we can defined big struct with > all possible fields or even empty struct (?) and use > bpf_core_field_exists.=20 > > bpf_core_field_exists will be resolve before loading program by libbpf > code. In normal case libbpf will look for btf with hints name in vmlinux > of running kernel and do offset rewrite and exsistence check. But as the > same hints struct will be define in multiple modules we want to add more > logic to libbpf to discover correct BTF ID based on netdev on which progr= am > will be loaded? I would expect that the program would decide ahead-of-time which BTF IDs it supports, by something like including the relevant structs from vmlinux.h. And then we need the BTF ID encoded into the packet metadata as well, so that it is possible to check at run-time which driver the packet came from (since a packet can be redirected, so you may end up having to deal with multiple formats in the same XDP program). Which would allow you to write code like: if (ctx->has_driver_meta) { /* this should be at a well-known position, like first (or last) in meta = area */ __u32 *meta_btf_id =3D ctx->data_meta; =20 if (*meta_btf_id =3D=3D BTF_ID_MLX5) { struct meta_mlx5 *meta =3D ctx->data_meta; /* do something with meta */ } else if (meta_btf_id =3D=3D BTF_ID_I40E) { struct meta_i40e *meta =3D ctx->data_meta; /* do something with meta */ } /* etc */ } and libbpf could do relocations based on the different meta structs, even removing the code for the ones that don't exist on the running kernel. -Toke