Add XDP hardware offload support for certain Mizar functions with Net…

…ronome NIC (#658)

* easy shell for ustc

* load transit_xdp1 with offload mode

* update offloaded endpoint_map

* add trn_transit_xdp1 with 3 layers function

* replace xdp1 with xdp1_3layers and add related offload_map fd

* update easy shell for ustc

* update offloaded network_map

* test for performance

* test for vpc and modify num of ep&net

* fix for merging

* load offload XDP with double check

* fix for ustc

* fix for ustc

* fix

* reset

* remove unnecessary file

* Remove easy shell for USTC

* Remove easy shell for USTC

* fix for merging

* Fix for PR

* Fix for USTC

* Fix update_vpc/net/ep unittest

* Fix the code format and add some err handlers.

* Remove offload CLI and Restrict offload NIC yaml

* Remove offload CLI

* Update original unittest in generic mode

* Add unittest in offload mode

* Fix log description err.

* Fix the code style and format.

* Remove changes of get_default_itf

* Update for code style and format

* Add rc to the log

* Fix some errors

* Fix offload codes

* Fix offload codes

* Remove blank line

* Update  docs

* Update docs

* Update

* Update

* Update

docs/design/features/offload_XDP/design_and_limitations.md

@@ -0,0 +1,30 @@
+<!--
+SPDX-License-Identifier: MIT
+Copyright (c) 2021 The Authors.
+
+Authors:
+Peng Yang               (@yangpenger)      
+
+-->
+Netronome SmartNIC has following limitations:
+
+- Only supports maps of type BPF_MAP_TYPE_ARRAY/BPF_MAP_TYPE_HASH.
+- Constraints on the size of the maps:
+  - Sum of map entries on the NIC should be less than 3,072,000;
+  - A maximum limit of 64 bytes per entry.
+- Constraints on the size of the program: the maximum number of instructions on the SmartNIC is only 2800.
+- Does not support XDP_REDIRECT.
+
+
+
+Considering the limitations above, bouncers and dividers (forwarding function) are offloaded to the NIC. 
+
+
+
+The first packet between two endpoints will pass through the bouncer/divider. This forwarding function will be implemented on the NIC. On the contrary, for the XDP program on the endpoint host's kernel, every packet (not only the first) will pass through it. The main function is to redirect rather than forward, so it is not the offloaded target.
+
+
+
+The offloaded workflow is as follows (The black line represents the logic of the mizar, and the red line represents the partial function offloaded to SmartNIC):
+
+![workflow of offloading XDP](offload_XDP_workflow.png)

etc/deploy/deploy.mizar.dev.yaml

@@ -527,6 +527,8 @@ spec:
           value: 'false'
         - name: FEATUREGATE_BWQOS
           value: 'false'
+        - name: FEATUREGATE_OFFLOAD_XDP
+          value: 'false'
         securityContext:
           privileged: true
       volumes:

mizar/common/common.py

@@ -28,6 +28,7 @@
 import datetime
 import json
 import dateutil.parser
+import yaml
 from kubernetes import watch, client, config
 from kubernetes.client.rest import ApiException
 from ctypes.util import find_library
@@ -434,6 +435,35 @@ def conf_list_has_max_elements(conf, conf_list):
         return True
     return False
 
+def supported_offload_xdp_itf_names():
+    """
+    According to the list of NIC names, corresponding logic interface names are returned.
+    """
+    logical_itf_names = []
+    with open("/var/mizar/supported_xdp_offload_nics.yaml", "r", encoding="utf-8") as f:
+        supported_nic_names = yaml.load(f, Loader=yaml.FullLoader)
+
+    rc, data = run_cmd("lspci -mm | grep 'Ethernet controller'")
+    if rc is not None:
+        logging.info("Failure running \"lspci -mm | grep 'Ethernet controller'\" with rc:" + f'''{rc}''')
+        return logical_itf_names
+
+    eth_crtls = [i for i in data.split('\n') if i]
+    for eth_crtl in eth_crtls:
+        for vendor_name in supported_nic_names.keys():
+            for model_name in supported_nic_names[vendor_name]:
+                if vendor_name.lower() in eth_crtl.lower() and model_name.lower() in eth_crtl.lower():
+                    pci_num = eth_crtl.split()[0]
+                    rc, data = run_cmd("ls -l /sys/class/net | grep %s" % pci_num)
+                    if rc is not None:
+                        continue
+                    else:
+                        logical_itfs = [i for i in data.split('\n') if i]
+                        for logical_itf in logical_itfs:
+                            logical_itf_names.append(logical_itf.split('/')[-1])
+
+    return logical_itf_names
+
 def get_default_itf():
     """
     Assuming "ip route" returns the following format:

mizar/daemon/app.py

@@ -94,6 +94,11 @@ def init(benchmark=False):
     output = r.stdout.read().decode().strip()
     logging.info("Removed existing XDP program: {}".format(output))
 
+    cmd = "nsenter -t 1 -m -u -n -i ip link set dev " + f'''{default_itf}''' + " xdpoffload off"
+    r = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+    output = r.stdout.read().decode().strip()
+    logging.info("Removed existing offload XDP program: {}".format(output))
+
     cmd = "nsenter -t 1 -m -u -n -i /trn_bin/transitd >transitd.log &"
     r = subprocess.Popen(cmd, shell=True)
     logging.info("Running transitd")
@@ -119,6 +124,24 @@ def init(benchmark=False):
     output = r.stdout.read().decode().strip()
     logging.info("Running load-transit-xdp: {}".format(output))
 
+    # Offload XDP program removes codes about debugging for size limitation.
+    if os.getenv('FEATUREGATE_OFFLOAD_XDP', 'false').lower() in ('true', '1'):
+        if default_itf in supported_offload_xdp_itf_names():
+            config = {
+                "xdp_path": "/trn_xdp/trn_transit_xdp_hardware_offload_ebpf.o",
+                "pcapfile": "/bpffs/transit_xdp_offload.pcap",
+                "xdp_flag": CONSTANTS.XDP_OFFLOAD
+            }
+            config = json.dumps(config)
+            cmd = (f'''nsenter -t 1 -m -u -n -i /trn_bin/transit -s {nodeip} load-transit-xdp -i {default_itf} -j '{config}' ''')
+            r = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+            output = r.stdout.read().decode().strip()
+            logging.info("Running load-transit-xdp with offload mode: {}".format(output))
+        else:
+            logging.info("Offloading transit XDP functionality not supported for interface {}".format(default_itf))
+    else:
+        logging.info("Offload XDP feature is disabled.")
+
     if os.getenv('FEATUREGATE_BWQOS', 'false').lower() in ('false', '0'):
         logging.info("Bandwidth QoS feature is disabled.")
         return