diff --git a/docs/apps/lnw/es2k/es2k-ecmp-topology.png b/docs/apps/lnw/es2k/es2k-ecmp-topology.png deleted file mode 100644 index 83af6f14..00000000 Binary files a/docs/apps/lnw/es2k/es2k-ecmp-topology.png and /dev/null differ diff --git a/docs/apps/lnw/es2k/es2k-frr-topology.png b/docs/apps/lnw/es2k/es2k-frr-topology.png deleted file mode 100644 index 9f5d66b4..00000000 Binary files a/docs/apps/lnw/es2k/es2k-frr-topology.png and /dev/null differ diff --git a/docs/apps/lnw/es2k/es2k-linux-networking-ecmp.md b/docs/apps/lnw/es2k/es2k-linux-networking-ecmp.md deleted file mode 100644 index c40f92e4..00000000 --- a/docs/apps/lnw/es2k/es2k-linux-networking-ecmp.md +++ /dev/null @@ -1,391 +0,0 @@ - - -# Linux Networking with ECMP (ES2K) - -This document explains how to run the Linux networking scenario on ES2K. - -## Topology - -![Linux Networking Topology](es2k-ecmp-topology.png) - -Notes about topology: - -- Four Kernel netdevs are created by default by loading IDPF driver during ACC bring-up. You can also create more than Four netdevs. For that, we need to modify `acc_apf` parameter under `num_default_vport` in `/etc/dpcp/cfg/cp_init.cfg` on IMC before starting `run_default_init_app`. -- In `/etc/dpcp/cfg/cp_init.cfg` file also modify default `sem_num_pages` value to the value mentioned in `/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking/README_P4_CP_NWS`. -- vlan1, vlan2, .... vlanN created using Linux commands and are on top of an IDPF Netdev. These VLAN ports should be equal to number of VM's that are spawned. -- br-int, VxLAN ports are created using ovs-vsctl command provided by the networking recipe and all the vlan ports are attached to br-int using ovs-vsctl command. - -System under test will have above topology running the networking recipe. Link Partner can have the networking recipe or legacy OvS or kernel VxLAN. Note the [Limitations](#limitations) section before setting up the topology. - -## Create P4 artifacts and start Infrap4d process - -- Use Linux networking p4 program present in the directory `/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking` for this scenario. -- See [Running Infrap4d on Intel IPU E2100](/guides/es2k/running-infrap4d) for compiling `P4 artifacts`, `bringing up ACC` and running `infrap4d` on ACC. - -## Creating the topology - -The p4rt-ctl and ovs-vsctl utilities can be found in $P4CP_INSTALL/bin. - -### Set the forwarding pipeline - -Once the application is started, set the forwarding pipeline config using -P4Runtime Client `p4rt-ctl` set-pipe command - -```bash -$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/linux_networking.pb.bin \ - $OUTPUT_DIR/linux_networking.p4info.txt -``` - -Note: Assuming `linux_networking.pb.bin` and `linux_networking.p4info.txt` -along with other P4 artifacts are created as per the steps mentioned in previous section. - -### Configure VSI Group and add a netdev - -Use one of the IPDF netdevs on ACC to receive all control packets from overlay -VM's by assigning to a VSI group. VSI group 3 is dedicated for this configuration, -execute below devmem commands on IMC. - -```bash -# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig -devmem 0x20292002a0 64 0x8000050000000008 - -# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi -# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-3 -devmem 0x2029200388 64 0x3 - -# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig -devmem 0x20292002a0 64 0xA000050000000008 -``` - -Note: Here VSI 8 has been used for receiving all control packets and added -to VSI group 3. This refers to HOST netdev VSIG 3 as per the topology -diagram. Modify this VSI based on your configuration. - -### Create Overlay network - -Option 1: Create VF's on HOST and spawn VM's on top of those VF's. -Example to create 4 VF's: echo 4 > /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/sriov_numvfs - -```bash -# VM1 configuration -telnet -ip addr add 99.0.0.1/24 dev -ifconfig up - -# VM2 configuration -telnet -ip addr add 99.0.0.2/24 dev -ifconfig up -``` - -Option 2: If we are unable to spawn VM's on top of the VF's, for this use case we can also leverage kernel network namespaces. -Move each VF to a network namespace and assign IP addresses - -```bash -ip netns add VM0 -ip link set netns VM0 -ip netns exec VM0 ip addr add 99.0.0.1/24 dev -ip netns exec VM0 ifconfig up - -ip netns add VM1 -ip link set netns VM1 -ip netns exec VM1 ip addr add 99.0.0.2/24 dev -ip netns exec VM1 ifconfig up -``` - -### Start OvS as a separate process - -Legacy OvS is used as a control plane for source MAC learning of overlay VM's. OvS should be started as a seperate process. - -```bash -export RUN_OVS=/tmp -rm -rf $RUN_OVS/etc/openvswitch -rm -rf $RUN_OVS/var/run/openvswitch -mkdir -p $RUN_OVS/etc/openvswitch/ -mkdir -p $RUN_OVS/var/run/openvswitch - -ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \ - /opt/p4/p4-cp-nws/share/openvswitch/vswitch.ovsschema - -ovsdb-server $RUN_OVS/etc/openvswitch/conf.db \ - --remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \ - --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ - --pidfile=$RUN_OVS/var/run/openvswitch/ovsdb-server.pid \ - --unixctl=$RUN_OVS/var/run/openvswitch/ovsdb-server.ctl \ - --detach - -ovs-vswitchd --detach \ - --pidfile=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.pid \ - --no-chdir unix:$RUN_OVS/var/run/openvswitch/db.sock \ - --unixctl=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.ctl \ - --mlockall \ - --log-file=/tmp/ovs-vswitchd.log - -alias ovs-vsctl="ovs-vsctl --db unix:$RUN_OVS/var/run/openvswitch/db.sock" -ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1 -ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1 - -ovs-vsctl show -``` - -### Create VLAN representers - -For each VM that is spawned for overlay network we need to have a port representer. -We create VLAN netdevs on top of the IPDF netdev which is assigned to VSI group 3 in step-2 mentioned above. - -```bash -ip link add link name vlan1 type vlan id 1 -ip link add link name vlan2 type vlan id 2 -ifconfig vlan1 up -ifconfig vlan2 up -``` - -Note: Here the assumption is, we have created 2 overlay VM's and creating 2 port representers for those VM's. -Port representer should always be in the format: `lowercase string 'vlan'+'vlanID'` - -### Create integration bridge and add ports to the bridge - -Create OvS bridge, VxLAN tunnel and assign ports to the bridge. - -```bash -ovs-vsctl add-br br-int -ifconfig br-int up - -ovs-vsctl add-port br-int vlan1 -ovs-vsctl add-port br-int vlan2 -ifconfig vlan1 up -ifconfig vlan2 up - -ovs-vsctl add-port br-int vxlan1 -- set interface vxlan1 type=vxlan \ - options:local_ip=30.1.1.1 options:remote_ip=40.1.1.1 options:dst_port=4789 -``` - -Note: Here we are creating VxLAN tunnel with VNI 0, you can create any VNI for tunneling. - -### Configure rules for overlay control packets - -Configure rules to send overlay control packets from a VM to its respective port representers. - -Below configuration assumes - -- Overlay VF1 has a VSI value 14 -- Overlay VF2 has a VSI value 15 - -These VSI values can be checked with `/usr/bin/cli_client -q -c` command -on IMC. This command provides VSI ID, Vport ID, and corresponding MAC -addresses for all: - -- IDPF netdevs on ACC -- VF's on HOST -- IDPF netdevs on HOST (if IDPF driver loaded by you on HOST) -- Netdevs on IMC - -```bash -# Rules for control packets coming from overlay VF (VSI-14). -# IPU will add a VLAN tag 1 and send to HOST1 (VSI-8). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(1,24)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ - "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ - "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_push(1,0,1)" - -# Rules for control packets coming from overlay VF (VSI-15). -# IPU will add a VLAN tag 2 and send to HOST1 (VSI-8). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(2,24)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ - "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ - "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_push(1,0,2)" - -# Rules for control packets coming from HOST1 (VSI-8). -# IPU will remove the VLAN tag 1 and send to overlay VF (VSI-14). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ - "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=1,action=linux_networking_control.remove_vlan_and_send_to_port(1,30)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ - "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(30)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ - "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_pop" - -# Rules for control packets coming from HOST1 (VSI-8). -# IPU will remove the VLAN tag 2 and send to overlay VF (VSI-15). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ - "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=2,action=linux_networking_control.remove_vlan_and_send_to_port(2,31)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ - "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(31)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ - "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_pop" -``` - -### Configure rules for underlay control packets - -Configure rules to send underlay control packets from IDPF netdev to physical port. - -Below configuration assumes - -- Underlay IDPF netdev has a VSI value 10 for first ECMP member -- First physical port will have a port ID of 0 -- Underlay IDPF netdev has a VSI value 11 for second ECMP member -- Second physical port will have a port ID of 1 - -```bash -# Configuration for control packets between physical port 0 to underlay IDPF netdev VSI-10 -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table \ - "vmeta.common.port_id=0,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(26)" - -# Configuration for control packets between underlay IDPF netdev VSI-10 to physical port 0 -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=10,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(0)" - -# Configuration for control packets between physical port 1 to underlay IDPF netdev VSI-11 -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table \ - "vmeta.common.port_id=1,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(27)" - -# Configuration for control packets between underlay IDPF netdev VSI-11 to physical port 1 -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=11,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(1)" -``` - -### Underlay configuration - -Create a dummy interface which is used for TEP termination and IDPF netdev for underlay connectivity. - -Option 1: Statically configure underlay ECMP routes to reach link partner (or) \ -Option 2: Use BGP protocol with FRR, for ECMP route redistribution to reach link partner. - -Below configuration assumes - -- Underlay IDPF netdev has a VSI value 10 for first ECMP member -- Underlay IDPF netdev has a VSI value 11 for second ECMP member - -```bash -p4rt-ctl add-entry br0 linux_networking_control.ecmp_lpm_root_lut \ - "user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=2,action=linux_networking_control.ecmp_lpm_root_lut_action(0)" - -ip link add dev TEP0 type dummy - -nmcli device set managed no -nmcli device set managed no -``` - -Option 1: Static configuration - -```bash -ifconfig TEP0 30.1.1.1/24 up -ifconfig 50.1.1.1/24 up -ifconfig 60.1.1.1/24 up - -ip route add 40.1.1.1 nexthop via 50.1.1.2 dev \ - weight 1 nexthop via 60.1.1.2 dev weight 1 -ip route show -``` - -Sample link partner underlay configuration. - -```bash -ip link add dev TEP0 type dummy -ifconfig TEP0 40.1.1.1/24 up -ifconfig 50.1.1.2/24 up -ifconfig 60.1.1.2/24 up - -ip route add 30.1.1.1 nexthop via 50.1.1.1 dev \ - weight 1 nexthop via 60.1.1.1 dev weight 1 -ip route show - -``` - -Option 2: FRR running configuration - -```bash -# -# - -interface TEP0 -ip address 30.1.1.1/24 -exit -! -interface -ip address 50.1.1.1/24 -exit -! -interface -ip address 60.1.1.1/24 -exit -! -router bgp 65000 -bgp router-id 30.1.1.1 -neighbor 50.1.1.2 remote-as 65000 -neighbor 60.1.1.2 remote-as 65000 -! -address-family ipv4 unicast - network 30.1.1.0/24 -exit-address-family -``` - -Sample link partner FRR VTYSH configuration. - -```bash -interface TEP0 -ip address 40.1.1.1/24 -exit -! -interface -ip address 50.1.1.2/24 -exit -! -interface -ip address 60.1.1.2/24 -exit -! -router bgp 65000 -bgp router-id 40.1.1.1 -neighbor 50.1.1.1 remote-as 65000 -neighbor 60.1.1.1 remote-as 65000 -! -address-family ipv4 unicast - network 40.1.1.0/24 -exit-address-family -``` - -### Test the ping scenarios - -- Ping between VM's on the same host -- First Underlay ping -- Second Underlay ping -- Remote TEP ping -- Overlay ping: Ping between VM's on different hosts - -## Limitations - -Current Linux Networking support for the networking recipe has following limitations: - -- All VLAN interfaces created on top of IDPF netdev, should always be in lowercase format "vlan+vlan_id" -Ex: vlan1, vlan2, vlan3 ... vlan4094. -- Set the pipeline before adding br-int port, vxlan0 port, and adding vlan ports to br-int bridge. -- VxLAN destination port should always be standard port. i.e., 4789. (limitation by p4 parser) -- We do not support any ofproto rules that would prevent FDB learning on OvS. -- VLAN Tagged packets are not supported. -- For VxLAN tunneled packets only IPv4-in-IPv4 is supported. diff --git a/docs/apps/lnw/es2k/es2k-linux-networking-frr.md b/docs/apps/lnw/es2k/es2k-linux-networking-frr.md deleted file mode 100644 index 23510894..00000000 --- a/docs/apps/lnw/es2k/es2k-linux-networking-frr.md +++ /dev/null @@ -1,333 +0,0 @@ - - -# Linux Networking with FRR (ES2K) - -This document explains how to run the Linux networking scenario on ES2K. - -## Topology - -![FRR Topology](es2k-frr-topology.png) - -Notes about topology: - -- Four Kernel netdevs are created by default by loading IDPF driver during ACC bring-up. You can also create more than Four netdevs. For that, we need to modify `acc_apf` parameter under `num_default_vport` in `/etc/dpcp/cfg/cp_init.cfg` on IMC before starting `run_default_init_app`. -- In `/etc/dpcp/cfg/cp_init.cfg` file also modify default `sem_num_pages` value to the value mentioned in `/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking/README_P4_CP_NWS`. -- vlan1, vlan2, .... vlanN created using Linux commands and are on top of an IDPF Netdev. These VLAN ports should be equal to number of VM's that are spawned. -- br-int, VxLAN ports are created using ovs-vsctl command provided by the networking recipe and all the vlan ports are attached to br-int using ovs-vsctl command. - -System under test will have above topology running the networking recipe. Link Partner can have the networking recipe or legacy OvS or kernel VxLAN. Note the [Limitations](#limitations) section before setting up the topology. - -## Create P4 artifacts and start Infrap4d process - -- Use Linux networking p4 program present in the directory `/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking` for this scenario. -- See [Running Infrap4d on Intel IPU E2100](/guides/es2k/running-infrap4d.md) for compiling `P4 artifacts`, `bringing up ACC` and running `infrap4d` on ACC. - -## Creating the topology - -The p4rt-ctl and ovs-vsctl utilities can be found in $P4CP_INSTALL/bin. - -### Set the forwarding pipeline - -Once the application is started, set the forwarding pipeline config using -P4Runtime Client `p4rt-ctl` set-pipe command - -```bash -$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/linux_networking.pb.bin \ - $OUTPUT_DIR/linux_networking.p4info.txt -``` - -Note: Assuming `linux_networking.pb.bin` and `linux_networking.p4info.txt` along with other P4 artifacts are created as per the steps mentioned in previous section. - -### Configure VSI Group and add a netdev - -Use one of the IPDF netdevs on ACC to receive all control packets from overlay VM's by assigning to a VSI group. VSI group 3 is dedicated for this configuration, execute below devmem commands on IMC. - -```bash -# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig -devmem 0x20292002a0 64 0x8000050000000008 - -# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi -# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-3 -devmem 0x2029200388 64 0x3 - -# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig -devmem 0x20292002a0 64 0xA000050000000008 -``` - -Note: Here VSI 8 has been used for receiving all control packets and added to VSI group 3. This refers to HOST netdev VSIG 3 as per the topology diagram. Modify this VSI based on your configuration. - -### Create Overlay network - -Option 1: Create VF's on HOST and spawn VM's on top of those VF's. -Example to create 4 VF's: echo 4 > /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/sriov_numvfs - -```bash -# VM1 configuration -telnet -ip addr add 99.0.0.1/24 dev -ifconfig up - -# VM2 configuration -telnet -ip addr add 99.0.0.2/24 dev -ifconfig up -``` - -Option 2: If we are unable to spawn VM's on top of the VF's, we can leverage kernel network namespaces. -Move each VF to a network namespace and assign IP addresses: - -```bash -ip netns add VM0 -ip link set netns VM0 -ip netns exec VM0 ip addr add 99.0.0.1/24 dev -ip netns exec VM0 ifconfig up - -ip netns add VM1 -ip link set netns VM1 -ip netns exec VM1 ip addr add 99.0.0.2/24 dev -ip netns exec VM1 ifconfig up -``` - -### Start OvS as a separate process - -Legacy OvS is used as a control plane for source MAC learning of overlay VM's. OvS should be started as a seperate process. - -```bash -export RUN_OVS=/tmp -rm -rf $RUN_OVS/etc/openvswitch -rm -rf $RUN_OVS/var/run/openvswitch -mkdir -p $RUN_OVS/etc/openvswitch/ -mkdir -p $RUN_OVS/var/run/openvswitch - -ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \ - /opt/p4/p4-cp-nws/share/openvswitch/vswitch.ovsschema - -ovsdb-server $RUN_OVS/etc/openvswitch/conf.db \ - --remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \ - --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ - --pidfile=$RUN_OVS/var/run/openvswitch/ovsdb-server.pid \ - --unixctl=$RUN_OVS/var/run/openvswitch/ovsdb-server.ctl \ - --detach - -ovs-vswitchd --detach \ - --pidfile=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.pid \ - --no-chdir unix:$RUN_OVS/var/run/openvswitch/db.sock \ - --unixctl=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.ctl \ - --mlockall \ - --log-file=/tmp/ovs-vswitchd.log - -alias ovs-vsctl="ovs-vsctl --db unix:$RUN_OVS/var/run/openvswitch/db.sock" -ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1 -ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1 - -ovs-vsctl show -``` - -### Create VLAN representers - -For each VM that is spawned for overlay network we need to have a port representer. We create VLAN netdevs on top of the IPDF netdev which is assigned to VSI group 3 in step-2 mentioned above. - -```bash -ip link add link name vlan1 type vlan id 1 -ip link add link name vlan2 type vlan id 2 -ifconfig vlan1 up -ifconfig vlan2 up -``` - -Note: Here the assumption is, we have created 2 overlay VM's and creating -2 port representers for those VM's. Port representer should always be in -the format: `lowercase string 'vlan'+'vlanID'` - -### Create integration bridge and add ports to the bridge - -Create OvS bridge, VxLAN tunnel and assign ports to the bridge. - -```bash -ovs-vsctl add-br br-int -ifconfig br-int up - -ovs-vsctl add-port br-int vlan1 -ovs-vsctl add-port br-int vlan2 -ifconfig vlan1 up -ifconfig vlan2 up - -ovs-vsctl add-port br-int vxlan1 -- set interface vxlan1 type=vxlan \ - options:local_ip=30.1.1.1 options:remote_ip=40.1.1.1 options:dst_port=4789 -``` - -Note: Here we are creating VxLAN tunnel with VNI 0, you can create any VNI for tunneling. - -### Configure rules for overlay control packets - -Configure rules to send overlay control packets from a VM to its respective port representers. - -Below configuration assumes - -- Overlay VF1 has a VSI value 14 -- Overlay VF2 has a VSI value 15 - -These VSI values can be checked with `/usr/bin/cli_client -q -c` command on IMC. -This command provides VSI ID, Vport ID, and corresponding MAC addresses for all: - -- IDPF netdevs on ACC -- VF's on HOST -- IDPF netdevs on HOST (if IDPF driver loaded by you on HOST) -- Netdevs on IMC - -```bash -# Rules for control packets coming from overlay VF (VSI-14). -# IPU will add a VLAN tag 1 and send to HOST1 (VSI-8). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(1,24)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ - "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ - "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_push(1,0,1)" - -# Rules for control packets coming from overlay VF (VSI-15). -# IPU will add a VLAN tag 2 and send to HOST1 (VSI-8). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(2,24)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ - "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ - "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_push(1,0,2)" - -# Rules for control packets coming from HOST1 (VSI-8). -# IPU will remove the VLAN tag 1 and send to overlay VF (VSI-14). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ - "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=1,action=linux_networking_control.remove_vlan_and_send_to_port(1,30)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ - "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(30)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ - "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_pop" - -# Rules for control packets coming from HOST1 (VSI-8). -# IPU will remove the VLAN tag 2 and send to overlay VF (VSI-15). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ - "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=2,action=linux_networking_control.remove_vlan_and_send_to_port(2,31)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ - "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(31)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ - "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_pop" -``` - -### Configure rules for underlay control packets - -Configure rules to send underlay control packets from IDPF netdev to physical port. - -Below configuration assumes - -- Underlay IDPF netdev has a VSI value 10 -- First physical port will have a port ID of 0 - -```bash -# Configuration for control packets between physical port 0 to underlay IDPF netdev VSI-10 -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table \ - "vmeta.common.port_id=0,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(26)" - -# Configuration for control packets between underlay IDPF netdev VSI-10 to physical port 0 -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=10,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(0)" -``` - -### Underlay configuration - -Create a dummy interface which is used for TEP termination and IDPF netdev for -underlay connectivity. Use BGP protocol with FRR, for route redistribution. - -Below configuration assumes - -- Underlay IDPF netdev has a VSI value 10 - -```bash -p4rt-ctl add-entry br0 linux_networking_control.ecmp_lpm_root_lut \ - "user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=2,action=linux_networking_control.ecmp_lpm_root_lut_action(0)" - -nmcli device set managed no -ip link add dev TEP0 type dummy -``` - -FRR running configuration - -```bash -# -# - -interface TEP0 -ip address 30.1.1.1/24 -exit -! -interface -ip address 70.1.1.1/24 -exit -! -router bgp 65000 -bgp router-id 30.1.1.1 -neighbor 70.1.1.1 remote-as 65000 -! -address-family ipv4 unicast -network 30.1.1.0/24 -exit-address-family -``` - -Sample link partner FRR VTYSH configuration. - -```bash -interface TEP0 -ip address 40.1.1.1/24 -exit -! -interface -ip address 70.1.1.2/24 -exit -! -router bgp 65000 -bgp router-id 40.1.1.1 -neighbor 70.1.1.2 remote-as 65000 -! -address-family ipv4 unicast - network 40.1.1.0/24 -exit-address-family -``` - -### Test the ping scenarios - -- Ping between VM's on the same host -- Underlay ping -- Remote TEP ping -- Overlay ping: Ping between VM's on different hosts - -## Limitations - -Current Linux Networking support for the networking recipe has following limitations: - -- All VLAN interfaces created on top of IDPF netdev, should always be in lowercase format "vlan+vlan_id" -Ex: vlan1, vlan2, vlan3 ... vlan4094. -- Set the pipeline before adding br-int port, vxlan0 port, and adding vlan ports to br-int bridge. -- VxLAN destination port should always be standard port. i.e., 4789. (limitation by p4 parser) -- We do not support any ofproto rules that would prevent FDB learning on OvS. -- VLAN Tagged packets are not supported. -- For VxLAN tunneled packets only IPv4-in-IPv4 is supported. diff --git a/docs/apps/lnw/es2k/es2k-linux-networking-ipv6.md b/docs/apps/lnw/es2k/es2k-linux-networking-ipv6.md deleted file mode 100644 index 31869ed1..00000000 --- a/docs/apps/lnw/es2k/es2k-linux-networking-ipv6.md +++ /dev/null @@ -1,324 +0,0 @@ - - -# Linux Networking with IPv6 (ES2K) - -This document explains how to run the Linux networking scenario on ES2K. - -## Topology - -![Linux Networking with IPv6](es2k-lnw-topology.png) - -Notes about topology: - -- Four Kernel netdevs are created by default by loading IDPF driver during -ACC bring-up. You can also create more than Four netdevs. For that, we -need to modify `acc_apf` parameter under `num_default_vport` in -`/etc/dpcp/cfg/cp_init.cfg` on IMC before starting `run_default_init_app`. - -- In `/etc/dpcp/cfg/cp_init.cfg` file also modify default `sem_num_pages` -value to the value mentioned in -`/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking/README_P4_CP_NWS`. - -- vlan1, vlan2, .... vlanN created using Linux commands and are on top of -an IDPF Netdev. These VLAN ports should be equal to number of VM's that -are spawned. - -- br-int, VxLAN ports are created using ovs-vsctl command provided by the -networking recipe and all the vlan ports are attached to br-int using -ovs-vsctl command. - -System under test will have above topology running the networking recipe. -Link Partner can have the networking recipe or legacy OvS or kernel VxLAN. -Note the [Limitations](#limitations) section before setting up the -topology. - -## Create P4 artifacts and start Infrap4d process - -- Use Linux networking p4 program present in the directory -`/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking` for this -scenario. - -- See [Running Infrap4d on Intel IPU E2100](/guides/es2k/running-infrap4d) -for compiling `P4 artifacts`, `bringing up ACC` and running `infrap4d` on -ACC. - -## Creating the topology - -The p4rt-ctl and ovs-vsctl utilities can be found in $P4CP_INSTALL/bin. - -### Set the forwarding pipeline - -Once the application is started, set the forwarding pipeline config using -P4Runtime Client `p4rt-ctl` set-pipe command - -```bash -$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/linux_networking.pb.bin \ - $OUTPUT_DIR/linux_networking.p4info.txt -``` - -Note: Assuming `linux_networking.pb.bin` and `linux_networking.p4info.txt` -along with other P4 artifacts are created as per the steps mentioned in -previous section. - -### Configure VSI Group and add a netdev - -Use one of the IPDF netdevs on ACC to receive all control packets from -overlay VM's by assigning to a VSI group. VSI group 3 is dedicated for -this configuration, execute below devmem commands on IMC. - -```bash -# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig -devmem 0x20292002a0 64 0x8000050000000008 - -# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi -# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-3. -devmem 0x2029200388 64 0x3 - -# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig -devmem 0x20292002a0 64 0xA000050000000008 -``` - -Note: Here VSI 8 has been used for receiving all control packets and added -to VSI group 3. This refers to HOST netdev VSIG 3 as per the topology -diagram. Modify this VSI based on your configuration. - -### Create Overlay network - -Option 1: Create VF's on HOST and spawn VM's on top of those VF's. -Example to create 4 VF's: echo 4 > /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/sriov_numvfs - -```bash -# VM1 configuration -telnet -ip -6 addr add 9::1/64 dev -ifconfig up - -# VM2 configuration -telnet -ip -6 addr add 9::2/64 -ifconfig up -``` - -Option 2: If we are unable to spawn VM's on top of the VF's, for this use -case we can also leverage kernel network namespaces. Move each VF to a -network namespace and assign IP addresses - -```bash -ip netns add VM0 -ip link set netns VM0 -ip netns exec VM0 ip -6 addr add 9::1/64 dev -ip netns exec VM0 ifconfig up - -ip netns add VM1 -ip link set netns VM1 -ip netns exec VM1 ip -6 addr add 9::2/64 dev -ip netns exec VM1 ifconfig up -``` - -### Start OvS as a separate process - -Legacy OvS is used as a control plane for source MAC learning of overlay -VM's. OvS should be started as a seperate process. - -```bash -export RUN_OVS=/tmp -rm -rf $RUN_OVS/etc/openvswitch -rm -rf $RUN_OVS/var/run/openvswitch -mkdir -p $RUN_OVS/etc/openvswitch/ -mkdir -p $RUN_OVS/var/run/openvswitch - -ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \ - /opt/p4/p4-cp-nws/share/openvswitch/vswitch.ovsschema - -ovsdb-server $RUN_OVS/etc/openvswitch/conf.db \ - --remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \ - --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ - --pidfile=$RUN_OVS/var/run/openvswitch/ovsdb-server.pid \ - --unixctl=$RUN_OVS/var/run/openvswitch/ovsdb-server.ctl \ - --detach - -ovs-vswitchd --detach \ - --pidfile=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.pid \ - --no-chdir unix:$RUN_OVS/var/run/openvswitch/db.sock \ - --unixctl=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.ctl \ - --mlockall \ - --log-file=/tmp/ovs-vswitchd.log - -alias ovs-vsctl="ovs-vsctl --db unix:$RUN_OVS/var/run/openvswitch/db.sock" -ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1 -ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1 - -ovs-vsctl show -``` - -### Create VLAN representers - -For each VM that is spawned for overlay network we need to have a port -representer. We create VLAN netdevs on top of the IPDF netdev which is -assigned to VSI group 3 in step-2 mentioned above. - -```bash -ip link add link name vlan1 type vlan id 1 -ip link add link name vlan2 type vlan id 2 -ifconfig vlan1 up -ifconfig vlan2 up -``` - -Note: Here the assumption is, we have created 2 overlay VM's and creating -2 port representers for those VM's. Port representer should always be in -the format: `lowercase string 'vlan'+'vlanID'` - -### Create integration bridge and add ports to the bridge - -Create OvS bridge, VxLAN tunnel and assign ports to the bridge. - -```bash -ovs-vsctl add-br br-int -ifconfig br-int up - -ovs-vsctl add-port br-int vlan1 -ovs-vsctl add-port br-int vlan2 -ifconfig vlan1 up -ifconfig vlan2 up - -ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan \ - options:local_ip=1000:1::1 options:remote_ip=1000:1::2 options:dst_port=4789 -``` - -Note: Here we are creating VxLAN tunnel with VNI 0, you can create any VNI -for tunneling. - -### Configure rules for overlay control packets - -Configure rules to send overlay control packets from a VM to its -respective port representers. - -Below configuration assumes - -- Overlay VF1 has a VSI value 14 -- Overlay VF2 has a VSI value 15 - -These VSI values can be checked with `/usr/bin/cli_client -q -c` command -on IMC. This command provides VSI ID, Vport ID, and corresponding MAC -addresses for all: - -- IDPF netdevs on ACC -- VF's on HOST -- IDPF netdevs on HOST (if IDPF driver loaded by you on HOST) -- Netdevs on IMC - -```bash -# Rules for control packets coming from overlay VF (VSI-14). -# IPU will add a VLAN tag 1 and send to HOST1 (VSI-8). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(1,24)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ - "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ - "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_push(1,0,1)" - -# Rules for control packets coming from overlay VF (VSI-15). -# IPU will add a VLAN tag 2 and send to HOST1 (VSI-8). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(2,24)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ - "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ - "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_push(1,0,2)" - -# Rules for control packets coming from HOST1 (VSI-8). -# IPU will remove the VLAN tag 1 and send to overlay VF (VSI-14). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ - "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=1,action=linux_networking_control.remove_vlan_and_send_to_port(1,30)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ - "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(30)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ - "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_pop" - -# Rules for control packets coming from HOST1 (VSI-8). -# IPU will remove the VLAN tag 2 and send to overlay VF (VSI-15). - -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ - "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=2,action=linux_networking_control.remove_vlan_and_send_to_port(2,31)" -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ - "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(31)" -p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ - "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_pop" -``` - -### Configure rules for underlay control packets - -Configure rules to send underlay control packets from IDPF netdev to physical port. - -Below configuration assumes - -- Underlay IDPF netdev has a VSI value 10 -- First physical port will have a port ID of 0 - -```bash -# Configuration for control packets between physical port 0 and underlay IDPF netdev VSI-10 -p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table \ - "vmeta.common.port_id=0,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(26)" - -# Configuration for control packets between underlay IDPF netdev VSI-10 and physical port 0 -p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ - "vmeta.common.vsi=10,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(0)" -``` - -### Underlay configuration - -Configure underlay IP addresses, and add static routes. - -Below configuration assumes - -- Underlay IDPF netdev has a VSI value 10 - -```bash -p4rt-ctl add-entry br0 linux_networking_control.ipv6_lpm_root_lut \ - "user_meta.cmeta.bit32_zeros=0/255.255.255.255,priority=1,action=linux_networking_control.ipv6_lpm_root_lut_action(0)" - -nmcli device set managed no -ip -6 addr add 1000:1::1/64 dev -ip route del 1000:1::/64 dev -ip route add 1000:1::/64 dev -ip -6 route change 1000:1::/64 via 1000:1::2 dev -``` - -### Test the ping scenarios - -- Ping6 between VM's on the same host -- Underlay ping6 -- Overlay ping: Ping6 between VM's on different hosts - -## Limitations - -Current Linux Networking support for the networking recipe has following limitations: - -- All VLAN interfaces created on top of IDPF netdev, should always be in lowercase format "vlan+vlan_id" -Ex: vlan1, vlan2, vlan3 ... vlan4094. -- Set the pipeline before adding br-int port, vxlan0 port, and adding vlan ports to br-int bridge. -- VxLAN destination port should always be standard port. i.e., 4789. (limitation by p4 parser) -- We do not support any ofproto rules that would prevent FDB learning on OvS. -- VLAN Tagged packets are not supported. -- For VxLAN tunneled packets only IPv6-in-IPv6 is supported. diff --git a/docs/apps/lnw/es2k/es2k-lnw-overlay-ipv6.md b/docs/apps/lnw/es2k/es2k-lnw-overlay-ipv6.md new file mode 100644 index 00000000..2d5e7db4 --- /dev/null +++ b/docs/apps/lnw/es2k/es2k-lnw-overlay-ipv6.md @@ -0,0 +1,362 @@ + + +# Linux Networking with IPv6 (ES2K) + +This document explains how to run the Linux networking scenario on ES2K. + +## Topology + +![Linux Networking Topology](es2k-lnw-topology.png) + +See [Linux Networking for ES2K](es2k-linux-networking.md) +for more details on this feature. + +Prerequisites: + +- Download `hw-p4-programs` TAR file specific to the build and extract it to get `fxp-net_linux-networking-v2` p4 artifacts. Go through `Limitations` specified in `README` and bring up the setup accordingly. +- Follow steps mentioned in [Deploying P4 Programs for E2100](/guides/es2k/deploying-p4-programs) for bringing up IPU with a custom P4 package. +Modify `load_custom_pkg.sh` with following parameters for linux_networking package: + +```text + sed -i 's/sem_num_pages = 1;/sem_num_pages = 25;/g' $CP_INIT_CFG + sed -i 's/lem_num_pages = 1;/lem_num_pages = 10;/g' $CP_INIT_CFG + sed -i 's/acc_apf = 4;/acc_apf = 8;/g' $CP_INIT_CFG +``` + +- Download `IPU_Documentation` TAR file specific to the build and refer to `Getting Started Guide` on how to install compatible `IDPF driver` on host. Once an IDPF driver is installed, bring up SRIOV VF by modifying the `sriov_numvfs` file present under one of the IDPF network devices. Example as below + + ```bash + echo 1 > /sys/class/net/ens802f0/device/sriov_numvfs + ``` + +Notes about topology: + +- VMs are spawned on top of each VF. Each VF will have a port representor in ACC. P4 runtime rules are configured to map VFs and their corresponding port representors. +- Each physical port will have a port representor in ACC. P4 runtime rules are configured to map VFs and their corresponding port representors. +- Each physical port will have a corresponding APF netdev on HOST. Create port representors in ACC for each HOST APF netdev. These APF netdevs on HOST will receive unknown traffic for applications to act on. +- All port representors should be part of an OvS bridge. Based on topology, this OvS bridge will just perform bridging or TEP termination on a bridge which is used to enable underlay connectivity for VxLAN traffic. +- OvS bridges and VxLAN ports are created using ovs-vsctl command (provided by the networking recipe). +- All port representors are attached to OvS bridge using ovs-vsctl command. +- This config has: + - 1 Overlay VF + - 1 Port representors in ACC for the 1 Overlay VF + - 2 physical ports + - 2 Port representors in ACC for the 2 physical ports + - 2 APF netdevs on HOST + - 2 Port representors in ACC for the 2 HOST APF netdevs + - TEP termination dummy interface + +System under test will have above topology running the networking recipe. Link Partner can have the networking recipe or legacy OvS or kernel VxLAN. Refer to the Limitations section in [Linux Networking for E2100](es2k-linux-networking.md) before setting up the topology. + +## Creating the topology + +Follow steps mentioned in [Running Infrap4d on Intel E2100](/guides/es2k/running-infrap4d.md) for starting `infrap4d` process and creating protobuf binary for `fxp-net_linux-networking-v2` p4 program. + +### Port mapping + +These VSI values can be checked with `/usr/bin/cli_client -q -c` command on IMC. This command provides VSI ID, Vport ID, and MAC addresses for all: + +- IDPF netdevs on ACC +- VFs on HOST +- IDPF netdevs on HOST (if IDPF driver loaded by you on HOST) +- Netdevs on IMC + +| Overlay VFs | Overlay VFs VSI ID | ACC port representor | ACC port representor VSI ID | +| ------------| ------------------ | -------------------- | --------------------------- | +| ens802f0v0 | (0x1b) 27 | enp0s1f0d1 | (0x09) 9 | + +| Physical port | Physical port ID | ACC Port presenter | ACC Port presenter VSI ID | +| ------------- | ----------------- | -------------------- | ------------------------- | +| Phy port 0 | (0x0) 0 | enp0s1f0d9 | (0x11) 17 | +| Phy port 1 | (0x1) 1 | enp0s1f0d11 | (0x13) 19 | + +| APF netdev | APF netdev VSI ID | ACC Port presenter | ACC Port presenter VSI ID | +| ------------- | ----------------- | -------------------- | ------------------------- | +| ens802f0d1 | (0x18) 24 | enp0s1f0d10 | (0x12) 18 | +| ens802f0d2 | (0x19) 25 | enp0s1f0d12 | (0x14) 20 | + +(NOTE: Port names and their VSI IDs may differ from setup to setup. Configure accordingly.) + +### Set the forwarding pipeline + +Once the application is started, set the forwarding pipeline config using +P4Runtime Client `p4rt-ctl` set-pipe command + +```bash +$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/fxp-net_linux-networking-v2.pb.bin \ + $OUTPUT_DIR/p4info.txt +``` + +Note: Assumes that `fxp-net_linux-networking-v2.pb.bin`, `p4info.txt` and other P4 artifacts, are created by following the steps in the previous section. + +### Configure VSI Group and add a netdev + +Add all ACC port representors to VSI group 1. VSI group 1 is dedicated for this configuration. +Execute the following devmem commands on IMC. + +```bash +# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which needs to map into vsig +devmem 0x20292002a0 64 0x8000050000000009 + +# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi +# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-1. +devmem 0x2029200388 64 0x1 + +# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which needs to map into vsig +devmem 0x20292002a0 64 0xA000050000000009 +``` + +Note: Here VSI 9 has been used as one of the ACC port representors and added to VSI group 1. For this use case, add all 5 IDPF interfaces created on ACC. Modify this VSI based on your configuration. + +### Start OvS as a separate process + +Enhanced OvS is used as a control plane for source MAC learning of overlay VMs. This OvS binary is available as part of ACC build and should be started as a separate process. + +```bash +export RUN_OVS=/opt/p4/p4-cp-nws + +rm -rf $RUN_OVS/etc/openvswitch +rm -rf $RUN_OVS/var/run/openvswitch +mkdir -p $RUN_OVS/etc/openvswitch/ +mkdir -p $RUN_OVS/var/run/openvswitch + + +ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \ + $RUN_OVS/share/openvswitch/vswitch.ovsschema + +ovsdb-server \ + --remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \ + --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ + --pidfile --detach + +ovs-vsctl --no-wait init + +mkdir -p /tmp/logs +ovs-vswitchd --pidfile --detach --mlockall \ + --log-file=/tmp/logs/ovs-vswitchd.log + +ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1 +ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1 + +ovs-vsctl show +``` + +### Create Overlay network + +Option 1: Create VFs on HOST and spawn VMs on top of those VFs. +Example: Below config is provided for one VM, and considering each VM is in one VLAN. Extend this to other VMs. + +```bash +# VM1 configuration +telnet +ip link add link name .10 type vlan id 10 +ip addr add 9::1/64 dev .10 +ifconfig up +ifconfig .10 up +``` + +Option 2: If we are unable to spawn VMs on top of the VFs, we can leverage kernel network namespaces. +Move each VF to a network namespace and assign IP addresses. +Example: Below config is provided for one overlay port, and tag each port in the namespace to a single VLAN. Extend this to other namespaces. + +```bash +ip netns add VM0 +ip link set netns VM0 +ip netns exec VM0 ip link add link name .10 type vlan id 10 +ip netns exec VM0 ip addr add 9::1/64 dev .10 +ip netns exec VM0 ifconfig up +ip netns exec VM0 ifconfig .10 up +``` + +### Configure rules for mapping between Overlay VF and ACC port representor + +Configure rules to send overlay packets from a VM to its associated port representors. + +Refer to above port mapping for overlay VF to ACC port representor mapping. Here, sample commands are shown for a single overlay network. Configure similar mapping for remaining VFs. + +Example: + +- Overlay VF1 has a VSI value 27, its corresponding port representor has VSI value 9 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for an overlay VF (VSI-27). Source port action can be any value. +# For simplicity add 16 to VSI ID. + p4rt-ctl add-entry br0 linux_networking_control.tx_source_port_v6 \ + "vmeta.common.vsi=27,zero_padding=0,action=linux_networking_control.set_source_port(43)" + +# Create a mapping between overlay VF (VSI-27/source port-43) and ACC port representor (VSI-9) + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=43,zero_padding=0,action=linux_networking_control.fwd_to_vsi(25)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=9,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(43)" + +# Create a mapping for traffic to flow between VSIs (VSI-27/source port-43) and (VSI-9) + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=9,target_vsi=27,action=linux_networking_control.fwd_to_vsi(43)" + + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=27,target_vsi=9,action=linux_networking_control.fwd_to_vsi(25)" + +``` + +### Configure rules for mapping between Physical port and ACC port representor + +Configure rules to send ingress packets from a physical port to its associated port representors. + +Refer to above port mapping for physical port to ACC port representor mapping. Here, sample commands are shown for a single physical port. Configure similar mapping for remaining physical ports. + +Example: + +- Physical port 0 port id is 0, its corresponding port representor has VSI value 17 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for a physical port (Phy port-0). Source port action can be any value. +# For simplicity consider the same value as phy port id. + p4rt-ctl add-entry br0 linux_networking_control.rx_source_port \ + "vmeta.common.port_id=0,zero_padding=0,action=linux_networking_control.set_source_port(0)" + +# Create a mapping between physical port (Phy port 0/src port 0) and ACC port representor (VSI-17) + p4rt-ctl add-entry br0 linux_networking_control.rx_phy_port_to_pr_map \ + "vmeta.common.port_id=0,zero_padding=0,action=linux_networking_control.fwd_to_vsi(33)" + + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=0,zero_padding=0,action=linux_networking_control.fwd_to_vsi(33)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=17,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(0)" +``` + +### Configure rules for mapping between APF netdev on HOST and ACC port representor + +Configure rules to send APF netdev on HOST to its associated port representors. + +Refer to above port mapping for APF netdev on HOST to ACC port representor mapping. Here, sample commands are shown for APF netdev on HOST. Configure similar mapping for remaining APF netdevs on HOST. + +Example: + +- APF netdev 1 on HOST has a VSI value 24, its corresponding port representor has VSI value 18 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for an overlay VF (VSI-24). Source port action can be any value. +# For simplicity add 16 to VSI ID. + p4rt-ctl add-entry br0 linux_networking_control.tx_source_port_v6 \ + "vmeta.common.vsi=24,zero_padding=0,action=linux_networking_control.set_source_port(40)" + +# Create a mapping between overlay VF (VSI-24/source port-40) and ACC port representor (VSI-18) + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=40,zero_padding=0,action=linux_networking_control.fwd_to_vsi(34)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=17,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(0)" + +# Create a mapping for traffic to flow between VSIs (VSI-24/source port-40) and (VSI-18) + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=18,target_vsi=24,action=linux_networking_control.fwd_to_vsi(40)" + + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=24,target_vsi=18,action=linux_networking_control.fwd_to_vsi(34)" +``` + +### Configure supporting p4 runtime tables + +For TCAM entry configure LPM LUT table + +```bash + p4rt-ctl add-entry br0 linux_networking_control.ipv6_lpm_root_lut \ + "user_meta.cmeta.bit32_zeros=0/255.255.255.255,priority=65535,action=linux_networking_control.ipv6_lpm_root_lut_action(0)" +``` + +Create a dummy LAG bypass table for all 8 hash indexes + +```bash + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=0,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=1,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=2,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=3,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=4,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=5,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=6,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=7,action=linux_networking_control.bypass" +``` + +### Create integration bridge and add ports to the bridge + +Create OvS bridge, VxLAN tunnel and assign overlay VFs port representor to individual bridges. +Reference provided for single overlay network, repeat similar steps for other VFs. + +Each bridge has: + +- One overlay PR which is Native-tagged to a specific VLAN +- One VxLAN port which is Native-untagged to the VLAN and with different remote TEP and VNI + +```bash +ovs-vsctl add-br br-1 +ovs-vsctl add-port br-1 enp0s1f0d1 tag=10 vlan_mode=native-tagged +ovs-vsctl add-port br-1 vxlan1 tag=10 vlan_mode=native-untagged -- set interface vxlan1 type=vxlan \ + options:local_ip=1000:1::1 options:remote_ip=1000:1::2 options:key=10 options:dst_port=4789 +``` + +Note: Here we are creating a VxLAN tunnel with VNI 10, you can create any VNI for tunneling. + +### Underlay configuration + +Create TEP termination bridge, add physical port's port representor and APF netdev port representor. Reference provided for underlay network, repeat similar steps for multiple underlay networks. + +```bash +ovs-vsctl add-br br-tun-1 +ovs-vsctl add-port br-tun-1 enp0s1f0d9 +ovs-vsctl add-port br-tun-1 enp0s1f0d10 +``` + +Configure underlay IP address on the termination bridge, and add change routes to reach remote IP. + +```bash +# On termiantion bridge, configure an IP to reach remote TEP, and modify route to include nexthop details. +ifconfig br-tun-1 up +ip -6 addr add 1000:1::1/64 dev br-tun-1 +ip -6 route del 1000:1::/64 dev br-tun-1 +ip -6 route add 1000:1::/64 dev br-tun-1 +ip -6 route change 1000:1::/64 via 1000:1::2 dev br-tun-1 +``` + +### Test the ping scenarios + +- Underlay ping6 +- Overlay ping: Ping6 between VM's on different hosts diff --git a/docs/apps/lnw/es2k/es2k-lnw-overlay-vms.md b/docs/apps/lnw/es2k/es2k-lnw-overlay-vms.md index 2c9bd2ca..8d3c7718 100644 --- a/docs/apps/lnw/es2k/es2k-lnw-overlay-vms.md +++ b/docs/apps/lnw/es2k/es2k-lnw-overlay-vms.md @@ -1,3 +1,23 @@ + + # Linux Networking with Overlay VMs This document explains how to run the Linux networking scenario on ES2K with 8 overlay VMs. @@ -14,12 +34,14 @@ Prerequisites: - Download `hw-p4-programs` TAR file specific to the build and extract it to get `fxp-net_linux-networking-v2` p4 artifacts. Go through `Limitations` specified in `README` and bringup the setup accordingly. - Follow steps mentioned in [Deploying P4 Programs for E2100](/guides/es2k/deploying-p4-programs) for bringing up IPU with a custom P4 package. Modify `load_custom_pkg.sh` with following parameters for linux_networking package: + ```text sed -i 's/sem_num_pages = 1;/sem_num_pages = 25;/g' $CP_INIT_CFG sed -i 's/lem_num_pages = 1;/lem_num_pages = 10;/g' $CP_INIT_CFG sed -i 's/acc_apf = 4;/acc_apf = 16;/g' $CP_INIT_CFG ``` + - Download `IPU_Documentation` TAR file specific to the build and refer to `Getting Started Guide` on how to install compatible `IDPF driver` on host. Once an IDPF driver is installed, bring up SRIOV VF by modifying the `sriov_numvfs` file present under one of the IDPF network devices. Example as below ```bash @@ -322,7 +344,7 @@ ovs-vsctl add-port br-1 vxlan1 tag=10 vlan_mode=native-untagged -- set interface options:local_ip=10.1.1.1 options:remote_ip=10.1.1.2 options:key=10 options:dst_port=4789 ``` -Note: Here we are creating a VxLAN tunnel with VNI 0, you can create any VNI for tunneling. +Note: Here we are creating a VxLAN tunnel with VNI 10, you can create any VNI for tunneling. ### Underlay configuration diff --git a/docs/apps/lnw/es2k/es2k-lnw-underlay-ecmp.md b/docs/apps/lnw/es2k/es2k-lnw-underlay-ecmp.md new file mode 100644 index 00000000..10ce9d63 --- /dev/null +++ b/docs/apps/lnw/es2k/es2k-lnw-underlay-ecmp.md @@ -0,0 +1,453 @@ + + +# Linux Networking with ECMP (ES2K) + +This document explains how to run the Linux networking scenario on ES2K. + +## Topology + +![Linux Networking Topology](es2k-lnw-topology.png) + +See [Linux Networking for ES2K](es2k-linux-networking.md) +for more details on this feature. + +Prerequisites: + +- Download `hw-p4-programs` TAR file specific to the build and extract it to get `fxp-net_linux-networking-v2` p4 artifacts. Go through `Limitations` specified in `README` and bring up the setup accordingly. +- Follow steps mentioned in [Deploying P4 Programs for E2100](/guides/es2k/deploying-p4-programs) for bringing up IPU with a custom P4 package. +Modify `load_custom_pkg.sh` with following parameters for linux_networking package: + +```text + sed -i 's/sem_num_pages = 1;/sem_num_pages = 25;/g' $CP_INIT_CFG + sed -i 's/lem_num_pages = 1;/lem_num_pages = 10;/g' $CP_INIT_CFG + sed -i 's/acc_apf = 4;/acc_apf = 8;/g' $CP_INIT_CFG +``` + +- Download `IPU_Documentation` TAR file specific to the build and refer to `Getting Started Guide` on how to install compatible `IDPF driver` on host. Once an IDPF driver is installed, bring up SRIOV VF by modifying the `sriov_numvfs` file present under one of the IDPF network devices. Example as below + + ```bash + echo 2 > /sys/class/net/ens802f0/device/sriov_numvfs + ``` + +Notes about topology: + +- VMs are spawned on top of each VF. Each VF will have a port representor in ACC. P4 runtime rules are configured to map VFs and their corresponding port representors. +- Each physical port will have a port representor in ACC. P4 runtime rules are configured to map VFs and their corresponding port representors. +- Each physical port will have a corresponding APF netdev on HOST. Create port representors in ACC for each HOST APF netdev. These APF netdevs on HOST will receive unknown traffic for applications to act on. +- All port representors should be part of an OvS bridge. Based on topology, this OvS bridge will just perform bridging or TEP termination on a bridge which is used to enable underlay connectivity for VxLAN traffic. +- OvS bridges and VxLAN ports are created using ovs-vsctl command (provided by the networking recipe). +- All port representors are attached to OvS bridge using ovs-vsctl command. +- This config has: + - 2 Overlay VFs + - 2 Port representors in ACC for the 2 Overlay VFs + - 2 physical ports + - 2 Port representors in ACC for the 2 physical ports + - 2 APF netdevs on HOST + - 2 Port representors in ACC for the 2 HOST APF netdevs + - TEP termination dummy interface + +System under test will have above topology running the networking recipe. Link Partner can have the networking recipe or legacy OvS or kernel VxLAN. Refer to the Limitations section in [Linux Networking for E2100](es2k-linux-networking.md) before setting up the topology. + +## Creating the topology + +Follow steps mentioned in [Running Infrap4d on Intel E2100](/guides/es2k/running-infrap4d.md) for starting `infrap4d` process and creating protobuf binary for `fxp-net_linux-networking-v2` p4 program. + +### Port mapping + +These VSI values can be checked with `/usr/bin/cli_client -q -c` command on IMC. This command provides VSI ID, Vport ID, and MAC addresses for all: + +- IDPF netdevs on ACC +- VFs on HOST +- IDPF netdevs on HOST (if IDPF driver loaded by you on HOST) +- Netdevs on IMC + +| Overlay VFs | Overlay VFs VSI ID | ACC port representor | ACC port representor VSI ID | +| ------------| ------------------ | -------------------- | --------------------------- | +| ens802f0v0 | (0x1b) 27 | enp0s1f0d1 | (0x09) 9 | +| ens802f0v1 | (0x1c) 28 | enp0s1f0d2 | (0x0a) 10 | + +| Physical port | Physical port ID | ACC Port presenter | ACC Port presenter VSI ID | +| ------------- | ----------------- | -------------------- | ------------------------- | +| Phy port 0 | (0x0) 0 | enp0s1f0d9 | (0x11) 17 | +| Phy port 1 | (0x1) 1 | enp0s1f0d11 | (0x13) 19 | + +| APF netdev | APF netdev VSI ID | ACC Port presenter | ACC Port presenter VSI ID | +| ------------- | ----------------- | -------------------- | ------------------------- | +| ens802f0d1 | (0x18) 24 | enp0s1f0d10 | (0x12) 18 | +| ens802f0d2 | (0x19) 25 | enp0s1f0d12 | (0x14) 20 | + +(NOTE: Port names and their VSI IDs may differ from setup to setup. Configure accordingly.) + +### Set the forwarding pipeline + +Once the application is started, set the forwarding pipeline config using +P4Runtime Client `p4rt-ctl` set-pipe command + +```bash +$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/fxp-net_linux-networking-v2.pb.bin \ + $OUTPUT_DIR/p4info.txt +``` + +Note: Assumes that `fxp-net_linux-networking-v2.pb.bin`, `p4info.txt` and other P4 artifacts, are created by following the steps in the previous section. + +### Configure VSI Group and add a netdev + +Add all ACC port representors to VSI group 1. VSI group 1 is dedicated for this configuration. +Execute the following devmem commands on IMC. + +```bash +# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which needs to map into vsig +devmem 0x20292002a0 64 0x8000050000000009 + +# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi +# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-1. +devmem 0x2029200388 64 0x1 + +# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which needs to map into vsig +devmem 0x20292002a0 64 0xA000050000000009 +``` + +Note: Here VSI 9 has been used as one of the ACC port representors and added to VSI group 1. For this use case, add all 6 IDPF interfaces created on ACC. Modify this VSI based on your configuration. + +### Start OvS as a separate process + +Enhanced OvS is used as a control plane for source MAC learning of overlay VMs. This OvS binary is available as part of ACC build and should be started as a separate process. + +```bash +export RUN_OVS=/opt/p4/p4-cp-nws + +rm -rf $RUN_OVS/etc/openvswitch +rm -rf $RUN_OVS/var/run/openvswitch +mkdir -p $RUN_OVS/etc/openvswitch/ +mkdir -p $RUN_OVS/var/run/openvswitch + + +ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \ + $RUN_OVS/share/openvswitch/vswitch.ovsschema + +ovsdb-server \ + --remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \ + --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ + --pidfile --detach + +ovs-vsctl --no-wait init + +mkdir -p /tmp/logs +ovs-vswitchd --pidfile --detach --mlockall \ + --log-file=/tmp/logs/ovs-vswitchd.log + +ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1 +ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1 + +ovs-vsctl show +``` + +### Create Overlay network + +Option 1: Create VFs on HOST and spawn VMs on top of those VFs. +Example: Below config is provided for one VM, and considering each VM is in one VLAN. Extend this to other VMs. + +```bash +# VM1 configuration +telnet +ip link add link name .10 type vlan id 10 +ip addr add 101.0.0.1/24 dev .10 +ifconfig up +ifconfig .10 up +``` + +Option 2: If we are unable to spawn VMs on top of the VFs, we can leverage kernel network namespaces. +Move each VF to a network namespace and assign IP addresses. +Example: Below config is provided for one overlay port, and tag each port in the namespace to a single VLAN. Extend this to other namespaces. + +```bash +ip netns add VM0 +ip link set netns VM0 +ip netns exec VM0 ip link add link name .10 type vlan id 10 +ip netns exec VM0 ip addr add 101.0.0.1/24 dev .10 +ip netns exec VM0 ifconfig up +ip netns exec VM0 ifconfig .10 up +``` + +### Configure rules for mapping between Overlay VF and ACC port representor + +Configure rules to send overlay packets from a VM to its associated port representors. + +Refer to above port mapping for overlay VF to ACC port representor mapping. Here, sample commands are shown for a single overlay network. Configure similar mapping for remaining VFs. + +Example: + +- Overlay VF1 has a VSI value 27, its corresponding port representor has VSI value 9 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for an overlay VF (VSI-27). Source port action can be any value. +# For simplicity add 16 to VSI ID. + p4rt-ctl add-entry br0 linux_networking_control.tx_source_port_v4 \ + "vmeta.common.vsi=27,zero_padding=0,action=linux_networking_control.set_source_port(43)" + +# Create a mapping between overlay VF (VSI-27/source port-43) and ACC port representor (VSI-9) + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=43,zero_padding=0,action=linux_networking_control.fwd_to_vsi(25)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=9,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(43)" + +# Create a mapping for traffic to flow between VSIs (VSI-27/source port-43) and (VSI-9) + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=9,target_vsi=27,action=linux_networking_control.fwd_to_vsi(43)" + + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=27,target_vsi=9,action=linux_networking_control.fwd_to_vsi(25)" + +``` + +### Configure rules for mapping between Physical port and ACC port representor + +Configure rules to send ingress packets from a physical port to its associated port representors. + +Refer to above port mapping for physical port to ACC port representor mapping. Here, sample commands are shown for a single physical port. Configure similar mapping for remaining physical ports. + +Example: + +- Physical port 0 port id is 0, its corresponding port representor has VSI value 17 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for a physical port (Phy port-0). Source port action can be any value. +# For simplicity consider the same value as phy port id. + p4rt-ctl add-entry br0 linux_networking_control.rx_source_port \ + "vmeta.common.port_id=0,zero_padding=0,action=linux_networking_control.set_source_port(0)" + +# Create a mapping between physical port (Phy port 0/src port 0) and ACC port representor (VSI-17) + p4rt-ctl add-entry br0 linux_networking_control.rx_phy_port_to_pr_map \ + "vmeta.common.port_id=0,zero_padding=0,action=linux_networking_control.fwd_to_vsi(33)" + + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=0,zero_padding=0,action=linux_networking_control.fwd_to_vsi(33)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=17,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(0)" +``` + +### Configure rules for mapping between APF netdev on HOST and ACC port representor + +Configure rules to send APF netdev on HOST to its associated port representors. + +Refer to above port mapping for APF netdev on HOST to ACC port representor mapping. Here, sample commands are shown for APF netdev on HOST. Configure similar mapping for remaining APF netdevs on HOST. + +Example: + +- APF netdev 1 on HOST has a VSI value 24, its corresponding port representor has VSI value 18 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for an overlay VF (VSI-24). Source port action can be any value. +# For simplicity add 16 to VSI ID. + p4rt-ctl add-entry br0 linux_networking_control.tx_source_port_v4 \ + "vmeta.common.vsi=24,zero_padding=0,action=linux_networking_control.set_source_port(40)" + + +# Create a mapping between overlay VF (VSI-24/source port-40) and ACC port representor (VSI-18) + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=40,zero_padding=0,action=linux_networking_control.fwd_to_vsi(34)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=17,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(0)" + +# Create a mapping for traffic to flow between VSIs (VSI-24/source port-40) and (VSI-18) + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=18,target_vsi=24,action=linux_networking_control.fwd_to_vsi(40)" + + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=24,target_vsi=18,action=linux_networking_control.fwd_to_vsi(34)" +``` + +### Configure supporting p4 runtime tables + +For TCAM entry configure LPM LUT table + +```bash + p4rt-ctl add-entry br0 linux_networking_control.ipv4_lpm_root_lut \ + "user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=65535,action=linux_networking_control.ipv4_lpm_root_lut_action(0)" +``` + +Create a dummy LAG bypass table for all 8 hash indexes + +```bash + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=0,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=1,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=2,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=3,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=4,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=5,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=6,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=7,action=linux_networking_control.bypass" +``` + +### Create integration bridge and add ports to the bridge + +Create OvS bridge, VxLAN tunnel and assign overlay VFs port representor to individual bridges. +Reference provided for single overlay network, repeat similar steps for other VFs. + +Each bridge has: + +- One overlay PR which is Native-tagged to a specific VLAN +- One VxLAN port which is Native-untagged to the VLAN and with different remote TEP and VNI + +```bash +ovs-vsctl add-br br-1 +ovs-vsctl add-port br-1 enp0s1f0d1 tag=10 vlan_mode=native-tagged +ovs-vsctl add-port br-1 vxlan1 tag=10 vlan_mode=native-untagged -- set interface vxlan1 type=vxlan \ + options:local_ip=10.1.1.1 options:remote_ip=20.1.1.1 options:key=10 options:dst_port=4789 +``` + +Note: Here we are creating a VxLAN tunnel with VNI 10, you can create any VNI for tunneling. + +### Underlay configuration + +Create a dummy interface which is used for TEP termination and create underlay bridge, add physical port's port representor and APF netdev port representor. +Reference provided for two multipath underlay network, repeat similar steps for multiple underlay networks. + +Option 1: Statically configure underlay ECMP routes to reach link partner (or) \ +Option 2: Use BGP protocol with FRR, for ECMP route redistribution to reach link partner. + +Below configuration assumes + +- `br-tun-1` OvS bridge with physical port representor `enp0s1f0d9` as first ECMP member +- `br-tun-2` OvS bridge with physical port representor `enp0s1f0d11` as second ECMP member + +```bash +ovs-vsctl add-br br-tun-1 +ovs-vsctl add-port br-tun-1 enp0s1f0d9 +ovs-vsctl add-port br-tun-1 enp0s1f0d10 + +ovs-vsctl add-br br-tun-2 +ovs-vsctl add-port br-tun-2 enp0s1f0d11 +ovs-vsctl add-port br-tun-2 enp0s1f0d12 +``` + +Option 1: Static configuration + +Configure underlay IP address on the TEP termination port, route to reach remote IP is on termination bridge, and add change routes to reach remote IP. + +```bash +# Create a dummy port and add TEP local IP +ip link add dev TEP0 type dummy +ifconfig TEP0 10.1.1.1/24 up + +# On tunnel bridge, configure an IP in the same network of remote underlay +ifconfig br-tun-1 50.1.1.1/24 up +ifconfig br-tun-2 60.1.1.1/24 up + +# Add multipath route to reach remote TEP +ip route add 20.1.1.1 nexthop via 50.1.1.2 dev br-tun-1 \ + weight 1 nexthop via 60.1.1.2 dev br-tun-2 weight 1 +ip route show +``` + +Sample link partner underlay configuration. + +```bash +ip link add dev TEP0 type dummy +ifconfig TEP0 20.1.1.1/24 up +ifconfig 50.1.1.2/24 up +ifconfig 60.1.1.2/24 up + +ip route add 10.1.1.1 nexthop via 50.1.1.1 dev \ + weight 1 nexthop via 60.1.1.1 dev weight 1 +ip route show + +``` + +Option 2: FRR running configuration + +```bash +# +# + +interface TEP0 +ip address 10.1.1.1/24 +exit +! +interface br-tun-1 +ip address 50.1.1.1/24 +exit +! +interface br-tun-2 +ip address 60.1.1.1/24 +exit +! +router bgp 65000 +bgp router-id 10.1.1.1 +neighbor 50.1.1.2 remote-as 65000 +neighbor 60.1.1.2 remote-as 65000 +! +address-family ipv4 unicast + network 10.1.1.0/24 +exit-address-family +``` + +Sample link partner FRR VTYSH configuration. + +```bash +interface TEP0 +ip address 20.1.1.1/24 +exit +! +interface +ip address 50.1.1.2/24 +exit +! +interface +ip address 60.1.1.2/24 +exit +! +router bgp 65000 +bgp router-id 20.1.1.1 +neighbor 50.1.1.1 remote-as 65000 +neighbor 60.1.1.1 remote-as 65000 +! +address-family ipv4 unicast + network 20.1.1.0/24 +exit-address-family +``` + +### Test the ping scenarios + +- Underlay ping from each tunnel bridge (`br-tun-1` and `br-tun-2`) +- Ping to remote tunnel IP from TEP interface +- Overlay ping: Ping between VMs on different hosts diff --git a/docs/apps/lnw/es2k/es2k-lnw-underlay-frr.md b/docs/apps/lnw/es2k/es2k-lnw-underlay-frr.md new file mode 100644 index 00000000..7c82c776 --- /dev/null +++ b/docs/apps/lnw/es2k/es2k-lnw-underlay-frr.md @@ -0,0 +1,403 @@ + + +# Linux Networking with FRR (ES2K) + +This document explains how to run the Linux networking scenario on ES2K. + +## Topology + +![Linux Networking Topology](es2k-lnw-topology.png) + +See [Linux Networking for ES2K](es2k-linux-networking.md) +for more details on this feature. + +Prerequisites: + +- Download `hw-p4-programs` TAR file specific to the build and extract it to get `fxp-net_linux-networking-v2` p4 artifacts. Go through `Limitations` specified in `README` and bring up the setup accordingly. +- Follow steps mentioned in [Deploying P4 Programs for E2100](/guides/es2k/deploying-p4-programs) for bringing up IPU with a custom P4 package. +Modify `load_custom_pkg.sh` with following parameters for linux_networking package: + +```text + sed -i 's/sem_num_pages = 1;/sem_num_pages = 25;/g' $CP_INIT_CFG + sed -i 's/lem_num_pages = 1;/lem_num_pages = 10;/g' $CP_INIT_CFG + sed -i 's/acc_apf = 4;/acc_apf = 8;/g' $CP_INIT_CFG +``` + +- Download `IPU_Documentation` TAR file specific to the build and refer to `Getting Started Guide` on how to install compatible `IDPF driver` on host. Once an IDPF driver is installed, bring up SRIOV VF by modifying the `sriov_numvfs` file present under one of the IDPF network devices. Example as below + + ```bash + echo 1 > /sys/class/net/ens802f0/device/sriov_numvfs + ``` + +Notes about topology: + +- VMs are spawned on top of each VF. Each VF will have a port representor in ACC. P4 runtime rules are configured to map VFs and their corresponding port representors. +- Each physical port will have a port representor in ACC. P4 runtime rules are configured to map VFs and their corresponding port representors. +- Each physical port will have a corresponding APF netdev on HOST. Create port representors in ACC for each HOST APF netdev. These APF netdevs on HOST will receive unknown traffic for applications to act on. +- All port representors should be part of an OvS bridge. Based on topology, this OvS bridge will just perform bridging or TEP termination on a bridge which is used to enable underlay connectivity for VxLAN traffic. +- OvS bridges and VxLAN ports are created using ovs-vsctl command (provided by the networking recipe). +- All port representors are attached to OvS bridge using ovs-vsctl command. +- This config has: + - 1 Overlay VF + - 1 Port representors in ACC for the 2 Overlay VF + - 2 physical ports + - 2 Port representors in ACC for the 2 physical ports + - 2 APF netdevs on HOST + - 2 Port representors in ACC for the 2 HOST APF netdevs + - TEP termination dummy interface + +System under test will have above topology running the networking recipe. Link Partner can have the networking recipe or legacy OvS or kernel VxLAN. Refer to the Limitations section in [Linux Networking for E2100](es2k-linux-networking.md) before setting up the topology. + +## Creating the topology + +Follow steps mentioned in [Running Infrap4d on Intel E2100](/guides/es2k/running-infrap4d.md) for starting `infrap4d` process and creating protobuf binary for `fxp-net_linux-networking-v2` p4 program. + +### Port mapping + +These VSI values can be checked with `/usr/bin/cli_client -q -c` command on IMC. This command provides VSI ID, Vport ID, and MAC addresses for all: + +- IDPF netdevs on ACC +- VFs on HOST +- IDPF netdevs on HOST (if IDPF driver loaded by you on HOST) +- Netdevs on IMC + +| Overlay VFs | Overlay VFs VSI ID | ACC port representor | ACC port representor VSI ID | +| ------------| ------------------ | -------------------- | --------------------------- | +| ens802f0v0 | (0x1b) 27 | enp0s1f0d1 | (0x09) 9 | + +| Physical port | Physical port ID | ACC Port presenter | ACC Port presenter VSI ID | +| ------------- | ----------------- | -------------------- | ------------------------- | +| Phy port 0 | (0x0) 0 | enp0s1f0d9 | (0x11) 17 | +| Phy port 1 | (0x1) 1 | enp0s1f0d11 | (0x13) 19 | + +| APF netdev | APF netdev VSI ID | ACC Port presenter | ACC Port presenter VSI ID | +| ------------- | ----------------- | -------------------- | ------------------------- | +| ens802f0d1 | (0x18) 24 | enp0s1f0d10 | (0x12) 18 | +| ens802f0d2 | (0x19) 25 | enp0s1f0d12 | (0x14) 20 | + +(NOTE: Port names and their VSI IDs may differ from setup to setup. Configure accordingly.) + +### Set the forwarding pipeline + +Once the application is started, set the forwarding pipeline config using +P4Runtime Client `p4rt-ctl` set-pipe command + +```bash +$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/fxp-net_linux-networking-v2.pb.bin \ + $OUTPUT_DIR/p4info.txt +``` + +Note: Assumes that `fxp-net_linux-networking-v2.pb.bin`, `p4info.txt` and other P4 artifacts, are created by following the steps in the previous section. + +### Configure VSI Group and add a netdev + +Add all ACC port representors to VSI group 1. VSI group 1 is dedicated for this configuration, +execute below devmem commands on IMC. + +```bash +# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which needs to map into vsig +devmem 0x20292002a0 64 0x8000050000000009 + +# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi +# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-1. +devmem 0x2029200388 64 0x1 + +# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which needs to map into vsig +devmem 0x20292002a0 64 0xA000050000000009 +``` + +Note: Here VSI 9 has been used as one of the ACC port representors and added to VSI group 1. For this use case, add all 5 IDPF interfaces created on ACC. Modify this VSI based on your configuration. + +### Start OvS as a separate process + +Enhanced OvS is used as a control plane for source MAC learning of overlay VMs. This OvS binary is available as part of ACC build and should be started as a separate process. + +```bash +export RUN_OVS=/opt/p4/p4-cp-nws + +rm -rf $RUN_OVS/etc/openvswitch +rm -rf $RUN_OVS/var/run/openvswitch +mkdir -p $RUN_OVS/etc/openvswitch/ +mkdir -p $RUN_OVS/var/run/openvswitch + + +ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \ + $RUN_OVS/share/openvswitch/vswitch.ovsschema + +ovsdb-server \ + --remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \ + --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ + --pidfile --detach + +ovs-vsctl --no-wait init + +mkdir -p /tmp/logs +ovs-vswitchd --pidfile --detach --mlockall \ + --log-file=/tmp/logs/ovs-vswitchd.log + +ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1 +ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1 + +ovs-vsctl show +``` + +### Create Overlay network + +Option 1: Create VFs on HOST and spawn VMs on top of those VFs. +Example: Below config is provided for one VM, and considering each VM is in one VLAN. Extend this to other VMs. + +```bash +# VM1 configuration +telnet +ip link add link name .10 type vlan id 10 +ip addr add 101.0.0.1/24 dev .10 +ifconfig up +ifconfig .10 up +``` + +Option 2: If we are unable to spawn VMs on top of the VFs, we can leverage kernel network namespaces. +Move each VF to a network namespace and assign IP addresses. +Example: Below config is provided for one overlay port, and tag each port in the namespace to a single VLAN. Extend this to other namespaces. + +```bash +ip netns add VM0 +ip link set netns VM0 +ip netns exec VM0 ip link add link name .10 type vlan id 10 +ip netns exec VM0 ip addr add 101.0.0.1/24 dev .10 +ip netns exec VM0 ifconfig up +ip netns exec VM0 ifconfig .10 up +``` + +### Configure rules for mapping between Overlay VF and ACC port representor + +Configure rules to send overlay packets from a VM to its respective port representors. + +Refer to above port mapping for overlay VF to ACC port representor mapping. Here, sample commands are shown for a single overlay network. Configure similar mapping for remaining VFs. + +Example: + +- Overlay VF1 has a VSI value 27, its corresponding port representor has VSI value 9 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for an overlay VF (VSI-27). Source port action can be any value. +# For simplicity add 16 to VSI ID. + p4rt-ctl add-entry br0 linux_networking_control.tx_source_port_v4 \ + "vmeta.common.vsi=27,zero_padding=0,action=linux_networking_control.set_source_port(43)" + +# Create a mapping between overlay VF (VSI-27/source port-43) and ACC port representor (VSI-9) + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=43,zero_padding=0,action=linux_networking_control.fwd_to_vsi(25)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=9,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(43)" + +# Create a mapping for traffic to flow between VSIs (VSI-27/source port-43) and (VSI-9) + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=9,target_vsi=27,action=linux_networking_control.fwd_to_vsi(43)" + + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=27,target_vsi=9,action=linux_networking_control.fwd_to_vsi(25)" + +``` + +### Configure rules for mapping between Physical port and ACC port representor + +Configure rules to send ingress packets from a physical port to its associated port representors. + +Refer to above port mapping for physical port to ACC port representor mapping. Here, sample commands are shown for a single physical port. Configure similar mapping for remaining physical ports. + +Example: + +- Physical port 0 port id is 0, its corresponding port representor has VSI value 17 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for a physical port (Phy port-0). Source port action can be any value. +# For simplicity consider the same value as phy port id. + p4rt-ctl add-entry br0 linux_networking_control.rx_source_port \ + "vmeta.common.port_id=0,zero_padding=0,action=linux_networking_control.set_source_port(0)" + +# Create a mapping between physical port (Phy port 0/src port 0) and ACC port representor (VSI-17) + p4rt-ctl add-entry br0 linux_networking_control.rx_phy_port_to_pr_map \ + "vmeta.common.port_id=0,zero_padding=0,action=linux_networking_control.fwd_to_vsi(33)" + + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=0,zero_padding=0,action=linux_networking_control.fwd_to_vsi(33)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=17,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(0)" +``` + +### Configure rules for mapping between APF netdev on HOST and ACC port representor + +Configure rules to send APF netdev on HOST to its associated port representors. + +Refer to above port mapping for APF netdev on HOST to ACC port representor mapping. Here, sample commands are shown for APF netdev on HOST. Configure similar mapping for remaining APF netdevs on HOST. + +Example: + +- APF netdev 1 on HOST has a VSI value 24, its corresponding port representor has VSI value 18 +- If a VSI is used as an action, add an offset of 16 to the VSI value + +```bash +# Create a source port for an overlay VF (VSI-24). Source port action can be any value. +# For simplicity add 16 to VSI ID. + p4rt-ctl add-entry br0 linux_networking_control.tx_source_port_v4 \ + "vmeta.common.vsi=24,zero_padding=0,action=linux_networking_control.set_source_port(40)" + + +# Create a mapping between overlay VF (VSI-24/source port-40) and ACC port representor (VSI-18) + p4rt-ctl add-entry br0 linux_networking_control.source_port_to_pr_map \ + "user_meta.cmeta.source_port=40,zero_padding=0,action=linux_networking_control.fwd_to_vsi(34)" + + p4rt-ctl add-entry br0 linux_networking_control.tx_acc_vsi \ + "vmeta.common.vsi=17,zero_padding=0,action=linux_networking_control.l2_fwd_and_bypass_bridge(0)" + +# Create a mapping for traffic to flow between VSIs (VSI-24/source port-40) and (VSI-18) + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=18,target_vsi=24,action=linux_networking_control.fwd_to_vsi(40)" + + p4rt-ctl add-entry br0 linux_networking_control.vsi_to_vsi_loopback \ + "vmeta.common.vsi=24,target_vsi=18,action=linux_networking_control.fwd_to_vsi(34)" +``` + +### Configure supporting p4 runtime tables + +For TCAM entry configure LPM LUT table + +```bash + p4rt-ctl add-entry br0 linux_networking_control.ipv4_lpm_root_lut \ + "user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=65535,action=linux_networking_control.ipv4_lpm_root_lut_action(0)" +``` + +Create a dummy LAG bypass table for all 8 hash indexes + +```bash + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=0,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=1,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=2,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=3,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=4,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=5,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=6,action=linux_networking_control.bypass" + + p4rt-ctl add-entry br0 linux_networking_control.tx_lag_table \ + "user_meta.cmeta.lag_group_id=0,hash=7,action=linux_networking_control.bypass" +``` + +### Create integration bridge and add ports to the bridge + +Create OvS bridge, VxLAN tunnel and assign overlay VFs port representor to individual bridges. +Reference provided for single overlay network, repeat similar steps for other VFs. + +Each bridge has: + +- One overlay PR which is Native-tagged to a specific VLAN +- One VxLAN port which is Native-untagged to the VLAN and with different remote TEP and VNI + +```bash +ovs-vsctl add-br br-1 +ovs-vsctl add-port br-1 enp0s1f0d1 tag=10 vlan_mode=native-tagged +ovs-vsctl add-port br-1 vxlan1 tag=10 vlan_mode=native-untagged -- set interface vxlan1 type=vxlan \ + options:local_ip=10.1.1.1 options:remote_ip=20.1.1.1 options:key=10 options:dst_port=4789 +``` + +Note: Here we are creating a VxLAN tunnel with VNI 10, you can create any VNI for tunneling. + +### Underlay configuration + +Create a dummy interface which is used for TEP termination and create underlay bridge, add physical port's port representor and APF netdev port representor. Use BGP protocol with FRR, for route redistribution. +Reference provided for single underlay network, repeat similar steps for multiple underlay networks. + +Below configuration assumes + +- `br-tun-1` OvS bridge with physical port representor `enp0s1f0d9` as port to reach remote underlay network + +```bash +ip link add dev TEP0 type dummy + +ovs-vsctl add-br br-tun-1 +ovs-vsctl add-port br-tun-1 enp0s1f0d9 +ovs-vsctl add-port br-tun-1 enp0s1f0d10 +``` + +FRR running configuration + +```bash +# +# + +interface TEP0 +ip address 10.1.1.1/24 +exit +! +interface br-tun-1 +ip address 70.1.1.1/24 +exit +! +router bgp 65000 +bgp router-id 10.1.1.1 +neighbor 70.1.1.1 remote-as 65000 +! +address-family ipv4 unicast +network 10.1.1.0/24 +exit-address-family +``` + +Sample link partner FRR VTYSH configuration. + +```bash +interface TEP0 +ip address 20.1.1.1/24 +exit +! +interface +ip address 70.1.1.2/24 +exit +! +router bgp 65000 +bgp router-id 20.1.1.1 +neighbor 70.1.1.2 remote-as 65000 +! +address-family ipv4 unicast + network 20.1.1.0/24 +exit-address-family +``` + +### Test the ping scenarios + +- Underlay ping from tunnel bridge (`br-tun-1`) +- Ping to remote tunnel IP from TEP interface +- Overlay ping: Ping between VMs on different hosts diff --git a/docs/apps/lnw/lnw-index.rst b/docs/apps/lnw/lnw-index.rst index 68392f8a..5d28ab91 100644 --- a/docs/apps/lnw/lnw-index.rst +++ b/docs/apps/lnw/lnw-index.rst @@ -1,4 +1,4 @@ -.. Copyright 2023 Intel Corporation +.. Copyright 2023-2024 Intel Corporation SPDX-License-Identifier: Apache-2.0 ================ @@ -22,8 +22,8 @@ ES2K es2k/es2k-linux-networking es2k/es2k-lnw-overlay-vms - es2k/es2k-linux-networking-ipv6 - es2k/es2k-linux-networking-ecmp - es2k/es2k-linux-networking-frr + es2k/es2k-lnw-overlay-ipv6 + es2k/es2k-lnw-underlay-ecmp + es2k/es2k-lnw-underlay-frr es2k/es2k-linux-networking-lag