Skip to content

Hardware Tests

Jump to bottom Edit New page
Christos Nikolaou edited this page Aug 22, 2017 · 20 revisions

This section will mainly focus on the system requirements, setting-up and running hardware tests for reference projects.

Section 1 - System Requirements

Hardware requirements

The following are the components required for doing the Hardware tests:

  1. NetFPGA-SUME card
  2. 10G NIC card
  3. SFP+ transceivers
  4. fiber optical cables (loopback and crossover)

While doing the Hardware Tests users have to be very careful in getting the connections right. Else the test won't pass and will report an error. Following guidelines are suggested but many more can be included.

  1. Ensure that the SFP+ transceivers are properly connected to the SFP+ cage of the NetFPGA board.
  2. Check if the fiber connections are intact. Take proper precautions and handle the fiber with care, as any damage to them can result in failed tests.

Software requirements

  1. Make sure you clone the latest version of the NetFPGA package. Please ensure that you have the necessary packages installed. The current Hardware testing infrastructure is Python based.
git clone https://github.com/NetFPGA/NetFPGA-SUME-live.git

Note: Update your settings with the ~/tools/settings.sh file provided in your clone. You will also need to disable the following: (Avahi daemon and ipv6) to stop the interference of traffic from the above application with the test traffic generated through scapy. To set up the environment variables, source both relevant settings files, e.g.:

source ~/NetFPGA-SUME-live/tools/settings.sh
source /opt/Xilinx/Vivado/2014.4/settings64.sh
  1. Do a make cores.
[root@nf-test109 ~]# cd $SUME_FOLDER 
[root@nf-test109 NetFPGA-SUME-live]# make
  1. Program the FPGA with the right bitfile using XMD (Xilinx tool).
  • If you want to run the Hardware tests with the pre-existing bitfile provided in the base repo:
[root@nf-test109 reference_nic]# cd bitfiles
[root@nf-test109 bitfiles]# xmd

On the xmd console, use fpga -f reference_nic.bit to program the FPGA with the bitfile. Reboot the machine.

  • If you want to create your own bitfile and run the Hardware tests:
[root@nf-test109 reference_nic]# make
[root@nf-test109 reference_nic]# cd bitfiles
[root@nf-test109 bitfiles]# xmd

On the xmd console, use fpga -f reference_nic.bit to program the FPGA with the bitfile. Reboot the machine.

  1. Check if the bit file is loaded using the following command.
[root@nf-test109 scripts]# lspci –vxx | grep Xilinx

If you don't see the Xilinx device, you need to reprogram the FPGA and reboot as mentioned earlier.

  1. Build the driver for the NetFPGA SUME board.
[root@nf-test109 ~]# cd $DRIVER_FOLDER
[root@nf-test109 sume_riffa_v1_0_0]# make all
[root@nf-test109 sume_riffa_v1_0_0]# make install
[root@nf-test109 sume_riffa_v1_0_0]# modprobe sume_riffa

Then run ifconfig -a to check if you are able to see the 'nfX' interfaces.

Section 2 - Test directory file structure

  1. All the tests for the project should be inside the specific project folder. For example all the tests inside the reference_nic project can be seen by going inside the test folder.
 cd NetFPGA-SUME-live/projects/reference_nic/test

The following shows the tree structure of the test folder.

 test
    ├── both_loopback_minsize
    │   └── run.py
    ├── both_loopback_maxsize
    │   └── run.py
    ├── connections
    │   └── conn
    └── global
        └── setup
  1. connections is the connections folder, where connections file for the project can be placed. A connection file specifies how the network interfaces are physically connected. The connection file is formatted with one connection per line, where the connection is specified by nfX:ethY, denoting that the interface nfX should be physically connected to ethY.nfX are the interfaces of NetFPGA and ethY are the interfaces of 10G NIC.

ie. For hw tests in reference_nic, the nfX are connected back to nfX interfaces (loopback tests) and the corresponding conn file will have the following configuration.

nf0:nf0
nf1:nf1
nf2:nf2
nf3:nf3

For hw tests in reference_switch, reference_switch_lite, reference_router, the nfX are connnected to ethX interfaces, the corresponding conn file will have the following configuration.

nf0:eth2
nf1:eth3
nf2:
nf3:

Note: nf3 is the interface of NetFPGA-SUME card closest to the PCIe and nf0 is the farthest from the PCIe. In this example, eth2 is the interface of 10G-NIC card closest to the PCIe and eth3 is the farthest. Please ensure that you change this according to your system configuration.

You can check the interfaces' configuration of your system using the following script: 

~/NetFPGA-SUME-live/tools/scripts/interface_config/nf_interface_config.py
This is used to create customized conn and setup configuration files based on your interfaces. The generated conn and setup files need to be placed in the projects's test folder, replacing the existing conn and setup files.

  1. global is the folder which contains the setup script. The script is used to configure the IP address for the interfaces while running the hardware tests. The IP's are hard-coded in this file. But the user has to remain concious that they will have to change the file if their interfaces are different(having a different X in ethX) or if they want to configure different IP address for the interfaces.

  2. All the tests for the project have a specific naming convention. Test directories should be named both_<major>_<minor> if they can be run in both simulation and hardware, hw_<major>_<minor> if they can be run only on hardware and sim_<major>_<minor> if they can be run only as simulation. Neither major or minor can have underscores in their name, nor can they be left blank. Inside each specific test folder, there is a run.py.

  3. run.py is the executable scripts which runs the test. The run.py calls other functions in the NFTest library. It is important to know what arguments needs to be passed to the run.py along with the tests, so that the system can perform the tests as simulation only, hardware only or both.

Section 3- Running the tests

  1. Make sure the ~/tools/settings.sh file is updated in your system.

  2. Check if the variables like SUME_FOLDER, NF_DESIGN_DIR are set in the right path. Try to do an echo $SUME_FOLDER , and echo $NF_DESIGN_DIR and see if they refer to the place you want.

  3. Running the test

The top level file nf_test.py file can be found inside NetFPGA-SUME-live/tools/scripts.

Tests are run using the nf_test.py command followed by the arguments indicating if it is a HW or sim test and what is the specific test that we would like to run. So when running the test, test mode should be specified (sim or hw). Other parameters include

  • -- major <string>
  • -- minor <string>

For instance : ./nf_test.py hw --major loopback --minor minsize . For a complete list of arguments type nf_test.py --help

You can also use the register_read script to view the contents of the register and how it is affected by running a specific test.

  1. We have sume_reg.c which has the following functions regread, regwrite and regread_expect. This can be found inside NetFPGA-SUME-live/lib/sw/std/hwtestlib. There is a Makefile which creates a shared library (libsume.so) of sume_reg.c. This shared library is used in the python infrastructure for register operations. Following are the use case for the functions in python inside the run.py 
rres1= nftest_regread_expect(SUME_INPUT_ARBITER_PKTIN_0(), 0x28)
rres2= nftest_regread_expect(SUME_OUTPUT_PORT_LOOKUP_PKTIN_0(), 0x28)
  1. In the current implementation if the Hardware Test fails, we can track down if the errors were due to:
  • packet loss and barriers related errors
  • reg read errors

Section 4- POINTS TO REMEMBER

  1. Set the design environment variables in ~/tools/settings.sh file. By default variables assume that the NetFPGA package is installed in root. Following are the default contents of the settings.sh
export SUME_FOLDER=${HOME}/NetFPGA-SUME-live
export XILINX_PATH=/opt/Xilinx/Vivado/2014.4
export NF_PROJECT_NAME=reference_nic
export PROJECTS=${SUME_FOLDER}/projects
export DEV_PROJECTS=${SUME_FOLDER}/dev-projects
export IP_FOLDER=${SUME_FOLDER}/lib/hw/std/cores
export CONSTRAINTS=${SUME_FOLDER}/lib/hw/std/constraints
export XILINX_IP_FOLDER=${SUME_FOLDER}/lib/hw/xilinx/cores
export NF_DESIGN_DIR=${SUME_FOLDER}/projects/${NF_PROJECT_NAME}
export NF_WORK_DIR=/tmp/${USER}
export PYTHONPATH=.:${SUME_FOLDER}/tools/scripts/:${NF_DESIGN_DIR}/lib/Python:${SUME_FOLDER}/tools/scripts/NFTest
export DRIVER_NAME=sume_riffa_v1_0_0
export DRIVER_FOLDER=${SUME_FOLDER}/lib/sw/std/driver/${DRIVER_NAME}
export APPS_FOLDER=${SUME_FOLDER}/lib/sw/std/apps/${DRIVER_NAME}
  1. Errors like:
ERROR: [Common 17-69] Command failed: Run 'impl_1' has not been launched. Unable to open

may be caused due to missing licenses in Vivado.

  1. Ensure the bitfile is loaded properly.

  2. Modify the conn file based on your system interfaces.

  3. Modify setup file is available to configure the right IP for your interfaces. Following are the default IP address and the interface names. In my work station there are two RJ-45 interfaces (eth0 and eth1), followed by the interfaces for 10G NICs (eth2 and eth3) that is the reason for the following interface names. If you have different interface names, instead of changing the interface names for all projects, you can edit the interface names in /etc/udev/rules.d/70-persistent-net.rules. Please refer to Ubuntu documentation for more details.

"eth2","192.168.100.1"
"eth3","192.168.101.1"
"nf0","192.168.200.1"
"nf1","192.168.201.1"
"nf2","192.168.202.1"
"nf3","192.168.203.1"
  1. Ensure that the sume driver is loaded. For the NetFPGA, the generic driver can be found inside the library directory (inside NetFPGA-SUME-live/lib/sw/std/driver/sume_riffa_v1_0_0/). Do a make all and you should be able to see the sume_riffa.ko. Install and load the driver by running the following commands
cd $DRIVER_FOLDER
sudo make install
sudo modprobe sume_riffa

  1. Check if your hardware connections are intact.

  2. Check if the setup and the run.py files are in the proper modes (executables). If not when you run the Hardware tests, it will give you a "permission denied" error and the "interface not UP" errors.

  3. When you run the nf_test.py file, you call the make reg on the project if it has register infrastructure. This will create the necessary python reg_defines file that will be used for the test infrastructure to access registers. When you do make reg the xparameters.h file will give a register map of the design (contains the base address and the offset). This xparameters.h is fed as input to the python file xparameters2regdefines.py, that generates reg_defines.h file which has the complete address map (baseaddress+offset for each register). This reg_defines.h is parsed by the python_parser.py to create reg_defines_{project_name}.py, which is a project specific python library containing the register map. So that the Hardware tests can reference the register by name instead of hard-coding address values.

Add a custom footer
Add a custom sidebar
Clone this wiki locally