Add support for RobotFramework based automated integration testing (#204

) Use Edk2 and Project Mu for testing since those are two public repositories that use Pytools
tianocore · Jun 18, 2020 · 1ee7fcc · 1ee7fcc
1 parent 01be590
commit 1ee7fcc
Show file tree

Hide file tree

Showing 9 changed files with 1,141 additions and 1 deletion.
diff --git a/.cspell.json b/.cspell.json
@@ -113,6 +113,8 @@
         "smbios",
         "Contoso",
         "MSFT",
-        "codecov"
+        "codecov",
+        "qemu",
+        "ovmf"
     ]
 }
diff --git a/integration_test/.gitignore b/integration_test/.gitignore
@@ -0,0 +1,2 @@
+/test
+/report
diff --git a/integration_test/ReadMe.md b/integration_test/ReadMe.md
@@ -0,0 +1,77 @@
+# Integration testing for Edk2 Pytool Extensions
+
+This describes what is needed to run integration testing on pytool extensions.  This testing will use known compatible and public open source repositories for validation.  This set of automated tests will run common user level operations and validate that pytools works as expected. Some basic examples are:
+
+* Build Edk2 OvmfPkg Platform CI
+* Run Core Ci on Edk2 MdeModulePkg
+* After committing a change in the repo use pr_eval to confirm the correct packages need testing.
+
+## Prerequisites
+
+* Standard Edk2 Prerequisites
+  * git cli
+  * Python 3.8.x with support for Pip and virtual environment
+  * VS 2019 (Windows Only)
+  * GCC (Linux Only)
+* Additional requirements
+  * robot framework
+  * QEMU (on your path)
+
+## Process
+
+1. Create new virtual environment
+2. Activate virtual environment
+3. Clone edk2-pytool-extension
+4. Check out desired version/branch and cd to root (folder where *setup.py* is located)
+5. Pip Install the local version by running `pip install -e .` in the root directory
+6. Pip install remaining dependencies by running `pip install --upgrade -r integration_test/pip-requirements.txt`
+7. cd to *integration_test* directory (folder where *this* document is located)
+8. Run robot `python -m robot.run -v TEST_OUTPUT_BASE:test -d report <additional flags here> <robot file to run or directory>`
+   1. use `.` to run all test cases in all robot files in the current directory
+   2. use `edk2_stuart_pr_eval.robot` robot file for testing stuart pr eval using edk2 repo
+   3. use `edk2_core_ci.robot` robot file for running parts of the core ci tests on edk2
+   4. use `edk2_platform_ci.robot` robot file for running parts of the platform ci tests on edk2
+
+## Additional useful CLI flags
+
+1. Only run test cases with a given tag
+   1. `--include CoreCI` to run only the CoreCI tests
+   2. `--include PlatformCI` to run only the Platform CI tests
+   3. `--include PrEval` to run only the pr_eval tests
+   4. `--include Edk2` to run only on edk2 repo
+   5. `--include ProjectMu` to run only on Project Mu basecore repo
+2. More Debug info in log
+   1. `-L TRACE` to run with most detailed info
+   2. `-L DEBUG` to turn on debug level.  This is less than trace
+   3. `-L INFO` this is the default level.
+3. Set more variables
+   1. `-v <var name>:<var value>` set a variable for your robot file
+
+## Lessons learned/Tips
+
+### Robot hangs for seemingly unknown reason
+
+It is well known that robot will hang if you "log too much" from Run Process.  For example running git clone can sometimes
+create a large log.  So for those commands which can create a large log redirect stdout and stderr to a file.  To do this
+add additional parameters to your "Run Process" cmd like so `stdout=stdout.txt  stderr=stderr.txt`.  For an example in action
+look at the keywords for all stuart commands.
+
+### Run a single test so you can collect the logs from the file system
+
+Robot framework supports running a single test by providing the test name.  The robot.run command should
+add `-t "the test case name here"`.  This has been useful to run a single failing test on your filesystem and then
+collect the stuart log to debug problems.
+
+## Helpful Robot Links
+
+https://robotframework.org is the main site.  Lots of good info here.  My most commonly used pages are:
+
+* UserGuide: https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html
+* Operating System Library: https://robotframework.org/robotframework/latest/libraries/OperatingSystem.html
+* BuiltIn Library: https://robotframework.org/robotframework/latest/libraries/BuiltIn.html
+* String Library: https://robotframework.org/robotframework/latest/libraries/String.html
+
+## Copyright
+
+Copyright (c) Microsoft Corporation.
+SPDX-License-Identifier: BSD-2-Clause-Patent
diff --git a/integration_test/Shared_Keywords.robot b/integration_test/Shared_Keywords.robot
@@ -0,0 +1,178 @@
+*** Settings ***
+Documentation     A shared set of common keywords for stuart operations and git operations
+#
+# Copyright (c), Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+
+Library  Process
+Library  OperatingSystem
+
+# Suite Setup
+
+*** Variables ***
+
+#Test output location
+${TEST_OUTPUT}          ${TEST_OUTPUT_BASE}
+
+*** Keywords ***
+
+### list comparison helper ###
+Confirm same contents
+    [Arguments]  ${actual}  ${expected}
+    @{epkgs}=  Split String  ${expected}  ,
+    @{apkgs}=  Split String  ${actual}  ,
+
+    FOR  ${a}  IN  @{apkgs}
+      Should Contain  ${expected}  ${a}
+    END
+
+    FOR  ${b}  IN  @{epkgs}
+      Should Contain  ${actual}  ${b}
+    END
+
+### Git operations ###
+Clone the git repo
+    [Arguments]    ${git_url}   ${ws_name}
+
+    Log To console    cloning ${git_url} to ${TEST_OUTPUT}
+    ${result}=  Run Process       git.exe   clone   ${git_url}   ${ws_name}
+    ...  cwd=${TEST_OUTPUT}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many  stdout: ${result.stdout}  stderr: ${result.stderr}
+
+    ${result}=  Run Process  git  fetch  --all  --prune
+    ...  cwd=${TEST_OUTPUT}${/}${ws_name}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many  stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Reset git repo to main branch
+    [Arguments]     ${ws}  ${main_branch_name}
+
+    # checkout remote tag for origin/master
+    ${result}=  Run Process  git  checkout  origin/${main_branch_name}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+    # clean non ignored files quietly to avoid log overflow
+    ${result}=  Run Process  git  clean  -qfd  cwd=${ws}
+    Log Many  stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+    # reset to restore files
+    ${result}=  Run Process  git  reset  --hard
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many  stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Make new branch
+    [Arguments]    ${name}  ${ws}
+    ${result}=   Run Process    git  checkout  -b  ${name}
+    ...  cwd=${ws}  shell=True
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Delete branch
+    [Arguments]    ${name}  ${to_branch}  ${ws}
+    Run Keyword  Switch branch  ${to_branch}  ${ws}
+    ${result}=   Run Process    git  branch  -D  ${name}
+    ...  cwd=${ws}  shell=True
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Switch branch
+    [Arguments]    ${name}  ${ws}
+    ${result}=   Run Process    git  checkout  ${name}
+    ...  cwd=${ws}  shell=True
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Stage changed file
+    [Arguments]    ${file_path}  ${ws}
+    ${result}=   Run Process    git  add  ${file_path}
+    ...  cwd=${ws}  shell=True
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Commit changes
+    [Arguments]    ${msg}  ${ws}
+    ${result}=   Run Process    git  commit  -m  ${msg}
+    ...  cwd=${ws}  shell=True
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+### Stuart operations ###
+Stuart setup
+    [Arguments]  ${setting_file}  ${arch}  ${target}  ${packages}  ${tool_chain}  ${ws}
+    Log to console  Stuart Setup
+    ${result}=   Run Process    stuart_setup
+    ...  -c  ${setting_file}  -a  ${arch}  TOOL_CHAIN_TAG\=${tool_chain}  -t  ${target}  -p  ${packages}  TARGET\=${target}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Stuart ci setup
+    [Arguments]  ${setting_file}  ${arch}  ${target}  ${packages}  ${tool_chain}  ${ws}
+    Log to console  Stuart CI Setup
+    ${result}=   Run Process    stuart_ci_setup
+    ...  -c  ${setting_file}  -a  ${arch}  TOOL_CHAIN_TAG\=${tool_chain}  -t  ${target}  -p  ${packages}  TARGET\=${target}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Stuart update
+    [Arguments]  ${setting_file}  ${arch}  ${target}  ${packages}  ${tool_chain}  ${ws}
+    Log to console  Stuart Update
+    ${result}=   Run Process    stuart_update
+    ...  -c  ${setting_file}  -a  ${arch}  TOOL_CHAIN_TAG\=${tool_chain}  -t  ${target}  -p  ${packages}  TARGET\=${target}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Stuart platform build
+    [Arguments]  ${setting_file}  ${arch}  ${target}  ${tool_chain}  ${ws}
+    Log to console  Stuart Build
+    ${result}=   Run Process    stuart_build
+    ...  -c  ${setting_file}  -a  ${arch}  TOOL_CHAIN_TAG\=${tool_chain}  TARGET\=${target}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Stuart platform run
+    [Arguments]  ${setting_file}  ${arch}  ${target}  ${tool_chain}  ${addtional_flags}  ${ws}
+    Log to console  Stuart Build Run
+    ${result}=   Run Process    stuart_build
+    ...  -c  ${setting_file}  -a  ${arch}  TOOL_CHAIN_TAG\=${tool_chain}  TARGET\=${target}  --FlashOnly  ${addtional_flags}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Stuart CI build
+    [Arguments]  ${setting_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws}
+    Log to console  Stuart CI Build
+    ${result}=   Run Process    stuart_ci_build
+    ...  -c  ${setting_file}  -a  ${archs}  -t  ${targets}  -p  ${packages}  TOOL_CHAIN_TAG\=${tool_chain}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+
+Stuart pr evaluation
+    [Documentation]  Run Pr Eval
+    [Arguments]  ${setting_file}  ${packages}  ${base_ref}  ${other_build_flags}  ${ws}
+
+    ${result}=   Run Process    stuart_pr_eval
+    ...  -c  ${setting_file}  -p  ${packages}  --pr-target  origin/${base_ref}
+    ...  --output-csv-format-string  {pkgcsv}
+    ...  cwd=${ws}  stdout=stdout.txt  stderr=stderr.txt
+    Log Many  stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
+    Return From Keyword    ${result.stdout}
+
+### Edk2 BaseTools Build operations ###
+Build BaseTools
+    [Arguments]  ${tool_chain}  ${ws}
+    Log to console  Compile basetools
+    ${result}=   Run Process    python
+    ...  BaseTools/Edk2ToolsBuild.py  -t  ${tool_chain}
+    ...  cwd=${ws}  shell=True  stdout=stdout.txt  stderr=stderr.txt
+    Log Many	stdout: ${result.stdout}  stderr: ${result.stderr}
+    Should Be Equal As Integers  ${result.rc}  0
diff --git a/integration_test/edk2_core_ci.robot b/integration_test/edk2_core_ci.robot
@@ -0,0 +1,105 @@
+*** Settings ***
+Documentation     A test suite to test Core CI on edk2 repo
+#
+# Copyright (c), Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+
+Library  Process
+Library  OperatingSystem
+Library  String
+
+Resource  Shared_Keywords.robot
+
+Suite Setup  One time setup  ${repo_url}  ${ws_dir}
+
+# Suite Setup
+
+*** Variables ***
+${repo_url}           https://github.com/tianocore/edk2.git
+${master_branch}      master
+${ws_dir}             edk2
+${ci_file}            .pytool/CISettings.py
+${ws_root}            ${TEST_OUTPUT}${/}${ws_dir}
+${tool_chain}         VS2019
+
+
+*** Keywords ***
+One time setup
+    [Arguments]  ${url}  ${folder}
+    ## Dump pip versions
+    ${result}=   Run Process    python  -m  pip   list  shell=True
+    Log  ${result.stdout}
+
+    ## Make output directory if doesn't already exist
+    Create Directory  ${TEST_OUTPUT}
+
+    ## Clone repo
+    Run Keyword  Clone the git repo  ${url}  ${folder}
+
+
+*** Test Cases ***
+
+Run Edk2 MdePkg CoreCI Debug
+    [Documentation]  This Test will run X64 DEBUG build of Core CI on the MdePkg
+    [Tags]           CoreCI  Windows  VS2019  Compile  Edk2
+
+    ${archs}=            Set Variable    X64
+    ${targets}=          Set Variable    DEBUG
+    ${packages}=         Set Variable    MdePkg
+
+    # make sure on master
+    Reset git repo to main branch  ${ws_root}  ${master_branch}
+
+    Stuart setup           ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Stuart update          ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Build BaseTools        ${tool_chain}  ${ws_root}
+    Stuart CI build        ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+
+Run Edk2 SecurityPkg CoreCI Release
+    [Documentation]  This Test will run IA32 RELEASE build of Core CI on the SecurityPkg
+    [Tags]           CoreCI  Windows  VS2019  Compile  Edk2
+
+    ${archs}=            Set Variable    IA32
+    ${targets}=          Set Variable    RELEASE
+    ${packages}=         Set Variable    SecurityPkg
+
+    # make sure on master
+    Reset git repo to main branch  ${ws_root}  ${master_branch}
+
+    Stuart setup           ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Stuart update          ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Build BaseTools        ${tool_chain}  ${ws_root}
+    Stuart CI build        ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+
+Run Edk2 UefiCpuPkg CoreCI for No-Target
+    [Documentation]  This Test will run NO-TARGET Core CI test on the UefiCpuPkg
+    [Tags]           CoreCI  Windows  VS2019  NO-TARGET  Edk2
+
+    ${archs}=            Set Variable    X64,IA32,AARCH64,ARM
+    ${targets}=          Set Variable    NO-TARGET
+    ${packages}=         Set Variable    UefiCpuPkg
+
+    # make sure on master
+    Reset git repo to main branch  ${ws_root}  ${master_branch}
+
+    Stuart setup           ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Stuart update          ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Build BaseTools        ${tool_chain}  ${ws_root}
+    Stuart CI build        ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+
+Run Edk2 MdeModulePkg CoreCI for NOOPT and HostTest
+    [Documentation]  This Test will run NOOPT Core CI which includes Host-Tests on the MdeModulePkg
+    [Tags]           CoreCI  Windows  VS2019  NOOPT  HOST-TEST  Edk2
+
+    ${archs}=            Set Variable    X64
+    ${targets}=          Set Variable    NOOPT
+    ${packages}=         Set Variable    MdeModulePkg
+
+    # make sure on master
+    Reset git repo to main branch  ${ws_root}  ${master_branch}
+
+    Stuart setup           ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Stuart update          ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+    Build BaseTools        ${tool_chain}  ${ws_root}
+    Stuart CI build        ${ci_file}  ${archs}  ${targets}  ${packages}  ${tool_chain}  ${ws_root}
+