Skip to content

Release Schedule

Jump to bottom
George Keishing edited this page Mar 26, 2024 · 47 revisions

June 28 2024 (upcoming)

Robot Framework 7.0: Update code syntax overall in the repository.

Refer: https://github.com/openbmc/openbmc-test-automation/wiki/Notification

Dec 08 2023

Release Note: v4.0-stable

The last supported level for Legacy REST, Angular JS and tools used in the test.

https://github.com/openbmc/openbmc-test-automation/releases/tag/v4.0-stable

As communicated earlier, the expected changes on the latest master will follow starting next year but in phases gracefully.

Remove tools and code supports which are no longer relevant (refer above release note if you still need it ) Update/revisit documentation README and other information as part of the activity to maintain the relevant data Consolidation / Grouping of test suites/files to minimize clutter test directories across the repository Disable Legacy Rest and default to Redfish ONLY and remove flows which we had to support both in code Continue to update codes are per new changes in the Robot and Python versions available. For more information, you can reach us out on OpenBMC discord https://discord.gg/8rvZJQbE #test-automation (https://discord.gg/4G3vd5rP )

Aug 05 2022

Upcoming release due to robotframework-requests upgrade

READ: https://github.com/openbmc/openbmc-test-automation/wiki/Notification

May 21 2021

Community 2.10 (Hardknott) Tagging build testing https://jenkins.openbmc.org/job/release-tag/9/

yocto release refer https://wiki.yoctoproject.org/wiki/Releases

OpenBMC test repository will also tag a branch and release once this activity is completed.

You can reach out to us on Discord https://discord.gg/69Km47zH98 #test-automation

Status: COMPLETED

Overall, the driver is stable. The certificate failures could potentially block LDAP secure functions.

Note: Witherspoon system have LDAP services removed in the driver, so we couldn't test it. If anyone in community check on it on their box.

Build: https://jenkins.openbmc.org/job/release-tag/9/
BMC Driver Version: 2.10.0-rc1-dirty    
(Linux witherspoon 5.10.36-fa5949e #1 Thu May 20 16:34:31 UTC 2021 armv6l GNU/Linux)
System: Witherspoon

Defect(s) List:

Test Report

1. Code update from recent upstream master to candidate tag 2.10.0-rc1-dirty: PASS

Test case: 1
Failed   : 0

Redfish Bmc Code Update :: Update the BMC code on a target BMC via... | [32mPASS[0m |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================

2. HW CI test(Refer test_lists/HW_CI) : PASS

Test Case: 20
Failed   : 0

==============================================================================
Ipmi & Tests & Redfish                                                | PASS |
20 critical tests, 20 passed, 0 failed
20 tests total, 20 passed, 0 failed
==============================================================================

3. Boot Test with Partition: PASS

Test Case: 13
Failed   : 0

Boot Type                                Total Pass Fail
---------------------------------------- ----- ---- ----
Redfish Power On                             2    2    0
IPMI Power On                                1    1    0
Redfish Power Off                            2    2    0
IPMI Power Off                               1    1    0
IPMI Power Soft                              1    1    0
OBMC Reboot (run)                            1    1    0
REST OBMC Reboot (run)                       1    1    0
OBMC Reboot (off)                            1    1    0
REST OBMC Reboot (off)                       1    1    0
IPMI Power Cycle                             1    1    0
IPMI Power Reset                             1    1    0
Auto Reboot                                  1    1    0
RF SYS GracefulRestart                       1    1    0
========================================================
Totals                                      15   15    0

4. Redfish validator: PASS

Test Case: 1
Failed   : 0

Redfish Service Validator :: Test BMC using https://github.com/DMT... | [32mPASS[0m |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================

5. Regression Test bucket (Refer test_lists/witherspoon/regression_bucket_master): FAIL

Test Case: 72
Failed   : 14

==============================================================================
Redfish & Ipmi                                                        | FAIL |
72 critical tests, 58 passed, 14 failed
72 tests total, 58 passed, 14 failed
==============================================================================

Note: Bug opened for the failure. Not a blocker since LDAP services are removed from Witherspoon, but should be working in others.

- Configure_Hostname_And_Verify      [New interface to use, need to check automation]
- Add_Valid_IPv4_Addres_And_Verify   [Working fine, need to check automation]
- Verify_Client_Certificate_Replace
- Verify_CA_Certificate_Replace
- Verify_Client_Certificate_Install
- Verify_CA_Certificate_Install
- Verify_Maximum_CA_Certificate_Install
- Verify_Error_While_Uploading_Same_CA_Certificate
- Verify_CSR_Generation_For_Client_Certificate
- Verify_Expired_Certificate_Install
- Verify_Not_Yet_Valid_Certificate_Install
- Verify_Not_Yet_Valid_Certificate_Replace
- Verify_Certificates_Location_Via_Redfish
- Login_And_Verify_HTTP_Response_Header

bug:

6. IPMI Test bucket: FAIL

Test Case: 86
Failed   : 21

==============================================================================
Ipmi                                                                  | FAIL |
86 critical tests, 65 passed, 21 failed
86 tests total, 65 passed, 21 failed
==============================================================================

Note: Bug opened for the failure. Not a blocker.

- Test_FRU_Device_Name
- Test_Activate_Payload
- Test_Deactivate_Payload
- Test_Get_Payload_Instance_Info
- Test_TPM_Enable_SDR_Info
- Test_Sensor_Threshold_Via_IPMI
- Set_SOL_Enabled
- Set_SOL_Disabled
- Set_Valid_SOL_Privilege_Level
- Set_Invalid_SOL_Privilege_Level
- Set_Invalid_SOL_Retry_Count
- Set_Invalid_SOL_Retry_Interval
- Set_Invalid_SOL_Character_Accumulate_Level
- Set_Invalid_SOL_Character_Send_Threshold
- Verify_SOL_During_Boot
- Verify_Deactivate_Non_Existing_SOL
- Set_Valid_SOL_Retry_Count
- Set_Valid_SOL_Retry_Interval
- Set_Valid_SOL_Character_Accumulate_Level
- Set_Valid_SOL_Character_Send_Threshold
- Verify_Continuous_Activation_And_Deactivation_Of_SOL

bug:

7. BMC Reset test (Loop 20 iteration at Host standby and check for errors): PASS

Test Case: 1
Failed   : 0

Boot Type                                Total Pass Fail
---------------------------------------- ----- ---- ----
Redfish OBMC Reboot (off)                   20   20    0
========================================================
Totals                                      20   20    0


Run Multiple BMC Reset Via Redfish :: Execute multiple reboots via... | [32mPASS[0m |
------------------------------------------------------------------------------
Test Bmc Reset Loop :: Power cycle loop. This is to test where net... | [32mPASS[0m |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================

8. SNMP Test: FAIL

Test Case: 13
Failed   : 6

Configure SNMP Manager With Less Octet IP And Verify :: Configure ... | FAIL |
Allowing the configuration of an invalid SNMP.: 200 != 400
------------------------------------------------------------------------------
Test Bmc Snmp Config :: This testing requires special setup where ... | FAIL |
13 critical tests, 7 passed, 6 failed
13 tests total, 7 passed, 6 failed
==============================================================================

Note: Bug opened for the failure. Not a blocker but negative use cases test failure.

- Configure_SNMP_Manager_With_Less_Octet_IP_And_Verify
- Configure_SNMP_Manager_On_BMC_With_Out_Of_Range_IP_And_Verify
- Configure_SNMP_Manager_On_BMC_With_Empty_Port_And_Verify
- Configure_SNMP_Manager_On_BMC_With_Negative_Port_And_Verify
- Configure_SNMP_Manager_On_BMC_With_Alpha_Port_And_Verify
- Configure_SNMP_Manager_On_BMC_With_Out_Of_Range_Port_And_Verify

bug:

9. BMC web GUI Test: PASS

Manually checked pages and it's operation and they look intact and working.

June 9 2020

Community 2.8 Tagging build testing report https://lists.ozlabs.org/pipermail/openbmc/2020-June/021961.html

Nov 20 2019

From Kurt Taylor IRC note:

There isn't an official yocto 2.8 (they are jumping to 3.0) and we won't release openbmc until May 2020 to line up with > the 3.1 yocto release since November was too soon after the August release. > https://github.com/openbmc/openbmc/wiki/Releases

Note: So we won't be releasing 2.8 bellow schedule to align with OpenBMC release.

Nov 08 2019

Community,

Its time for the 2.8 release of OpenBMC test automation which is scheduled on Dec 6th, 2019.

The testing is progressing on the latest master and we are accepting patches only to get a stable release on the following test

  • Local User (Redfish / IPMI )
  • LDAP
  • Firmware update
  • Certificate
  • Network
  • BMC web GUI
  • Others for back compatible

Note: There is no support planned for earlier versions of Python (<3.x) and Robot CLI package releases after OpenBMC test automation 2.8 release.

Clone this wiki locally