Skip to content

Commit

Permalink
Added variable tables
Browse files Browse the repository at this point in the history
Signed-off-by: Vipul Gupta (@vipulgupta2048) <[email protected]>
  • Loading branch information
vipulgupta2048 authored and rcooke-warwick committed Jul 17, 2023
1 parent 3425985 commit 8cf6d4b
Showing 1 changed file with 44 additions and 22 deletions.
66 changes: 44 additions & 22 deletions documentation/quickstart-autokit.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,31 +16,46 @@ To use the autokit with the leviathan framework, your autokit must be running a
3. [Provision](https://docs.balena.io/learn/deploy/deployment/#overview) your autokit to this fleet. Depending on the device-type of your autokit host, the process may differ. Ensure that you provision the device using the image configured to use **ethernet only**.
4. When the device is successfully provisioned on balenaCloud, make sure to enable the public URL for your device.
5. Add the appropriate environment variables to the device. You may have to adjust these depending on the autokit setup and the device under test. The main ones, relevant to the leviathan-worker are:
1. `TESTBOT_DUT_TYPE` must be set to the device type slug of the DUT. See supported devices [here] - although if your device is not there, you can pick another DUT with the closest procedure and use that instead. For example for a simple sd card boot DUT, use `raspberrypi4-64`, and for a flasher image use `intel-nuc`.
2. `WORKER_TYPE` must be set to `autokit`
3. The appropriate [autokit setup variables](https://github.com/balena-os/leviathan-worker/blob/master/lib/workers/autokit.ts#L17), if any of the hardware differs from the default. Cross reference these strings with the [autokit feature implementations](https://github.com/balena-io-hardware/autokit-interface-sw/tree/master/lib/features).
4. Any other relevent autokit variables must be set, if the default values aren't appropriate
1. `WIFI_IF` - the name of the primary wifi interface of the autokit host (default `wlan0`)
2. `WIRED_IF` - the name of the USB-ethernet interface of the autokit host - this can be found by going into the autokit host over ssh and finding it (default `eth0`)
3. `DEV_SERIAL` - the name of the USB-serial interface of the autokit (default `/dev/ttyUSB0`)
5. Ensure that the `worker` container on your autokit host has started. In the dashboard logs window, you should see the message `worker setup completed`. If the container is not starting, the most likely cause is either hardware is not plugged in, or env vars are not set up correctly.

| Variable Name | Description | Required |
| --- | --- | --- |
| `TESTBOT_DUT_TYPE` | Set to the device type slug of the DUT from the [supported devices](https://github.com/balena-io-hardware/autokit-interface-sw/tree/master/lib/flashing/devices) listed here. If your DUT is not listed, then proceed to choose the device type that resembles the flashing procedure of the DUT. For example, a flashing a sd card and booting process resembles the process of `raspberrypi4-64`. Similarly, for a flasher image use `intel-nuc`. | True |
| `WORKER_TYPE` | Set to the type of worker you intend to use with Leviathan. In this case, it's an autokit. Hence, set to `autokit`. | True |
| Autokit Setup Variables | Refer to the [autokit setup variables](https://github.com/balena-os/leviathan-worker/blob/master/lib/workers/autokit.ts#L17) and ensure they are appropriately configured if any hardware differs from the default. Cross-reference these strings with the [autokit feature implementations](https://github.com/balena-io-hardware/autokit-interface-sw/tree/master/lib/features) for more information. |

Any other relevent autokit variables must be set, if the default values aren't appropriate

| Variable Name | Description | Default | Required |
| --- | --- | --- | --- |
| `WIFI_IF` | The name of the primary WiFi interface of the autokit host. | `wlan0` | No |
| `WIRED_IF` | The name of the USB-Ethernet interface of the autokit host. This can be found by accessing the autokit host over SSH. | `eth0` | No |
| `DEV_SERIAL` | The name of the USB-serial interface of the autokit. | `/dev/ttyUSB0` | No |

6. Ensure that the `worker` container on your autokit host has started. In the dashboard logs window, you should see the message `worker setup completed`. If the container is not starting, the most likely cause is either hardware is not plugged in, or env vars are not set up correctly.
7. Lastly, add a device tag for your autokit in balenaCloud. The tag will be for the key named "DUT" with the value being the slug of the device under test. For example, the following as a balenaCloud device tag.

```
DUT: raspberrypi3
```

Future work will simplify this process to avoid manual configuration of environment variables where possible.

## Start your first test run

For your first test run, we will be running the [e2e test suite](https://github.com/balena-os/leviathan/tree/master/suites/e2e).
For your first test run, we will be running the [e2e test suite](https://github.com/balena-os/leviathan/tree/master/suites/e2e). This is a basic testing suite to test your worker configuration and if the setup is correct. Each Levaithan test run needs the following to start testing that the user has to provide:

A balenaOS image that is used to flash and provision the device under test will be needed for the test. You can download an unmanaged genericx86-64-ext balenaOS image from [balena.io/os](https://www.balena.io/os/#download) and place it in the workspace folder. Leviathan support OS images uncompressed or compressed in `.gz` or `.zip` format.
1. **The test suite you want to run**: [e2e test suite](https://github.com/balena-os/leviathan/tree/master/suites/e2e) (already provided)
2. **A balenaOS image used to flash, provision and test the DUT**. The test will automatically download an OS image this time. So no need to provide your own image.
3. **A test configuration using the `config.js` file**: Let's create one.

### Build your `config.js` file

The `config.js` file is the master configuration file for your test run. Leviathan runs and configures your device under test (DUT) accordingly with the settings provided in the `config.js` file. To know more about each property, refer to the {@page Config.js Reference | Config.js reference}.
The `config.js` file is the master configuration file for your test run. Leviathan runs and configures your device under test (DUT) according to the settings configured under `config.js` file. To know more about each property, refer to the {@page Config.js Reference | Config.js reference}.

To get started,
Create your own config.js file

- Navigate to the `workspace` directory in leviathan.
- Create a `config.js` file in the `workspace` directory and paste the contents below:
- Create a `config.js` file in the `workspace` directory and paste the contents below.

```js
module.exports = {
Expand All @@ -57,7 +72,7 @@ module.exports = {
debug: {
unstable: ["Kill the device under test"],
}
image: `${__dirname}/path/to/image`,
image: false,
workers: ['<Public device URL of your autokit>'],
}
```
Expand All @@ -77,17 +92,19 @@ BALENA_ARCH=amd64
BALENACLOUD_APP_NAME=<app-name>
```

### Start the run

To start the test run, navigate to the root of the leviathan directory and run the following command:
To start the test run, navigate to the root of the Leviathan directory and run the following command:

```
make test
```

The logs will start streaming on the terminal for the test run. You will observe the related dependencies being built (This is a one time process). After then, the test will start. Wait for the test scenario to finish and check the device logs on the dashboard in the meantime.
This will trigger a build of client and core services using docker-compose and begin the test. The logs by various componenet will start streaming on the terminal. Wait for the test scenario to finish and check the device logs on the dashboard in the meantime.

A successful run of the e2e test suite without any errors makes sure that your autokit worker is set up correctly and can be used for further tests.


## Let's run some "real" tests

We will start with a test run of the [balenaOS unmanaged testing suite](https://github.com/balena-os/meta-balena/tree/master/tests/suites). To get the tests, clone the meta-balena repository. The OS tests are located in the `tests/` directory.
Expand All @@ -98,25 +115,30 @@ We will start with a test run of the [balenaOS unmanaged testing suite](https://
```js
module.exports = {
deviceType: "raspberrypi3",
suite: `${__dirname}/../suites/os`,
suite: `${__dirname}/../suites/os`, // this path is relative to the workspace directory
config: {
networkWired: false,
networkWireless: true,
downloadVersion: 'latest',
balenaApiKey: process.env.BALENACLOUD_API_KEY,
balenaApiUrl: 'balena-cloud.com',
organization: 'BALENACLOUD_ORG_GOES_HERE',
organization: process.env.BALENACLOUD_ORG,
},
image: `${__dirname}/path/to/image`,
workers: {
balenaApplication: 'balena/testbot-personal',
balenaApplication: 'your-fleet-slug',
apiKey: process.env.BALENACLOUD_API_KEY
}
};
```
- Add the slug of the device under test to the autokit. In the Admin Panel > Device Variables add the variable TESTBOT_DUT_TYPE:\<DUT_SLUG\>, and a autokit Tag named "DUT" that contains the slug of the device under test.
- Run `make test` in the root of the project and watch the logs. The logs will start streaming on the terminal for the test run.
- At the end of the run, reports and logs for the test run will be stored in `workspace/reports` directory.

- This time you can provide your own OS image to test. You can download an unmanaged genericx86-64-ext balenaOS image from [balena.io/os](https://www.balena.io/os/#download) and place it in the workspace folder if you need to provide one. Change the value of the `image` property to the path of the image you downloaded.

### Start the OS test

Run `make test` in the root of the project and watch the logs.

The logs will start streaming on the terminal for the test run. At the end of the run, reports and logs for the test run will be stored in `workspace/reports` directory.

That's the end of the quick start guide, you successfully setup your autokit worker and ran your first test suite.

Expand Down

0 comments on commit 8cf6d4b

Please sign in to comment.