Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable testing without autokit on physical DUTs #1060

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

rcooke-warwick
Copy link
Contributor

@rcooke-warwick rcooke-warwick commented Oct 11, 2023

  • there's been interest in running tests on a phsyical DUT, but no access to autokit hardware
  • this limits some functionality - mainly concerned with flashing, but also hdmi, bluetooth, and some networking manipulation, and GPIO
  • however many of the OS tests actually are fine - so this PR will have a collection of changes to allow for running tests on a physical DUT, from your laptop

How to use:
set your workspace/config.js to:

module.exports = [{
    deviceType: '<DEVICE_TYPE_SLUG',
    suite: `${__dirname}/../suites/os`,
    config: {
        networkWired: false,
        networkWireless: false,
        downloadVersion: 'latest',
        balenaApiKey: '<API_KEY>',
        balenaApiUrl: 'balena-cloud.com',
        organization: '<YOUR_BALENA_CLOUD_USERNAME',
        dutIp: '<LOCAL_IP_OF_YOUR_DUT',
        dutUuid: '<UUID_OF_YOUR_DUT',
	manual: true
    },
    debug: {
        unstable: ["SSH authentication test"],
    },
    image: false, 
    workers: ['http://worker'],
}]

Test suite specific caveats:
OS suite: Must put device in local mode first (via dashboard). Also manually enable persistent logging in the dashboard too. If your device has wifi, provision it to use your wifi (same network as laptop).
Cloud suite: Must not be in local mode, "SSH authentication test" must be set as unstable
Hup suite: Annoying in general - make sure you provision your DUT into the OLD OS version you want to hup from. Then in the image property put the path to the <hostapp>.docker hostapp image, instead of false

You must run this branch of the tests from meta balena: https://github.com/balena-os/meta-balena/tree/ryan/manual_tests

You must provision your DUT manually to a balena cloud fleet, with no release, and your organization must have access to it. The DUT must be on the same network as your laptop. Ensure the DUT is in development mode first. Then run:

make local-test

If there are ssh authentication errors, then the organization you put in config.js doesn't have access to the DUT you provisioned.

Other considerations:

 sshConfig: {
                host: 'ssh.balena-staging-devices.com',
                port: '22',
    }

should be added in the above config.js, right after manual:true, only if testing in staging.

The device should have both local mode enabled in the dashboard, and developmentMode:true in it's config.json, for the OS suite.

If the OS test suite need to be re-run, /mnt/boot/system-proxy should be deleted, the device should then be re-booted and local-mode should be toggled to off and back on prior to restarting the test suite. Depending on the leftovers from a failed or interrupted test suite, re-flashing may be necessary.

On particular devices that also have a p2p-dev-wlP1p1s0 interface for example, the connectivity test should be modified to

 let connection = adaptor === 'wireless' ? 'wifi' : 'Wired';
                                                return this.worker.executeCommandInHostOS(
                                                        `nmcli d  | grep '${connection}' | grep connected | awk '{print $1}' | head -n1`,

For the cloud suite, the following line should be changed to use "that" instead of "this": https://github.com/balena-os/meta-balena/blob/ryan/manual_tests/tests/suites/cloud/suite.js#L248

Change-type: patch
Signed-off-by: Ryan Cooke <[email protected]>
Change-type: patch
Signed-off-by: Ryan Cooke <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant