Components Test Management

The extension provides testing for Python code through one of the supported frameworks. This involves a variety of capabilities in addition to just running tests. The following frameworks are supported:

• stdlib unittest
• pytest
• nose2

Functionality

operations

• install selected test framework if not installed
• test discovery
• run all/selected tests & suites
• debug all/selected tests & suites
• breakpoints while running tests
• show (high level) test results

Note that the active Python interpreter is used for all testing operations.

resources

• (somewhat) granular test framework config settings
• dedicated output log

commands

• "Python: Discover Unit Tests"
• "Python: Run All Unit Tests"
• "Python: Run Unit Test Method..."
• "Python: View Unit Test Output"

UI

• status bar element
  • shows test results
  • triggers quick pick for testing-related operations
• pop-up (& quick pick) to configure test framework
• code lens on tests and suites:
  • run/debug
  • results
• output panel: "Python Test Log"
• TBD: Test Explorer activity (top-level VSC element)

config settings

• "python.unitTest.cwd"
• "python.unitTest.debugPort"
• "python.unitTest.nosetestArgs"
• "python.unitTest.nosetestsEnabled"
• "python.unitTest.nosetestPath"
• "python.unitTest.promptToConfigure"
• "python.unitTest.pyTestArgs"
• "python.unitTest.pyTestEnabled"
• "python.unitTest.pyTestPath"
• "python.unitTest.unittestArgs"
• "python.unitTest.unittestEnabled"
• "python.unitTest.autoTestDiscoverOnSaveEnabled"

Extension Code

codebase structure

All test-related code is contained entirely under a single directory, src/client/unittests/:

src/client/unittest
├── codeLenses
├── common
│   ├── managers
│   ├── services
│   └── testVisitors
├── display
├── nosetest
│   └── services
├── pytest
│   └── services
└── unittest
    └── services

files: external API

src/client/unittest
├── serviceRegistry.ts
└── types.ts

files: top-level code

src/client/unittest
├── common
│   ├── argumentsHelper.ts
│   ├── constants.ts
│   ├── debugLauncher.ts
│   ├── managers
│   │   ├── baseTestManager.ts
│   │   └── testConfigurationManager.ts
│   ├── runner.ts
│   ├── services
│   │   ├── configSettingService.ts
│   │   ├── storageService.ts
│   │   ├── testManagerService.ts
│   │   ├── testResultsService.ts
│   │   ├── unitTestDiagnosticService.ts
│   │   └── workspaceTestManagerService.ts
│   ├── testUtils.ts
│   ├── testVisitors
│   │   ├── flatteningVisitor.ts
│   │   ├── folderGenerationVisitor.ts
│   │   └── resultResetVisitor.ts
│   ├── types.ts
│   └── xUnitParser.ts
└── main.ts

files: configuration:

src/client/unittest
├── configurationFactory.ts
├── configuration.ts
├── nosetest
│   └── testConfigurationManager.ts
├── pytest
│   └── testConfigurationManager.ts
└── unittest
    └── testConfigurationManager.ts

files: testing frameworks:

src/client/unittest
├── nosetest
│   ├── main.ts
│   ├── runner.ts
│   └── services
│       ├── argsService.ts
│       ├── discoveryService.ts
│       └── parserService.ts
├── pytest
│   ├── main.ts
│   ├── runner.ts
│   └── services
│       ├── argsService.ts
│       ├── discoveryService.ts
│       ├── parserService.ts
│       └── testMessageService.ts
└── unittest
    ├── helper.ts
    ├── main.ts
    ├── runner.ts
    ├── services
    │   ├── argsService.ts
    │   ├── discoveryService.ts
    │   └── parserService.ts
    └── socketServer.ts

files: UI features

src/client/unittest
├── codeLenses
│   ├── main.ts
│   └── testFiles.ts
└── display
    ├── main.ts
    └── picker.ts

components

• config
• test discovery
• code lens
• picker
• runner

Adapter Script

• source: pythonFiles/testing_tools/adapter/__main__.py
• invocation:
  • [PYTHON] [SCRIPT] discover pytest [ARGS] -- [PYTEST ARGS]
  • SCRIPT (any python): pythonFiles/testing_tools/run_adapter.py
  • SCRIPT (python3-only): -m pythonFiles.testing_tools.adapter
• args:
  • --simple - simplified output for use while debugging
  • --no-hide-stdio - do not hide pytest stdout/stderr during discover (useful when debugging)
  • --pretty - format output for readability
• implicit pytest args:
  • --collect-only

Output

OUTPUT: ROOTED_TESTS[]
ROOTED_TESTS: {
    # Uniquely identifies test root.
    "rootid": str
    # Absolute path to test root.
    "root": str
    # The discovered files, suites, etc. under this root.
    "parents": PARENT[]
    # The discovered tests under this root.
    "tests": TEST[]
}
PARENT: {
    # Uniquely identifies the parent.
    "id": str
    # The kind of parent.
    "kind": "folder"|"file"|"suite"|"function"|"subtest"
    # A human-friendly identifier.
    "name": str
    # The parent's parent (i.e. in a tree from the root.
    "parentid": str
}
TEST: {
    # Uniquely identifies the test.
    "id": str
    # A human-friendly identifier.
    "name": str
    # The location of the test's source code.
    "source": str ("<FILENAME>:<LINENO>")
    # Any supported markers associated with the test.
    "markers": ("skip"|"skip-if"|"expected-failure")[]
    # The ID of the test's parent.
    "parentid": str
}

Notes:

• IDs may be used to identify a test or parent to the test framework (e.g. to run specific tests)
• every child of the root will have the "rootid" as its "parentid"
• supported test markers are framework-agnostic but correspond to framework-specific ones
• a test is considered a "subtest" if its parent is a function or a subtest
• a subtest is a test that is effectively a parameterization of a more general test
  • in pytest parameterized tests are the only kind of subtest
• a subtest may be a parent if there is a nested subtest in it
  • this is not supported in pytest, but is in unittest

Example:

[{
    "rootid": ".",
    "root": "/x/y/z",
    "parents": [{
        "id": "./test_spam.py",
        "kind": "file",
        "name": "test_spam.py",
        "parentid": "."
    }, {
        "id": "./test_spam.py::SpamTests",
        "kind": "suite",
        "name": "SpamTests",
        "parentid": "./test_spam.py"
    },
    "tests" [{
        "id": "./test_spam.py::test_all",
        "name": "test_all",
        "source": "test_spam.py:11",
        "markers": ["skip", "expected-failure"],
        "parentid": "./test_spam.py"
    }, {
        "id": "./test_spam.py::SpamTests::test_spam1",
        "name": "test_spam1",
        "source": "test_spam.py:23",
        "markers": ["skip"],
        "parentid": "./test_spam.py::SpamTests"
    }]
}]