From fccd5c205ddc91ab7c23665c3e4e9717bb260672 Mon Sep 17 00:00:00 2001 From: Munir Abdinur Date: Tue, 26 Nov 2024 12:39:03 -0500 Subject: [PATCH] document incomplete-test-app decorator --- docs/edit/skip-tests.md | 9 +++++++-- docs/execute/test-outcomes.md | 16 ++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/docs/edit/skip-tests.md b/docs/edit/skip-tests.md index 26682bc9d3..b1e07d47dc 100644 --- a/docs/edit/skip-tests.md +++ b/docs/edit/skip-tests.md @@ -3,7 +3,8 @@ Three decorators allow you to skip test functions or classes for a library: * `@irrelevant`: The tested feature/behavior is irrelevant to the library, meaning the feature is either purposefully not supported by the lib or cannot reasonably be implemented * `@bug`: The lib does not implement the feature correctly/up to spec * `@flaky` (subclass of `bug`): The feature sometimes fails, sometimes passes. It's not reliable, so don't run it. -* `@missing_feature`: The tested feature/behavior does not exist in the library or there is a deficit in the test library that blocks this test from executing for the lib +* `@missing_feature`: The tested feature/behavior does not exist in the library +* `@incomplete_test_app`: There is a deficit in the weblog/parametric apps or testing interface that prevents us from validating a feature across different applications. To skip specific test functions within a test class, use them as in-line decorators (Example below). To skip test classes or test files, use the decorator in the library's [manifest file](./manifest.md). @@ -17,7 +18,7 @@ The decorators take several arguments: ```python -from utils import irrelevant +from utils import irrelevant, incomplete_test_app, bug, missing_feature @irrelevant(library="nodejs") @@ -35,4 +36,8 @@ class Test_AwesomeFeature: @missing_feature(reason="Maybe too soon") def test_full(self) assert 42 + + @incomplete_test_app(library="python", "trace/span/start endpoint does not exist") + def test_span_creation(self): + assert 68 ``` diff --git a/docs/execute/test-outcomes.md b/docs/execute/test-outcomes.md index c0e3b0e889..f5425cf639 100644 --- a/docs/execute/test-outcomes.md +++ b/docs/execute/test-outcomes.md @@ -2,11 +2,11 @@ Each test can be flagged with an expected outcomes, with a declaration in [manif Those declaration are interpreted by system-tests and impacts the test execution, and the outcome of the entire run : -| Declaration | Test is executed | Test actual outcome | System test output | Comment -| - | - | - | - | - -| \ | Yes | ✅ Pass | 🟢 Success | All good :sunglasses: -| Missing feature or bug | Yes | ❌ Fail | 🟢 Success | Expected failure -| Missing feature or bug | Yes | ✅ Pass | 🟠 Success | XPASS: The feature has been implemented, bug has been fixed -> easy win -| Flaky | No | N.A. | N.A. | A flaky test doesn't provide any usefull information, and thus, is not executed. -| Irrelevant | No | N.A. | N.A | There is no purpose of running such a test -| \ | Yes | ❌ Fail | 🔴 Fail | Only use case where system test fails : the test should have been ok, and is not +| Declaration | Test is executed | Test actual outcome | System test output | Comment +| - | - | - | - | - +| \ | Yes | ✅ Pass | 🟢 Success | All good :sunglasses: +| Missing feature or bug or incomplete test app | Yes | ❌ Fail | 🟢 Success | Expected failure +| Missing feature or bug or incomplete test app | Yes | ✅ Pass | 🟠 Success | XPASS: The feature has been implemented, bug has been fixed -> easy win +| Flaky | No | N.A. | N.A. | A flaky test doesn't provide any usefull information, and thus, is not executed. +| Irrelevant | No | N.A. | N.A | There is no purpose of running such a test +| \ | Yes | ❌ Fail | 🔴 Fail | Only use case where system test fails : the test should have been ok, and is not