Skip to content

Commit

Permalink
DOCS-8109 Add Test Visibility Explorer Documentation (#23456)
Browse files Browse the repository at this point in the history 
* DOCS-8109

Separate Test Visibility Explorer doc pages about test runs from CI Visibility doc pages about pipeline executions.

* Copy Nit

* Update Anchor Links
  • Loading branch information
@alai97
alai97 authored Jun 3, 2024
1 parent eea73dc commit a196d2f
Show file tree
Hide file tree
Showing 18 changed files with 612 additions and 239 deletions.
24 changes: 22 additions & 2 deletions config/_default/menus/main.en.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3578,7 +3578,7 @@ menu:
parent: ci_explorer
identifier: ci_explorer_search_syntax
weight: 301
- name: Search Test Runs or Pipeline Executions
- name: Search Pipeline Executions
url: continuous_integration/explorer/facets/
parent: ci_explorer
identifier: ci_explorer_facets
Expand Down Expand Up @@ -3711,10 +3711,30 @@ menu:
identifier: tests_search
weight: 3
- name: Explorer
url: continuous_integration/explorer
url: tests/explorer/
parent: tests
identifier: tests_explorer
weight: 4
- name: Search Syntax
url: tests/explorer/search_syntax
parent: tests_explorer
identifier: tests_explorer_search_syntax
weight: 401
- name: Search Test Runs
url: tests/explorer/facets/
parent: tests_explorer
identifier: tests_explorer_facets
weight: 402
- name: Export
url: tests/explorer/export/
parent: tests_explorer
identifier: tests_explorer_export
weight: 403
- name: Saved Views
url: tests/explorer/saved_views/
parent: tests_explorer
identifier: tests_explorer_saved_views
weight: 404
- name: Monitors
url: monitors/types/ci/?tab=tests
parent: tests
Expand Down
136 changes: 19 additions & 117 deletions content/en/continuous_integration/explorer/_index.md
View file
Original file line number Diff line number Diff line change
@@ -1,122 +1,27 @@
---
title: Continuous Integration Visibility Explorer
kind: documentation
description: Learn how to search and filter your test runs or pipeline executions in the CI Visibility Explorer.
description: Learn how to search and filter your pipeline executions in the CI Visibility Explorer.
further_reading:
- link: "/continuous_integration/pipelines/"
tag: "Documentation"
text: "Explore pipeline data to resolve build problems"
- link: "/continuous_integration/tests/"
tag: "Documentation"
text: "Explore test data to find and fix problem tests"
- link: "https://www.datadoghq.com/blog/configure-pipeline-alerts-with-ci-monitors/"
tag: "Blog"
text: "Configure pipeline alerts with Datadog CI monitors"
---

## Overview

The CI Visibility Explorer allows you to [search and filter](#search-and-filter), [visualize](#visualize), and [export](#export) test runs or pipeline executions at multiple levels using any tag.

For more information about common facets that you can use as part of your search query in the CI Visibility Explorer, see [Test Run or Pipeline Execution Facets][3].

{{< tabs >}}
{{% tab "Test Runs" %}}

Navigate to [**Software Delivery** > **Test Visibility** > **Test Runs**][101] to see your CI test run results across the following levels: **Session**, **Module**, **Suite**, and **Test**.

{{< img src="/continuous_integration/test_runs.png" text="Test Runs page" style="width:100%" >}}

The **Test** panel on the left lists default facets you can use to search for your tests.
The CI Visibility Explorer allows you to [search and filter](#search-and-filter), [visualize](#visualize), and [export](#export) pipeline executions at multiple levels using any tag.

| Facet | Description |
|---|---|
| Test Status | The outcome of the test: `Passed`, `Failed`, or `Skipped`. |
| Duration | Length of time for the test to complete. |
| Test Service | The [test service][107] instrumented with CI Visibility. |
| Test Full Name | Identifier for a test that includes the test name, test suite name, and configuration or parameter if present. |
| Test Name | A concise name for a test case. Defined in the test itself. |
| Test Suite | A [group of tests][108] exercising the same unit of code depending on your language and testing framework. |
| Flaky | Exhibits both a passing and failing status across multiple test runs for the same commit. |
| Has Parameters | Whether or not a test has parameters: `true` or `false`. |
| Known Flaky | Whether or not a test is known to be flaky: `true` or `false`. <br><br>This test run failed and the test is identified as a flaky test in the current branch or the default branch. |
| Language | The programming language of the library that generated the test. |
| New Flaky | Whether or not this flaky test has occurred before: `true` or `false`. <br><br>The test run identifies the test as flaky in the commit. The test wasn't previously identified as a flaky test in the current branch or the default branch. |
| Performance Regression | A test run is marked as a regression when its duration is both five times the mean and greater than the max duration for the same test in the default branch. |
| Baseline Mean | For a test regression, indicates the mean duration of the same test in the default branch calculated over the last week of test runs. |
| Baseline Standard Deviation | For a test regression, indicates the standard deviation of the same test in the default branch calculated over the last week of test runs durations. |
| Absolute Change | For a test regression, indicates the absolute change of the test run duration compared to the baseline mean. |
| Relative Change | For a test regression, Indicates the relative change of the test run duration compared to the baseline mean. |
| Standard Deviation Change | Indicates if the test has been newly added. |
| Test Code Owners | The name of the test's codeowners as inferred from the repository configuration. |
| Test Fingerprint | The unique identifier for an individual test run. |
| Test Framework | The underlying framework or set of tools used for creating and executing tests. |
| Test Command | The command that was used to execute tests. |
| Test Bundle | Equivalent to a test module. This is used by earlier Datadog testing library versions. |
| Test Full Name | The full name of the test. |
| Test Module | The module of the test, which varies depending on the language:<br><br>In .NET, a test module groups every test that is run under the same unit test project.<br>In Swift, a test module groups every test that is run for a given bundle.<br>In JavaScript, the test modules map one-to-one to test sessions.<br>In Java, a test module groups every test that is run by the same Maven Surefire, Failsafe, or Gradle Test task execution.<br>In Python, a test module groups every test that is run under the same `.py` file as part of a test suite, which is typically managed by a framework like `unittest` or `pytest`.<br>In Ruby, a test module groups every test that is run within the same test file, which is typically managed by a framework like `RSpec` or `Minitest`. |
| Test Traits | The traits of the test such as `category:flaky`. |
| Test Type | The type of the test such as `unit benchmark` or `browser`. |
| RUM Active | Indicates if the test was run inside of an active [Real User Monitoring][109] web session. |
| Is New | Indicates if the test has been newly added. |
| Is Retry | Indicates if the test has been run as a result of a retry. |
| Code Coverage Enabled | Indicates if the [Intelligent Test Runner][111] has enabled [code coverage][112] per test for the session. |
| Skipped by ITR | Number of tests that were skipped during the session by the Intelligent Test Runner. |
| Test Skipping Enabled | Whether the test session or module is allowed to be skipped by ITR. |
| Test Skipping Type | The method or criteria used by the Intelligent Test Runner to determine which tests to skip. |
| Test Skipped | The total count of tests that were not executed during the test session, which may include tests that were configured to skip, or were set as manual exclusions. |
| Time Saved | The length of time saved for the session by Intelligent Test Runner usage. |
| Early Flake Detection Enabled | Indicates if the test has been run using [Early Flake Detection][110]. |
| Early Flake Detection Abort Reason | Indicates the Early Flake Detection abort reason for a test. |

You can filter by test level: session, module, suite, and test run. Each test level represents a different level of aggregation of tests.

{{< img src="ci/ci-test-suite-visibility.png" alt="Test Suite Visibility" style="width:100%;">}}

### Sessions
Test sessions are the highest level of aggregation. They correspond one to one to a test command, such as `yarn test`, `mvn test`, or `dotnet test`.

For JUnit report uploads there is 1 session per report file uploaded.

### Module
The definition of module changes slightly per language:

* In .NET, a test module groups every test that is run under the same [unit test project][104].
* In Swift, a test module groups every test that is run for a given bundle.
* In JavaScript, the test modules map one-to-one to test sessions.
* In Java, a test module groups every test that is run by the same Maven Surefire/Failsafe or Gradle Test task execution.
* In JUnit report uploads, the test modules map one-to-one to test sessions.

An example of a module is `SwiftLintFrameworkTests`, which corresponds to a test target in [`SwiftLint`][105].

### Suite
A test suite is a group of tests exercising the same unit of code.

An example of a test suite is `src/commands/junit/__tests__/upload.test.ts`, which corresponds to a test file in [`datadog-ci`][106].

Test run data is available in [dashboards][102] and [notebooks][103], enabling build engineering teams to customize their communication about high-priority work and CI trends over time.

[101]: https://app.datadoghq.com/ci/test-runs
[102]: https://app.datadoghq.com/dashboard/lists
[103]: https://app.datadoghq.com/notebook/list
[104]: https://learn.microsoft.com/en-us/visualstudio/test/create-a-unit-test-project?view=vs-2022#to-create-a-unit-test-project
[105]: https://github.com/realm/SwiftLint/blob/7738f0c0a5990201ca6556bdb2f13f8e67b5191d/Package.swift#L71
[106]: https://github.com/DataDog/datadog-ci/blob/6de6ea3bbffa57d8576422535061ca35c759feb6/src/commands/junit/__tests__/upload.test.ts
[107]: /glossary/?product=ci-cd#test-service
[108]: /glossary/?product=ci-cd#test-suite
[109]: /real_user_monitoring/
[110]: /tests/early_flake_detection/
[111]: /intelligent_test_runner/
[112]: /tests/code_coverage/

{{% /tab %}}
{{% tab "Pipeline Executions" %}}

Navigate to [**Software Delivery** > **CI Visibility** > **Executions**][101] to see your CI pipeline execution results across the following levels: **Pipeline**, **Stage**, **Job**, **Step**, and **Command**.
Navigate to [**Software Delivery** > **CI Visibility** > **Executions**][5] to see your CI pipeline execution results across the following levels: **Pipeline**, **Stage**, **Job**, **Step**, and **Command**.

{{< img src="/continuous_integration/pipeline_executions.png" text="CI Pipeline Executions page" style="width:100%" >}}

The **CI** panel on the left lists default facets you can use to search for your tests.
## Default CI facets

The **CI** panel on the left lists default facets you can use to search for your pipeline executions.

| Facet | Description |
|---|---|
Expand Down Expand Up @@ -147,6 +52,8 @@ The **CI** panel on the left lists default facets you can use to search for your
| Is Deployment | Indicates whether a job within the pipeline initiated a deployment. |
| Contains Deployment | Indicates whether the pipeline includes any jobs that trigger a deployment. |

For more information about common facets that you can use as part of your search query in the CI Visibility Explorer, see [Pipeline Execution Facets][3].

## Pipeline executions details and traces

You can see aggregated data about pipeline executions over the selected time frame. Use the search field and facets to scope the list down to the executions you want to investigate. Change the list to show pipelines, stages, or jobs using the buttons at the top.
Expand All @@ -157,39 +64,29 @@ Below are three graphs that visualize the durations of your most active pipeline

Each pipeline execution is reported as a trace, which includes stage and job information. Access individual pipeline, stage, and job execution traces by clicking on an execution in the list (similar to clicking into a pipeline execution from the Pipeline Details view).

CI pipeline data is available in [dashboards][102] and [notebooks][103], enabling build engineering teams to customize their communication about high-priority work and CI trends over time.

[101]: https://app.datadoghq.com/ci/pipeline-executions
[102]: https://app.datadoghq.com/dashboard/lists
[103]: https://app.datadoghq.com/notebook/list
[104]: https://app.datadoghq.com/ci/pipeline-executions
[105]: https://app.datadoghq.com/ci/pipeline-executions?viz=timeseries

{{% /tab %}}
{{< /tabs >}}
CI pipeline data is available in [dashboards][6] and [notebooks][7], enabling build engineering teams to customize their communication about high-priority work and CI trends over time.

## Search and filter

You can narrow down, broaden, or shift your focus on a subset of test runs or pipeline executions by clicking on the facets to the left or writing your own custom query in the search bar. When you select and deselect facets, the search bar automatically reflects your changes. Similarly, you can modify the search bar query or write a query from scratch in the search bar to select and deselect the facets on the left.
You can narrow down, broaden, or shift your focus on a subset of pipeline executions by clicking on the facets to the left or writing your own custom query in the search bar. When you select and deselect facets, the search bar automatically reflects your changes. Similarly, you can modify the search bar query or write a query from scratch in the search bar to select and deselect the facets on the left.

- To learn how to search for tests and pipelines, see [Search and Manage][1].
- To learn how to search for pipelines, see [Search and Manage][1].
- To learn how to create queries, see [Search Syntax][2].

## Analyze

Group your queried test runs or pipeline executions into higher-level entities such as fields, patterns, and transactions in order to derive or consolidate information. By using [facets][3], which you do not need to create to search for attributes, you can accomplish the following actions:
Group your queried pipeline executions into higher-level entities such as fields, patterns, and transactions in order to derive or consolidate information. By using [facets][3], which you do not need to create to search for attributes, you can accomplish the following actions:

- Search and keep track of the progress of tests running in a CI/CD pipeline.
- Investigate every CI/CD job execution to identify and troubleshoot failing test runs.
- Identify flaky tests to fix.

## Visualize

Select a visualization type to visualize the outcomes of your filters and aggregations and better understand your test runs or pipeline executions. For example, you can view your test results in a list to organize your test data into columns, or in a timeseries graph to measure your pipeline data over time.
Select a visualization type to visualize the outcomes of your filters and aggregations and better understand your pipeline executions. For example, you can view your pipeline executions in a list to organize your pipeline data into columns, or in a [timeseries graph][8] to measure your pipeline data over time.

## Export

Export your view in the CI Visibility Explorer to reuse it later or in different contexts.
[Export your view][4] in the CI Visibility Explorer to reuse it later or in different contexts.

## Further reading

Expand All @@ -198,3 +95,8 @@ Export your view in the CI Visibility Explorer to reuse it later or in different
[1]: /continuous_integration/search
[2]: /continuous_integration/explorer/search_syntax
[3]: /continuous_integration/explorer/facets
[4]: /continuous_integration/explorer/saved_views
[5]: https://app.datadoghq.com/ci/pipeline-executions
[6]: https://app.datadoghq.com/dashboard/lists
[7]: https://app.datadoghq.com/notebook/list
[8]: https://app.datadoghq.com/ci/pipeline-executions?viz=timeseries
44 changes: 12 additions & 32 deletions content/en/continuous_integration/explorer/export.md
View file
Original file line number Diff line number Diff line change
@@ -1,63 +1,43 @@
---
title: Export Test Runs or Pipeline Executions
title: Export Pipeline Executions
kind: documentation
further_reading:
- link: "/continuous_integration/search/"
tag: "Documentation"
text: "Search for your pipelines"
- link: "/dashboards"
- link: "/continuous_integration/explorer/saved_views"
tag: "Documentation"
text: "Learn about Dashboards"
- link: "/notebooks"
text: "Learn about Saved Views"
- link: "/monitors/types/ci"
tag: "Documentation"
text: "Learn about Notebooks"
- link: "/monitors"
tag: "Documentation"
text: "Learn about Monitors"
text: "Learn about CI Pipeline Monitors"
---

## Overview

You can use your CI Visibility search query and visualization graphs in dashboards, monitors, and notebooks, or programmatically search for events using the [Search CI Visibility Tests][1] or [Search CI Visibility Pipelines endpoint][2].
You can use your CI Visibility search query and visualization graphs in dashboards, monitors, and notebooks, or programmatically search for events using the [Search Pipeline Events endpoint][1].

## Export the search query or visualization

{{< tabs >}}
{{% tab "Test Runs" %}}

You can copy, export, or download your aggregated search query and visualization graphs in the [Test Visibility Explorer][101].

{{< img src="continuous_integration/explorer/test_export.png" alt="Export your test runs view in the Test Visibility Explorer" style="width:100%;">}}

[101]: https://app.datadoghq.com/ci/test-runs

{{% /tab %}}
{{% tab "Pipeline Executions" %}}

You can copy, export, or download your aggregated search query and visualization graphs in the [CI Visibility Explorer][101].
You can copy, export, or download your aggregated search query and visualization graphs in the [CI Visibility Explorer][2].

{{< img src="continuous_integration/explorer/pipeline_export.png" alt="Export your pipelines view in the CI Visibility Explorer" style="width:100%;">}}

[101]: https://app.datadoghq.com/ci/pipeline-executions

{{% /tab %}}
{{< /tabs >}}

Click the **Export** button on the right hand corner and select an option from the dropdown menu:

- Share your [view][7] of the [CI Visibility Explorer][3].
- Export your search results to a [monitor][5] that triggers alerts on predefined thresholds.
- Share your [saved view][7] of the [CI Visibility Explorer][3].
- Export your search results to a [CI Pipeline monitor][5] that triggers alerts on predefined thresholds.
- Export your search results to an [existing notebook][6] for reporting or consolidation purposes.
- Download your search results as a CSV file for individual CI Visibility test or pipeline events and specific aggregations.

Options available for some visualization types are not supported in others. For example, you cannot download a distribution graph into a CSV file.

## Further Reading
## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /api/latest/ci-visibility-tests/#search-tests-events
[2]: /api/latest/ci-visibility-pipelines/#search-pipelines-events
[1]: /api/latest/ci-visibility-pipelines/#search-pipelines-events
[2]: https://app.datadoghq.com/ci/pipeline-executions
[3]: /continuous_integration/explorer/
[4]: /api/latest/
[5]: /monitors/types/ci/
Expand Down
Loading

0 comments on commit a196d2f

Please sign in to comment.