Q3 Test analytics UI & PR

[Adal3n3]

We are releasing flaky test detection in Q3. Customers will be able to see the flaky test label, flake rate, and flake pattern (e.g., test passed x times, failed x times). We have a strong understanding of the 'Test-fixing mode workflow' and the 'Test-performance workflow.' These two mental models focus on what developers need to fix a test vs. what they need to improve a test.

PM doc: link

Figma: link

Design exploration for user testing

Latest Design

User testing feedback: link

PR comment

Beta Give feedback

1. Provide copy-and-paste unparsed test "string" #2068
   Test Analytics +1
   
2. Color-coded test outputs #2180
   Test Analytics +1
   
3. XML File not found #2149
   Test Analytics +1
   
4. Calculate, store, and update flake rate for each Flake we process #2162
   investigation +1
   
5. https://github.com/codecov/internal-issues/issues/670
6. Trigger an error message if the PR didn't update due to Codecov processing issue #2218
   Test Analytics +1
7. Communicate why a test results file couldn't be parsed #2376
   Area: PR Comment - UX Area: Report Processing Test Analytics +3
   
8. {Test}{GA Launch Prep} Ensure TA honors plan details #2436
   High Test Analytics +2
   

   
9. Add TA feedback link in the comment #2470
   Test Analytics +1
   
10. Usability improvements of the PR design with more collapse, ordering, limit to 3 tests for each section #2179
    Dev-Ready Test Analytics +2
    
11. Rerun "Github Action" button #2186
    investigation +1
    
12. Bring Flaky Test Detection out of shadow mode. #2457
    Test Analytics +1
    
13. {GA Launch Prep} Add seat enforce logic to TestResultFinisher #2264
    Test Analytics +1
    
14. https://github.com/codecov/internal-issues/issues/826

Dashboard UI

Beta Give feedback

1. Onboarding: The ci.yaml file examples should change based on the selected framework #2151
   Test Analytics +1
2. Improve the copy and paste-able test name + file path output in the UI #2181
   Test Analytics +1
   
3. GA: Private repo upsell path and roll back plan or limited feature #2242
   Test Analytics +1
   
4. UI banner: when XML File not found #2269
   Test Analytics +1
5. TA: Early Q4 push a UI survey banner if it's configured (same logic as pro, team plan and BA) #2401
   Test Analytics +1
6. [TA:backlog] Add view link for the commits failed #2471
   Dev-Ready Test Analytics carryover +3
7. [TA] UI Support Work #2528
   7 of 11
   Test Analytics +1
   
8. Main branch: 30 days historical trend filter
9. Main branch: Test suites dropdown with search capability
10. Main branch: Flags dropdown with search capability
11. Main branch: Flake rate column and flake pattern
12. Main branch: 5 dashboard metrics
13. Feature branch: last test run compare to main
14. Use Lato front for all numeric on table #2163
    +0
15. Expired flake experience (no need anymore)
16. Flaky Test Detection MVP UI #2078
    P0: must do epic +2
    
17. Change "Environment" UI copy to "Flag" #2388
    +0
18. Fix for no repo passed into GQL query #2406
    +0
    
19. UI search by test name
20. Backlog: Display the latest failed time in the UI maybe next to the failed commits
21. include filepath for Jest on UI onboarding #2605
    Test Analytics +1

Doc

Beta Give feedback

1. Doc: On the setup flow add instruction on how to change Pytest test suites name on Junit XML (as an optional)
2. Doc: Explain the 30 days retentions section (changed it to 60 days)
3. Doc: Are we doing anything to improve the public API in q3?
4. Doc: Mention about flags setup (env setup)
5. Doc: Provide instruction on what are the testing frameworks and languages can be converted to JUnit XML reports #2619
   +0

Applications Team

Beta Give feedback

1. [API] Create test suites API filter #2411
   +0
   
2. [API] Create historical trend API filter #2409
   +0
   
3. [API] Create environment AKA flags filter #2413
   +0
   
4. [API] Create filter by slowest tests number, total of flake tests number, total of failures, and total of skipped tests #2430
   +0
   
5. [Gazebo] Create filter of test name #2424
   +0
   
6. <Temp for test string coloring> #2431
   +0
   
7. [Gazebo] Update test name to test string in the table UI #2432
   +0
   
8. Add conditional logic for Flakey tests #2635
   +0
   
9. Update with plan specfic rendering #2664
   +0
   
10. [Gazebo] Polish Time Duration Formats #2663
    +0
    
11. [Gazebo] Polish Percent Badge Format #2662
    +0
12. [Gazebo] Create Buttons for Test Type Filtering #2666
    +0
13. [Gazebo] Create Hook for fetching Test Suites #2668
    +0
    
14. [Gazebo] Create Hook for fetching Flags #2669
    +0
    
15. [Gazebo] Glue Test suite hook to Selector Section #2670
    +0
    
16. [Gazebo] Glue Flags hook to Selector Section #2671
    +0
    
17. [Gazebo] Add number of flaky tests #2422
    +0
    
18. [Gazebo] Glue Flakey Test Aggregates values to Metrics Section #2657
    +0
    
19. [Gazebo] Create Flakey Test Aggregates Hook #2658
    +0
    
20. [API] Create filter of test name #2425
    +0
    
21. Create Branch Table View #2438
    +0
    
22. [Gazebo] Create Tests Results Aggregates Hook #2660
    +0
    
23. [Gazebo] Glue Test Results Aggregates values to Metrics Section #2656
    +0
    
24. [Gazebo] Create tootip component used over the CI effeciany section components #2452
    +0
    
25. Technical Planning for Test Analytics UI #2169
    investigation +1
    
26. [Gazebo] Create Historical trend select component #2408
    +0
    
27. [Gazebo] Create Test suites multi select component #2410
    +0
    
28. [Gazebo] Create Environment aka Flags multi select component #2412
    +0
    
29. [Gazebo] Create CI run efficiency section #2415
    epic +1
    
30. [API] Create hours of total tests run time, percentage increased in 30 days, hours increased in 30 days #2416
    +0
    
31. [API] Create number of total slowest tests, total run time of the slowest tests in hours #2417
    +0
    
32. [API] Create number of total flaky tests number, percentage increased in 30 days, total rerun time for flaky tests in hours #2418
    +0
    
33. [API] Create average percentage of flake rate, percentage increased in the latest 30 days, avg number of times a test ran before it passed #2419
    +0
    
34. [API] Create number of total failures across all branches #2420
    +0
    
35. [API] Create number of total skipped tests across all branches #2421
    +0
    
36. [Gazebo] Create Metric reuseable component #2414
    High +1
    
37. [API] Create total number of flaky tests number #2423
    +0
    
38. [API] Update to use last duration in seconds #2427
    +0
    
39. [Gazebo] Update avg duration to last duration in the table #2426
    +0
    
40. [API] Create flake rate field percantage #2429
    +0
    
41. [Gazebo] Create flake rate new column #2428
    +0
    
42. Update public api