Reporting API test expansion and clean up #18986

stacey-gammon · 2018-05-10T20:38:58Z

This started as some prep work to cover CSV reports generated over data that includes a closed index but ended up growing to include quite a bit of clean up and re-organization. (since this bug: #18901)

Separated out specific test focuses into different files. One for merging new reporting data into an existing index, one for simple report generation urls, and then one for stats.
Removed reporting stat tests from the main stats file especially since with the work @tsullivan is doing, stats should no longer be specifically aware of reporting - reporting will register it's stats as a plugin.
using the new and expanded .kibana archive created in OSS.
remove .kibana index from the reporting archives, they are now specifically only include .reporting* indexes.
Updated the OSS .kibana archive to include some dashboards and visualizations created in 6.2. Was planning on using that data to create a 6.2 report, though it ended up not being necessary (the current data/kibana index is irrelevant when it comes to old reporting data).
Hopefully makes it easier to see what kinds of reports are being tested by extracting the report generation urls into a separate file and exporting them as consts.
Created a reportingAPI service and statsAPI service to extract common functionality.

TODO:

reporting/historic data is left in for the reporting functional tests, but we could lessen the build size if we get rid of it and replace it with the OSS version. That being said, eventually we'll want reporting tests to include x-pack specific saved objects (for instance, if we ever embed graphs on a dashboard). But when that happens we can still use the large dashboard/current/data archive and just import an xpack specific one for .kibana.

stacey-gammon · 2018-05-10T20:41:15Z

x-pack/test/api_integration/apis/xpack_main/stats/stats.js

-      await esArchiver.load('reporting/6_2');
-      // Not really neccessary to have data indexes, but it feels incomplete to leave out, and it is possible that
-      // having data available could potentially interefere with the stats api (unlikely... but possible).
-      await esArchiver.load('logstash_functional');


don't know who originally wrote that comment (could have been me and I forgot, hah) but it seems like it would be a bug if the stats api relied on data, so I decided to remove this archive unless we have a more concrete reason to pull it back in.

elasticmachine · 2018-05-10T21:40:42Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-16T17:43:40Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-16T19:18:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-05-17T12:43:06Z

jenkins test this

(time range of the reports I made for this test should be out of the recent window by now so they should pass...)

elasticmachine · 2018-05-17T13:45:35Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-17T15:59:20Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-21T18:43:30Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-21T20:44:26Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

nreese

lgtm

elasticmachine · 2018-05-21T22:03:47Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-05-21T23:03:59Z

Failed on:

3:08 
21:13:08 Summary of all failing tests
21:13:08  FAIL  packages/kbn-plugin-helpers/tasks/build/create_build.test.js (10.538s)
21:13:08   ● creating the build › adds build metadata to package.json
21:13:08 
21:13:08     Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.
21:13:08       
21:13:08       at node_modules/jest-jasmine2/build/queue_runner.js:68:21
21:13:08 
21:13:08   ● creating the build › adds build metadata to package.json
21:13:08 
21:13:08     Error
21:13:08       Error: ENOENT: no such file or directory, open '/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-intake/kibana/packages/kbn-plugin-helpers/tasks/build/__fixtures__/create_build_test_plugin/build/kibana/create_build_test_plugin/node_modules/noop3/factory.js'
21:13:08

elasticmachine · 2018-05-22T00:13:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-22T13:38:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-05-22T14:14:34Z

Failed on something that looks unrelated:

12:51:23 ................................................................................
12:51:24 ....................................................................
12:51:24 Chrome 66.0.3359 (Linux 0.0.0) Promise service Promise.race() does not wait for subsequent promises to resolve/reject FAILED
12:51:24 	Error: expected 99 to be above 100
12:51:24 	    at Assertion.assert (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:12474:13)
12:51:24 	    at Assertion.lessThan.Assertion.below (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:12691:10)
12:51:24 	    at Function.lessThan (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:12877:17)
12:51:24 	    at Suite._callee3$ (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:734294:53)
12:51:24 	    at tryCatch (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:363302:40)
12:51:24 	    at Generator.invoke [as _invoke] (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:363540:22)
12:51:24 	    at Generator.prototype.(anonymous function) [as next] (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:363354:21)
12:51:24 	    at step (http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:734111:191)
12:51:24 	    at http://localhost:5610/bundles/tests.bundle.js?shards=4&shard_num=1:734111:361

and another failure from a bad refactor. Fixing the later and rebasing to hopefully fix the former.

elasticmachine · 2018-05-22T15:32:37Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-22T16:40:48Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* Update & expand reporting API tests * remove xpack stuff from oss archive * wrap "it"s so they are in expected order. * Update expected recent stats * url was pointing to a visualization not a csv * Move comment around * Merging with changes on master renaming stats to usage * fix reference to old stats file. * bad merge

stacey-gammon added test :Sharing (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead labels May 10, 2018

stacey-gammon commented May 10, 2018

View reviewed changes

stacey-gammon force-pushed the 2018-05-reporting-test-expansion branch from 49dac52 to 33e4e1a Compare May 16, 2018 15:39

stacey-gammon force-pushed the 2018-05-reporting-test-expansion branch from a102462 to 4ca452f Compare May 17, 2018 14:52

stacey-gammon force-pushed the 2018-05-reporting-test-expansion branch from 4ca452f to efff55e Compare May 21, 2018 17:36

stacey-gammon requested review from nreese and chrisdavies May 21, 2018 20:48

nreese approved these changes May 21, 2018

View reviewed changes

stacey-gammon force-pushed the 2018-05-reporting-test-expansion branch from 2a20e26 to f8d854b Compare May 21, 2018 23:05

stacey-gammon force-pushed the 2018-05-reporting-test-expansion branch from f8d854b to 55f0b66 Compare May 22, 2018 12:33

stacey-gammon added 6 commits May 22, 2018 10:14

Update & expand reporting API tests

48b4178

remove xpack stuff from oss archive

1c071e5

wrap "it"s so they are in expected order.

dc878f6

Update expected recent stats

2cf2a8c

url was pointing to a visualization not a csv

f39386a

Move comment around

8b6fb57

stacey-gammon added 2 commits May 22, 2018 10:14

Merging with changes on master renaming stats to usage

aa4bceb

fix reference to old stats file.

377a2d0

stacey-gammon force-pushed the 2018-05-reporting-test-expansion branch from fc1383f to 377a2d0 Compare May 22, 2018 14:14

elasticmachine mentioned this pull request May 22, 2018

[Reporting] Make Chromium the default browser #18357

Closed

5 tasks

bad merge

7cd6d14

stacey-gammon merged commit c14f7a7 into elastic:master May 22, 2018

stacey-gammon mentioned this pull request May 22, 2018

[6.x] Reporting API test expansion and clean up (#18986) #19316

Merged

tsullivan mentioned this pull request May 23, 2018

[Monitoring/Reporting Stats] Move hardcoded Reporting stats type collection out of monitoring plugin #18894

Merged

4 tasks

stacey-gammon deleted the 2018-05-reporting-test-expansion branch May 31, 2018 17:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reporting API test expansion and clean up #18986

Reporting API test expansion and clean up #18986

stacey-gammon commented May 10, 2018 •

edited

Loading

stacey-gammon May 10, 2018

elasticmachine commented May 10, 2018

elasticmachine commented May 16, 2018

elasticmachine commented May 16, 2018

stacey-gammon commented May 17, 2018

elasticmachine commented May 17, 2018

elasticmachine commented May 17, 2018

elasticmachine commented May 21, 2018

elasticmachine commented May 21, 2018

nreese left a comment

elasticmachine commented May 21, 2018

stacey-gammon commented May 21, 2018

elasticmachine commented May 22, 2018

elasticmachine commented May 22, 2018

stacey-gammon commented May 22, 2018

elasticmachine commented May 22, 2018

elasticmachine commented May 22, 2018

Reporting API test expansion and clean up #18986

Reporting API test expansion and clean up #18986

Conversation

stacey-gammon commented May 10, 2018 • edited Loading

stacey-gammon May 10, 2018

Choose a reason for hiding this comment

elasticmachine commented May 10, 2018

💔 Build Failed

elasticmachine commented May 16, 2018

💔 Build Failed

elasticmachine commented May 16, 2018

💔 Build Failed

stacey-gammon commented May 17, 2018

elasticmachine commented May 17, 2018

💔 Build Failed

elasticmachine commented May 17, 2018

💔 Build Failed

elasticmachine commented May 21, 2018

💔 Build Failed

elasticmachine commented May 21, 2018

💚 Build Succeeded

nreese left a comment

Choose a reason for hiding this comment

elasticmachine commented May 21, 2018

💔 Build Failed

stacey-gammon commented May 21, 2018

elasticmachine commented May 22, 2018

💚 Build Succeeded

elasticmachine commented May 22, 2018

💔 Build Failed

stacey-gammon commented May 22, 2018

elasticmachine commented May 22, 2018

💔 Build Failed

elasticmachine commented May 22, 2018

💚 Build Succeeded

stacey-gammon commented May 10, 2018 •

edited

Loading