Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sometimes reporting pdf optimize fails after upgrade #108963

Closed
liza-mae opened this issue Aug 17, 2021 · 5 comments
Closed

Sometimes reporting pdf optimize fails after upgrade #108963

liza-mae opened this issue Aug 17, 2021 · 5 comments
Labels
bug Fixes for quality problems that affect the customer experience (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead feedback_needed impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort

Comments

@liza-mae
Copy link
Contributor

liza-mae commented Aug 17, 2021 •
edited
Loading

When running automated upgrade testing, I have some inconsistent results with generating a PDF optimized report on the built-in flights sample data, since it does not happen too often, it has been hard to reproduce outside the CI runs. Need to find a way to capture more debug information in the CI runs.

So far seen on upgrade paths:

  • 7.8.x -> 7.14.0
  • 6.8.x -> 7.14.0

The test environment is:

  • Kibana has 8GB of memory
  • Test timeout is set to 360000 ms

Do these seem acceptable for reliable results?

[00:28:12]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:13]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:15]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:16]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:18]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:19]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:20]           └- ✖ fail: upgrade reporting smoke tests generate report space automation name flights type pdf_optimize
[00:28:20]           │      Error: Timeout of 360000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-tests/JOB/upgradeTest2/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/x-pack/test/upgrade/apps/reporting/reporting_smoke_tests.ts)

https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-upgrade-tests/lastCompletedBuild/JOB=upgradeTest2,TASK=kibana_upgrade_tests,node=ess-testing/testReport/
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-services (Team:AppServices)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

@liza-mae liza-mae added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead bug Fixes for quality problems that affect the customer experience labels Aug 17, 2021
@liza-mae
Copy link
Contributor Author

It is possible this is related to SIGHUP, I recall seeing that in the log from one of the failures early on, although I don't see it in the most recent failure, but I see an Internal Server Error

Logs:

  | Aug 14, 2021 @ 10:49:16.000 | Creating browser page driver | -

  | Aug 14, 2021 @ 10:49:17.000 | opening url https://0.0.0.0:18596/s/automation/app/dashboards#/view/7adfa750-4c81-11e8-b3d7-01146121b73d?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!f%2Cvalue%3A900000)%2Ctime%3A(from%3Anow-24h%2Cto%3Anow))&_a=(description%3A'Analyze%20mock%20flight%20data%20for%20ES-Air%2C%20Logstash%20Airways%2C%20Kibana%20Airlines%20and%20JetBeats'%2Cfilters%3A!()%2CfullScreenMode%3A!f%2Coptions%3A(darkTheme%3A!f%2ChidePanelTitles%3A!f%2CuseMargins%3A!t)%2Cpanels%3A!((embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A7%2Ci%3A'1'%2Cw%3A32%2Cx%3A0%2Cy%3A0)%2Cid%3Aaeb212e0-4c84-11e8-b3d7-01146121b73d%2CpanelIndex%3A'1'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A('Average%20Ticket%20Price'%3A%230A50A1%2C'Flight%20Count'%3A%2382B5D8)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A12%2Ci%3A'3'%2Cw%3A23%2Cx%3A17%2Cy%3A7)%2Cid%3Ac8fc3d30-4c87-11e8-b3d7-01146121b73d%2CpanelIndex%3A'3'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A15%2Ci%3A'4'%2Cw%3A48%2Cx%3A0%2Cy%3A85)%2Cid%3A'571aaf70-4c88-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'4'%2Ctype%3Asearch%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(ES-Air%3A%23447EBC%2CJetBeats%3A%2365C5DB%2C'Kibana%20Airlines'%3A%23BA43A9%2C'Logstash%20Airways'%3A%23E5AC0E)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A12%2Ci%3A'5'%2Cw%3A17%2Cx%3A0%2Cy%3A7)%2Cid%3A'8f4d0c00-4c86-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'5'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A('Carrier%20Delay'%3A%235195CE%2C'Late%20Aircraft%20Delay'%3A%231F78C1%2C'NAS%20Delay'%3A%2370DBED%2C'No%20Delay'%3A%23BADFF4%2C'Security%20Delay'%3A%23052B51%2C'Weather%20Delay'%3A%236ED0E0)))%2CgridData%3A(h%3A14%2Ci%3A'6'%2Cw%3A24%2Cx%3A24%2Cy%3A33)%2Cid%3Af8290060-4c88-11e8-b3d7-01146121b73d%2CpanelIndex%3A'6'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A14%2Ci%3A'7'%2Cw%3A24%2Cx%3A24%2Cy%3A19)%2Cid%3Abcb63b50-4c89-11e8-b3d7-01146121b73d%2CpanelIndex%3A'7'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(Count%3A%231F78C1)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A12%2Ci%3A'10'%2Cw%3A24%2Cx%3A0%2Cy%3A35)%2Cid%3A'9886b410-4c8b-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'10'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(Count%3A%231F78C1)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'13'%2Cw%3A14%2Cx%3A10%2Cy%3A19)%2Cid%3A'76e3c090-4c8c-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'13'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(Count%3A%231F78C1)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'14'%2Cw%3A14%2Cx%3A10%2Cy%3A27)%2Cid%3A'707665a0-4c8c-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'14'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2CmapCenter%3A!(27.421687059550266%2C15.371002131141724)%2CmapZoom%3A1)%2CgridData%3A(h%3A15%2Ci%3A'18'%2Cw%3A24%2Cx%3A24%2Cy%3A70)%2Cid%3A'79e8ff60-4c8e-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'18'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A8%2Ci%3A'21'%2Cw%3A48%2Cx%3A0%2Cy%3A62)%2Cid%3A'293b5a30-4c8f-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'21'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A7%2Ci%3A'22'%2Cw%3A16%2Cx%3A32%2Cy%3A0)%2Cid%3A'129be430-4c93-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'22'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2CmapCenter%3A!(42.19556096274418%2C9.536742995308601e-7)%2CmapZoom%3A1)%2CgridData%3A(h%3A15%2Ci%3A'23'%2Cw%3A24%2Cx%3A0%2Cy%3A70)%2Cid%3A'334084f0-52fd-11e8-a160-89cc2ad9e8e2'%2CpanelIndex%3A'23'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(defaultColors%3A('0%20-%2050'%3A'rgb(247%2C251%2C255)'%2C'100%20-%20150'%3A'rgb(107%2C174%2C214)'%2C'150%20-%20200'%3A'rgb(33%2C113%2C181)'%2C'200%20-%20250'%3A'rgb(8%2C48%2C107)'%2C'50%20-%20100'%3A'rgb(198%2C219%2C239)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'25'%2Cw%3A10%2Cx%3A0%2Cy%3A19)%2Cid%3Af8283bf0-52fd-11e8-a160-89cc2ad9e8e2%2CpanelIndex%3A'25'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(defaultColors%3A('0%20-%2050'%3A'rgb(247%2C251%2C255)'%2C'100%20-%20150'%3A'rgb(107%2C174%2C214)'%2C'150%20-%20200'%3A'rgb(33%2C113%2C181)'%2C'200%20-%20250'%3A'rgb(8%2C48%2C107)'%2C'50%20-%20100'%3A'rgb(198%2C219%2C239)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'27'%2Cw%3A10%2Cx%3A0%2Cy%3A27)%2Cid%3A'08884800-52fe-11e8-a160-89cc2ad9e8e2'%2CpanelIndex%3A'27'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(defaultColors%3A('0%20-%2011'%3A'rgb(247%2C251%2C255)'%2C'11%20-%2022'%3A'rgb(208%2C225%2C242)'%2C'22%20-%2033'%3A'rgb(148%2C196%2C223)'%2C'33%20-%2044'%3A'rgb(74%2C152%2C201)'%2C'44%20-%2055'%3A'rgb(23%2C100%2C171)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A15%2Ci%3A'28'%2Cw%3A24%2Cx%3A0%2Cy%3A47)%2Cid%3Ae6944e50-52fe-11e8-a160-89cc2ad9e8e2%2CpanelIndex%3A'28'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A6%2Ci%3A'29'%2Cw%3A8%2Cx%3A40%2Cy%3A7)%2Cid%3A'01c413e0-5395-11e8-99bf-1ba7b1bdaa61'%2CpanelIndex%3A'29'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A6%2Ci%3A'30'%2Cw%3A8%2Cx%3A40%2Cy%3A13)%2Cid%3A'2edf78b0-5395-11e8-99bf-1ba7b1bdaa61'%2CpanelIndex%3A'30'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A15%2Ci%3A'31'%2Cw%3A24%2Cx%3A24%2Cy%3A47)%2Cid%3Aed78a660-53a0-11e8-acbd-0be0ad9d822b%2CpanelIndex%3A'31'%2Ctype%3Avisualization%2Cversion%3A'7.14.0'))%2Cquery%3A(language%3Alucene%2Cquery%3A'')%2Ctags%3A!()%2CtimeRestore%3A!t%2Ctitle%3A'%5BFlights%5D%20Global%20Flight%20Dashboard'%2CviewMode%3Aview)&forceNow=2021-08-14T10%3A45%3A06.143Z | -

  | Aug 14, 2021 @ 10:49:17.000 | Error in browser console: { text: "Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.", url: https://0.0.0.0:18596/s/automation/app/dashboards#/view/7adfa750-4c81-11e8-b3d7-01146121b73d?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!f%2Cvalue%3A900000)%2Ctime%3A(from%3Anow-24h%2Cto%3Anow))&_a=(description%3A'Analyze%20mock%20flight%20data%20for%20ES-Air%2C%20Logstash%20Airways%2C%20Kibana%20Airlines%20and%20JetBeats'%2Cfilters%3A!()%2CfullScreenMode%3A!f%2Coptions%3A(darkTheme%3A!f%2ChidePanelTitles%3A!f%2CuseMargins%3A!t)%2Cpanels%3A!((embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A7%2Ci%3A'1'%2Cw%3A32%2Cx%3A0%2Cy%3A0)%2Cid%3Aaeb212e0-4c84-11e8-b3d7-01146121b73d%2CpanelIndex%3A'1'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A('Average%20Ticket%20Price'%3A%230A50A1%2C'Flight%20Count'%3A%2382B5D8)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A12%2Ci%3A'3'%2Cw%3A23%2Cx%3A17%2Cy%3A7)%2Cid%3Ac8fc3d30-4c87-11e8-b3d7-01146121b73d%2CpanelIndex%3A'3'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A15%2Ci%3A'4'%2Cw%3A48%2Cx%3A0%2Cy%3A85)%2Cid%3A'571aaf70-4c88-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'4'%2Ctype%3Asearch%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(ES-Air%3A%23447EBC%2CJetBeats%3A%2365C5DB%2C'Kibana%20Airlines'%3A%23BA43A9%2C'Logstash%20Airways'%3A%23E5AC0E)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A12%2Ci%3A'5'%2Cw%3A17%2Cx%3A0%2Cy%3A7)%2Cid%3A'8f4d0c00-4c86-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'5'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A('Carrier%20Delay'%3A%235195CE%2C'Late%20Aircraft%20Delay'%3A%231F78C1%2C'NAS%20Delay'%3A%2370DBED%2C'No%20Delay'%3A%23BADFF4%2C'Security%20Delay'%3A%23052B51%2C'Weather%20Delay'%3A%236ED0E0)))%2CgridData%3A(h%3A14%2Ci%3A'6'%2Cw%3A24%2Cx%3A24%2Cy%3A33)%2Cid%3Af8290060-4c88-11e8-b3d7-01146121b73d%2CpanelIndex%3A'6'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A14%2Ci%3A'7'%2Cw%3A24%2Cx%3A24%2Cy%3A19)%2Cid%3Abcb63b50-4c89-11e8-b3d7-01146121b73d%2CpanelIndex%3A'7'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(Count%3A%231F78C1)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A12%2Ci%3A'10'%2Cw%3A24%2Cx%3A0%2Cy%3A35)%2Cid%3A'9886b410-4c8b-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'10'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(Count%3A%231F78C1)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'13'%2Cw%3A14%2Cx%3A10%2Cy%3A19)%2Cid%3A'76e3c090-4c8c-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'13'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(colors%3A(Count%3A%231F78C1)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'14'%2Cw%3A14%2Cx%3A10%2Cy%3A27)%2Cid%3A'707665a0-4c8c-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'14'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2CmapCenter%3A!(27.421687059550266%2C15.371002131141724)%2CmapZoom%3A1)%2CgridData%3A(h%3A15%2Ci%3A'18'%2Cw%3A24%2Cx%3A24%2Cy%3A70)%2Cid%3A'79e8ff60-4c8e-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'18'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A8%2Ci%3A'21'%2Cw%3A48%2Cx%3A0%2Cy%3A62)%2Cid%3A'293b5a30-4c8f-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'21'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A7%2Ci%3A'22'%2Cw%3A16%2Cx%3A32%2Cy%3A0)%2Cid%3A'129be430-4c93-11e8-b3d7-01146121b73d'%2CpanelIndex%3A'22'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2CmapCenter%3A!(42.19556096274418%2C9.536742995308601e-7)%2CmapZoom%3A1)%2CgridData%3A(h%3A15%2Ci%3A'23'%2Cw%3A24%2Cx%3A0%2Cy%3A70)%2Cid%3A'334084f0-52fd-11e8-a160-89cc2ad9e8e2'%2CpanelIndex%3A'23'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(defaultColors%3A('0%20-%2050'%3A'rgb(247%2C251%2C255)'%2C'100%20-%20150'%3A'rgb(107%2C174%2C214)'%2C'150%20-%20200'%3A'rgb(33%2C113%2C181)'%2C'200%20-%20250'%3A'rgb(8%2C48%2C107)'%2C'50%20-%20100'%3A'rgb(198%2C219%2C239)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'25'%2Cw%3A10%2Cx%3A0%2Cy%3A19)%2Cid%3Af8283bf0-52fd-11e8-a160-89cc2ad9e8e2%2CpanelIndex%3A'25'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(defaultColors%3A('0%20-%2050'%3A'rgb(247%2C251%2C255)'%2C'100%20-%20150'%3A'rgb(107%2C174%2C214)'%2C'150%20-%20200'%3A'rgb(33%2C113%2C181)'%2C'200%20-%20250'%3A'rgb(8%2C48%2C107)'%2C'50%20-%20100'%3A'rgb(198%2C219%2C239)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A8%2Ci%3A'27'%2Cw%3A10%2Cx%3A0%2Cy%3A27)%2Cid%3A'08884800-52fe-11e8-a160-89cc2ad9e8e2'%2CpanelIndex%3A'27'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A()%2Cvis%3A(defaultColors%3A('0%20-%2011'%3A'rgb(247%2C251%2C255)'%2C'11%20-%2022'%3A'rgb(208%2C225%2C242)'%2C'22%20-%2033'%3A'rgb(148%2C196%2C223)'%2C'33%20-%2044'%3A'rgb(74%2C152%2C201)'%2C'44%20-%2055'%3A'rgb(23%2C100%2C171)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A15%2Ci%3A'28'%2Cw%3A24%2Cx%3A0%2Cy%3A47)%2Cid%3Ae6944e50-52fe-11e8-a160-89cc2ad9e8e2%2CpanelIndex%3A'28'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A6%2Ci%3A'29'%2Cw%3A8%2Cx%3A40%2Cy%3A7)%2Cid%3A'01c413e0-5395-11e8-99bf-1ba7b1bdaa61'%2CpanelIndex%3A'29'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A6%2Ci%3A'30'%2Cw%3A8%2Cx%3A40%2Cy%3A13)%2Cid%3A'2edf78b0-5395-11e8-99bf-1ba7b1bdaa61'%2CpanelIndex%3A'30'%2Ctype%3Avisualization%2Cversion%3A'7.14.0')%2C(embeddableConfig%3A(enhancements%3A())%2CgridData%3A(h%3A15%2Ci%3A'31'%2Cw%3A24%2Cx%3A24%2Cy%3A47)%2Cid%3Aed78a660-53a0-11e8-acbd-0be0ad9d822b%2CpanelIndex%3A'31'%2Ctype%3Avisualization%2Cversion%3A'7.14.0'))%2Cquery%3A(language%3Alucene%2Cquery%3A'')%2Ctags%3A!()%2CtimeRestore%3A!t%2Ctitle%3A'%5BFlights%5D%20Global%20Flight%20Dashboard'%2CviewMode%3Aview)&forceNow=2021-08-14T10%3A45%3A06.143Z } | -

  | Aug 14, 2021 @ 10:49:17.000 | 140536031156096:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46 | error

  | Aug 14, 2021 @ 10:49:18.000 | 140536031156096:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46 | error

  | Aug 14, 2021 @ 10:49:28.000 | handled 148 page requests | -

  | Aug 14, 2021 @ 10:50:07.000 | taking screenshots | -

  | Aug 14, 2021 @ 10:50:07.000 | timeRange: Aug 13, 2021 @ 10:49:31.020 to Aug 14, 2021 @ 10:49:31.022 | -

  | Aug 14, 2021 @ 10:51:13.000 | Request uses Space ID: automation | -

  | Aug 14, 2021 @ 10:51:13.000 | Scheduled PNG reporting task. Task ID: task:8ac6dfb0-fced-11eb-a15a-6501ed9b2077. Report ID: ksbnsflg001e20f75853dklg | -

  | Aug 14, 2021 @ 10:51:16.000 | Error: Max attempts (3) reached for job ksbnkkkd001e20f7588a9op9. Failed with: TimeoutError: Timeout has occurred     at Object.run (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:408:58)     at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:267:22) | -

  | Aug 14, 2021 @ 10:51:16.000 | Cancelling job ksbnkkkd001e20f7588a9op9... | -

  | Aug 14, 2021 @ 10:51:16.000 | Internal Server Error | error

  | Aug 14, 2021 @ 10:51:16.000 | Cancelling task report:execute "437f7f90-fced-11eb-a15a-6501ed9b2077" as it expired at 2021-08-14T10:51:16.094Z after running for 02m 00s (with timeout set at 120s).

Error: Internal Server Error
    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:128:19)
    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:82:19)
    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)

@tsullivan
Copy link
Member

tsullivan commented Aug 17, 2021 •
edited
Loading

@liza-mae

  • 8GB of RAM seems fine
  • What are the steps in the automation that run the upgrade and test reporting?
  • Are there pending jobs in the Reporting queue prior to the upgrade or anything like that?
  • Are the reports manually generated, or is there a scripted request being made to a POST URL?
  • Are the tests in question any of the x-pack/test functional / API integration tests?
[00:28:12]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:13]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:15]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:16]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:18]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending
[00:28:19]           │ debg Report at path /api/reporting/jobs/download/ksbnkkkd001e20f7588a9op9 is pending

This sample of logs shows that a report was requested to be generated, but isn't getting executed. Task Manager be losing the task, or possibly marking the task as failed for some reason.

  | Aug 14, 2021 @ 10:50:07.000 | taking screenshots | -
  | Aug 14, 2021 @ 10:50:07.000 | timeRange: Aug 13, 2021 @ 10:49:31.020 to Aug 14, 2021 @ 10:49:31.022 | -
  | Aug 14, 2021 @ 10:51:13.000 | Request uses Space ID: automation | -
  | Aug 14, 2021 @ 10:51:13.000 | Scheduled PNG reporting task. Task ID: task:8ac6dfb0-fced-11eb-a15a-6501ed9b2077. Report ID: ksbnsflg001e20f75853dklg | -
  | Aug 14, 2021 @ 10:51:16.000 | Error: Max attempts (3) reached for job ksbnkkkd001e20f7588a9op9. Failed with: TimeoutError: Timeout has occurred     at Object.run (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:408:58)     at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:267:22) | -

It looks like the report execution got stuck for over a minute after logging taking screenshots. What were the dimensions of the PNG? If it is a very large image, Kibana can get stuck handling the buffer and converting it to base64 and adding it to a document for the update call to ES.

cc @dokmic @jloleysens

@exalate-issue-sync exalate-issue-sync bot added impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort and removed impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort loe:medium Medium Level of Effort labels Aug 17, 2021
@exalate-issue-sync exalate-issue-sync bot added impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. and removed impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. labels Sep 27, 2021
@tsullivan
Copy link
Member

Duplicate of #106873

@tsullivan tsullivan marked this as a duplicate of #106873 Sep 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Fixes for quality problems that affect the customer experience (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead feedback_needed impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tsullivan @elasticmachine @liza-mae