[Dashboard] Time shift error on dashboards

[jsanz]

Reported at this discuss thread.

Kibana version:
Reproduced in main, 8.8.1, and 8.9.0-SNAPSHOT

Elasticsearch version:
Same

Server OS version:
Elastic Cloud for 8.8.1 and Ubuntu 22.04 for main and 8.9.0-SNAPSHOT

Browser version:
Firefox 115.0.2 (64-bit)

Browser OS version:
Ubuntu 22.04

Original install method (e.g. download page, yum, from source, etc.):
Elastic Cloud for 8.8.1, from sources for main, and Docker for 8.9.0-SNAPSHOT

Describe the bug:
Kibana global time range is not correctly applied to visualizations that include a time shift parameter.

Steps to reproduce:

Create the following index that adds two data points per year from 2019 to 2023, increasing 10 units per year.

Data to reproduce

PUT discuss-337273
{
  "mappings": {
    "properties": {
      "timestamp": {"type": "date"},
      "metric": {"type": "float"}
    }
  }
}

POST discuss-337273/_bulk
{ "index": {}}
{ "timestamp": "2019-07-20", "metric": 10 }
{ "index": {}}
{ "timestamp": "2019-07-20", "metric": 10 }
{ "index": {}}
{ "timestamp": "2020-07-20", "metric": 20 }
{ "index": {}}
{ "timestamp": "2020-07-20", "metric": 20 }
{ "index": {}}
{ "timestamp": "2021-07-20", "metric": 30 }
{ "index": {}}
{ "timestamp": "2021-07-20", "metric": 30 }
{ "index": {}}
{ "timestamp": "2022-07-20", "metric": 40 }
{ "index": {}}
{ "timestamp": "2022-07-20", "metric": 40 }
{ "index": {}}
{ "timestamp": "2023-07-20", "metric": 50 }
{ "index": {}}
{ "timestamp": "2023-07-20", "metric": 50 }

Import the following Dashboard and Data view: export.zip

Additionally you can also configure these customer ranges for the date picker:

Custom time ranges

[
  {
    "from": "2019-01-01",
    "to": "2020-01-10",
    "display": "2019"
  },
  {
    "from": "2020-01-01",
    "to": "2021-01-10",
    "display": "2020"
  },
  {
    "from": "2021-01-01",
    "to": "2022-01-10",
    "display": "2021"
  },
  {
    "from": "2022-01-01",
    "to": "2023-01-10",
    "display": "2022"
  },
  {
    "from": "2023-01-01",
    "to": "2024-01-10",
    "display": "2023"
  }
]

It has three panels that present the elements of a fourth panel with a ratio.

The main panel on the right has all the elements of this "formula" as follows:

(average(metric) - average(metric, shift='startAt(2018-12-31T23:00:00.000Z)'))
/
(average(metric, shift='startAt(2018-12-31T23:00:00.000Z)'))

So when filtering 2023 (average 50), the ratio is a growth of 400%, when filtering 2022, it should be 300%, and so on.

This works well inside the Lens editor, but on the Dashboard is not working correctly.

Expected behavior:
Correct results in Dashboard.

Screenshots (if relevant):

vokoscreenNG-2023-07-20_14-55-09.webm

The user contributed a video here

These are screenshots of the time range fetched from the browser for a request for 2021:

In Lens	In Dashboard

Errors in browser console (if relevant):
Nothing I could see.

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[ThomThomson]

Just commenting here that this is still on our radar.

[JoshStage]

This issue has a pretty large impact on our reporting, as we often times are reporting things between this year (TY) and last year (LY) and having this issue makes the graphs not work at all.

We were originally on 8.5.3 and upgraded recently to 8.11.3, the fact that this bug is still around after this many months is embarrassing, considering dashboard reporting is a large part of the Kibana appeal.

Dashboarding is one of the primary focuses of Kibana, I do not understand how dashboards reporting incorrect data would be considered only medium in terms of impact.

[ThomThomson]

@elastic/kibana-visualizations I believe the Lens editor is based on very different code than the Lens Embeddable. Is there a chance that this issue stems from different implementations there?

I'm not convinced that this could be an issue from the Dashboard side, as we provide only the time range from the Dashboard, and it's up to the individual Embeddable to determine what to do with it.

Reassigning, but if it's discovered that this is coming from the Dashboard directly, feel free to reassign back to us.

[dej611]

I've tracked down the bug here, and it is another instance of #170762

Basically the embeddable is compiling into an expression but the absolute shift is not updated again in the expression as the "relative" conversion happens at compiling time rather than runtime. This is an optimisation within the Lens embeddable instance to avoid computing too much.
When in the editor the expression gets compiled at every change so the problem won't appear, hence the visualization works fine there.

A proposed solution would be to push down the absolute shift resolution at expression time.