Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[E2E] Upgrading to Kibana 7.17.21 may fail #7953

Closed
barkbay opened this issue Jul 23, 2024 · 1 comment
Closed

[E2E] Upgrading to Kibana 7.17.21 may fail #7953

barkbay opened this issue Jul 23, 2024 · 1 comment
Assignees
Labels
>test Related to unit/integration/e2e tests v2.14.0

Comments

@barkbay
Copy link
Contributor

barkbay commented Jul 23, 2024

Likely related to elastic/kibana#158182 which is not available in 7.x until 7.17.23.

Kibana Pod never comes ready, following error appears in the logs:

{"type":"log","@timestamp":"2024-07-22T17:19:10+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] Migration completed after 319ms"}
{"type":"log","@timestamp":"2024-07-22T17:21:10+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2024-07-22T17:21:10+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 120008ms."}
...
{"type":"log","@timestamp":"2024-07-22T17:47:32+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 11 in 64 seconds."}
{"type":"log","@timestamp":"2024-07-22T17:47:32+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 184003ms."}
==== END logs for e2e-kktsf-mercury/test-version-upgrade-to-7x-tpcr-kb-599484746c-zw4kh ====

Kibana is actually waiting for some shards to be assigned:

.kibana_task_manager_7.17.21_reindex_temp                     0     p      STARTED       0    227b 10.37.129.117 test-version-upgrade-to-7x-g597-es-masterdata-0
.kibana_task_manager_7.17.21_reindex_temp                     0     r      UNASSIGNED  

Which is never going to happen since tests like TestVersionUpgradeToLatest7x or TestVersionUpgradeAndRespecToLatest7x are run with a 1 node ES cluster.

One workaround would be to add a version gate to create ES clusters with at least 2 nodes (credits to @pebrc for the idea)

@barkbay barkbay added >test Related to unit/integration/e2e tests v2.14.0 labels Jul 23, 2024
@barkbay barkbay self-assigned this Jul 23, 2024
@barkbay
Copy link
Contributor Author

barkbay commented Jul 23, 2024

Workaround has been merged in main and 2.14, closing.

@barkbay barkbay closed this as completed Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>test Related to unit/integration/e2e tests v2.14.0
Projects
None yet
Development

No branches or pull requests

1 participant