Skip to content

Commit

Permalink
[ES] Upgrade client to v8.0 (#113950)
Browse files Browse the repository at this point in the history
* bump to a pre-8.0 version

* export KibanaClient from /lib sub-folder

* workaround the problem of the absence of estypes

* update es client usage in pacakges

* export estypes from another path

* import errors from root

* import errors from root 2

* update transport import

* update import path for /api/types

* update import path for /api/types

* import errors from top export

* use TransportResult instead if ApiResponse

* fix errors in client_config

* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts

* use KibanaClient in mock. we dont export the original Client

* fix client mocks

* fix errors on SO

* fix remaining core errors

* update estype import path

* fix errors in data plugin

* fix data_views

* fix es_ui_shared

* fix errors in interactive_setup

* fix errors in ./test folder

* add @elastic/transport to the runtime deps

* fix errors in packages

* fix erros in src/core

* fix errors in test/

* fix an error in actions plugin

* woraround and fix errors in APM plugin

* fix errors in canvas

* fix errors in event_log

* fix errors in fleet

* fix errors in ILM

* fix errors in infra

* fix errors in ingest_pipeline

* fix errors in lens

* fix errors in license_management

* fix errors in licensing

* fix errors in logstash

* fix errors in ml

* fix errors in monitoring

* fix errors in observability

* fix errors in rule_registry

* fix errors in reporting

* fix errors in rule_registry

* fix errors in security

* fix errors in security_solution

* fix errors in snapshot_restore

* fix errors in transform

* fix errors in UA

* fix errors in uptime

* fix errors in x-pack/test

* fix eslint errors

* fix new errors

* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets

* create does not accept require_alias option

* update deps

* use transport types exported from ES client package

* fix ErrorCause | string errors

* do not use enum

* fix errors in data plugin

* update x-pack code

* fix transport

* fix apm search request

* do not crash on reporting

* fix kbn-test build

* mute reporting error to start

* fix ftr build

* another attempt

* update import path

* address or mute new errors

* REMOVE me. pin transport version temporarily.

* remove deep imports from transport package

* fix jest crash

* fix product check tests

* remove unnecessary ts-expect-error

* fix a few failed unit tests

* bump to canary 24

* remove unnecessary ts-expect-error

* remove dependency on transport

* fix types in tests

* mute errors in xpack tests

* product check doesn;t  spam in logs anymore

* filterPath --> filter_path

* ignoreUnavailable --> ignore_unavailable

* ignoreUnavailable --> ignore_unavailable

* trackScores --> track_scores

* trackTotalHits --> track_total_hits

* fix es-arcives

* fix data plugin crashes

* fix watcher test utils

* rollback unnecessary changes

* fix another problem in es-archiver

* fix scroll. for whatever reason scroll fails when request scroll_id in body

* add meta: true in kbn-securitysolution-es-utils

* bump client to canary 25

* fix errors in accordance with the es client spec

* update securityscolution-es-utils

* unify scroll api in reporting and fix tests

* fix unit tests in watcher

* refactor APM to abort request with AbortController API

* fix missing es client calls in tests

* fix missing meta in detection engine FTR tests

* fix another bunch of errors in js tests

* fix wrong coercion

* remove test-grep pattern

* fix apm unit test

* rename terminateAfter to terminate_after in infra plugin

* rename terminateAfter to terminate_after in uptime plugin

* rename terminateAfter to terminate_after in apm plugin

* fix security roles FTR tests

* fix reference

* fix post_privilidges test

* fix post_privilidges

* bump client to 26

* add meta for index_management test helpers

* remove ts-expect-error caused by bad type in reason

* bump client to 27

* REMOVE me. workaround until fixed in the es client

* fix incorrect type casting

* swtich from camelCase params

* use `HttpConnection` for FTR-related clients

* bump client to 29

* Revert "REMOVE me. workaround until fixed in the es client"

This reverts commit c038850.

* fix new util

* revert repository changes

* do not crash if cannot store event_loop data

* fix new estypes imports

* fix more types

* fix security test types and add ts-ignore for custom ES client

* fix more estypes imports

* yet more ts violations

* line by line fixing is hard

* adapt `evaluateAlert` from infra as it's also used from FTR tests

* use convertToKibanaClient in FTR test instead of meta:true in plugin code

* migrate from deprecated API in fleet

* fix intergration tests

* fix fleet tests

* fix another fleet test

* fix more tests

* let's call it a day

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional

* Updates proxy response integration test

* disable APM until compatible with client v8

* skip async_search FTR test

* use kbnClient in integration tests

* bump version to 29

* bump to 30

* have configureClient return a KibanaClient instead of Client, remove resolved violations.

* bump to 31

* bump to 31

* Revert "bump to 31"

This reverts commit 5ac713e.

* trigger stop to unusubscribe

* update generated docs

* remove obsolete test

* put "as" back

* cleanup

* skip test

* remove new type errors in apm package

* remove ErrorCause casting

* update a comment

* bump version to 32

* remove unnecessary ts-expect-error in apm code

* update comments

* update to client v33

* remove outdated type definition

* bump to 34 without params mutation

* unskip the test that should not fail anymore

* remove unnecessary ts-expect-error comments

* update to v35. body can be string

* move `sort` to body and use body friendly syntax

* fix a failing test. maps register the same SO that has been already registered by home

Co-authored-by: pgayvallet <[email protected]>
Co-authored-by: Christiane (Tina) Heiligers <[email protected]>
  • Loading branch information
3 people authored Oct 26, 2021
1 parent 252aef8 commit 3c8fa52
Show file tree
Hide file tree
Showing 814 changed files with 3,045 additions and 3,350 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ Client used to query the elasticsearch cluster.
<b>Signature:</b>

```typescript
export declare type ElasticsearchClient = Omit<KibanaClient, 'connectionPool' | 'transport' | 'serializer' | 'extend' | 'child' | 'close'> & {
export declare type ElasticsearchClient = Omit<KibanaClient, 'connectionPool' | 'transport' | 'serializer' | 'extend' | 'child' | 'close' | 'diagnostic'> & {
transport: {
request(params: TransportRequestParams, options?: TransportRequestOptions): TransportRequestPromise<ApiResponse>;
request<TResponse = unknown>(params: TransportRequestParams, options?: TransportRequestOptions): Promise<TransportResult<TResponse>>;
};
};
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [ElasticsearchErrorDetails](./kibana-plugin-core-server.elasticsearcherrordetails.md) &gt; [error](./kibana-plugin-core-server.elasticsearcherrordetails.error.md)

## ElasticsearchErrorDetails.error property

<b>Signature:</b>

```typescript
error?: {
type: string;
reason?: string;
};
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [ElasticsearchErrorDetails](./kibana-plugin-core-server.elasticsearcherrordetails.md)

## ElasticsearchErrorDetails interface


<b>Signature:</b>

```typescript
export interface ElasticsearchErrorDetails
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [error](./kibana-plugin-core-server.elasticsearcherrordetails.error.md) | <code>{</code><br/><code> type: string;</code><br/><code> reason?: string;</code><br/><code> }</code> | |

1 change: 1 addition & 0 deletions docs/development/core/server/kibana-plugin-core-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ The plugin integrates with the core system via lifecycle events: `setup`<!-- -->
| [DeprecationsServiceSetup](./kibana-plugin-core-server.deprecationsservicesetup.md) | The deprecations service provides a way for the Kibana platform to communicate deprecated features and configs with its users. These deprecations are only communicated if the deployment is using these features. Allowing for a user tailored experience for upgrading the stack version.<!-- -->The Deprecation service is consumed by the upgrade assistant to assist with the upgrade experience.<!-- -->If a deprecated feature can be resolved without manual user intervention. Using correctiveActions.api allows the Upgrade Assistant to use this api to correct the deprecation upon a user trigger. |
| [DiscoveredPlugin](./kibana-plugin-core-server.discoveredplugin.md) | Small container object used to expose information about discovered plugins that may or may not have been started. |
| [ElasticsearchConfigPreboot](./kibana-plugin-core-server.elasticsearchconfigpreboot.md) | A limited set of Elasticsearch configuration entries exposed to the <code>preboot</code> plugins at <code>setup</code>. |
| [ElasticsearchErrorDetails](./kibana-plugin-core-server.elasticsearcherrordetails.md) | |
| [ElasticsearchServicePreboot](./kibana-plugin-core-server.elasticsearchservicepreboot.md) | |
| [ElasticsearchServiceSetup](./kibana-plugin-core-server.elasticsearchservicesetup.md) | |
| [ElasticsearchServiceStart](./kibana-plugin-core-server.elasticsearchservicestart.md) | |
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ export declare class SavedObjectsErrorHelpers
| [createBadRequestError(reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.createbadrequesterror.md) | <code>static</code> | |
| [createConflictError(type, id, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.createconflicterror.md) | <code>static</code> | |
| [createGenericNotFoundError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.creategenericnotfounderror.md) | <code>static</code> | |
| [createGenericNotFoundEsUnavailableError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.creategenericnotfoundesunavailableerror.md) | <code>static</code> | |
| [createIndexAliasNotFoundError(alias)](./kibana-plugin-core-server.savedobjectserrorhelpers.createindexaliasnotfounderror.md) | <code>static</code> | |
| [createInvalidVersionError(versionInput)](./kibana-plugin-core-server.savedobjectserrorhelpers.createinvalidversionerror.md) | <code>static</code> | |
| [createTooManyRequestsError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.createtoomanyrequestserror.md) | <code>static</code> | |
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@
"@elastic/apm-rum-react": "^1.3.1",
"@elastic/charts": "38.0.1",
"@elastic/datemath": "link:bazel-bin/packages/elastic-datemath",
"@elastic/elasticsearch": "npm:@elastic/elasticsearch-canary@^8.0.0-canary.21",
"@elastic/elasticsearch": "npm:@elastic/elasticsearch-canary@^8.0.0-canary.35",
"@elastic/ems-client": "7.16.0",
"@elastic/eui": "40.0.0",
"@elastic/filesaver": "1.1.2",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ export async function cleanWriteTargets({
wait_for_completion: false,
});

const task = response.body.task;
const task = response.task;

if (task) {
await new Promise<void>((resolve, reject) => {
Expand All @@ -45,13 +45,13 @@ export async function cleanWriteTargets({
});

logger.debug(
`Polled for task:\n${JSON.stringify(taskResponse.body, ['completed', 'error'], 2)}`
`Polled for task:\n${JSON.stringify(taskResponse, ['completed', 'error'], 2)}`
);

if (taskResponse.body.completed) {
if (taskResponse.completed) {
resolve();
} else if (taskResponse.body.error) {
reject(taskResponse.body.error);
} else if (taskResponse.error) {
reject(taskResponse.error);
} else {
setTimeout(pollForTaskCompletion, 2500);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,15 @@ export async function getWriteTargets({
]);

function getDataStreamName(filter: string) {
return datastreamsResponse.body.data_streams.find((stream) => stream.name.includes(filter))
?.name;
return datastreamsResponse.data_streams.find((stream) => stream.name.includes(filter))?.name;
}

function getAlias(filter: string) {
return Object.keys(indicesResponse.body)
return Object.keys(indicesResponse)
.map((key) => {
return {
key,
writeIndexAlias: Object.entries(indicesResponse.body[key].aliases).find(
writeIndexAlias: Object.entries(indicesResponse[key].aliases).find(
([_, alias]) => alias.is_write_index
)?.[0],
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ export function uploadEvents({
)
.then((results) => {
const errors = results
.flatMap((result) => result.body.items)
.flatMap((result) => result.items)
.filter((item) => !!item.index?.error)
.map((item) => item.index?.error);

Expand Down
3 changes: 2 additions & 1 deletion packages/kbn-apm-config-loader/src/init_apm.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ import { mockLoadConfiguration } from './init_apm.test.mocks';
import { initApm } from './init_apm';
import apm from 'elastic-apm-node';

describe('initApm', () => {
// TODO: unskip when https://github.com/elastic/kibana/issues/116109 is fixed
describe.skip('initApm', () => {
let apmAddFilterSpy: jest.SpyInstance;
let apmStartSpy: jest.SpyInstance;
let getConfig: jest.Mock;
Expand Down
3 changes: 3 additions & 0 deletions packages/kbn-apm-config-loader/src/init_apm.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ export const initApm = (
isDistributable: boolean,
serviceName: string
) => {
// TODO: re-enabled when https://github.com/elastic/kibana/issues/116109 is fixed
return;

const apmConfigLoader = loadConfiguration(argv, rootDir, isDistributable);
const apmConfig = apmConfigLoader.getConfig(serviceName);

Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-es-archiver/src/actions/empty_kibana_index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
* Side Public License, v 1.
*/

import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import { ToolingLog } from '@kbn/dev-utils';
import { KbnClient } from '@kbn/test';

Expand All @@ -17,7 +17,7 @@ export async function emptyKibanaIndexAction({
log,
kbnClient,
}: {
client: KibanaClient;
client: Client;
log: ToolingLog;
kbnClient: KbnClient;
}) {
Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-es-archiver/src/actions/load.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { createReadStream } from 'fs';
import { Readable } from 'stream';
import { ToolingLog, REPO_ROOT } from '@kbn/dev-utils';
import { KbnClient } from '@kbn/test';
import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import { createPromiseFromStreams, concatStreamProviders } from '@kbn/utils';
import { ES_CLIENT_HEADERS } from '../client_headers';

Expand Down Expand Up @@ -47,7 +47,7 @@ export async function loadAction({
inputDir: string;
skipExisting: boolean;
useCreate: boolean;
client: KibanaClient;
client: Client;
log: ToolingLog;
kbnClient: KbnClient;
}) {
Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-es-archiver/src/actions/save.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import { resolve, relative } from 'path';
import { createWriteStream, mkdirSync } from 'fs';
import { Readable, Writable } from 'stream';
import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import { ToolingLog, REPO_ROOT } from '@kbn/dev-utils';
import { createListStream, createPromiseFromStreams } from '@kbn/utils';

Expand All @@ -31,7 +31,7 @@ export async function saveAction({
}: {
outputDir: string;
indices: string | string[];
client: KibanaClient;
client: Client;
log: ToolingLog;
raw: boolean;
query?: Record<string, any>;
Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-es-archiver/src/actions/unload.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import { resolve, relative } from 'path';
import { createReadStream } from 'fs';
import { Readable, Writable } from 'stream';
import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import { ToolingLog, REPO_ROOT } from '@kbn/dev-utils';
import { KbnClient } from '@kbn/test';
import { createPromiseFromStreams } from '@kbn/utils';
Expand All @@ -31,7 +31,7 @@ export async function unloadAction({
kbnClient,
}: {
inputDir: string;
client: KibanaClient;
client: Client;
log: ToolingLog;
kbnClient: KbnClient;
}) {
Expand Down
5 changes: 3 additions & 2 deletions packages/kbn-es-archiver/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import Fs from 'fs';

import { RunWithCommands, createFlagError, CA_CERT_PATH } from '@kbn/dev-utils';
import { readConfigFile, KbnClient } from '@kbn/test';
import { Client } from '@elastic/elasticsearch';
import { Client, HttpConnection } from '@elastic/elasticsearch';

import { EsArchiver } from './es_archiver';

Expand Down Expand Up @@ -106,7 +106,8 @@ export function runCli() {

const client = new Client({
node: esUrl,
ssl: esCa ? { ca: esCa } : undefined,
tls: esCa ? { ca: esCa } : undefined,
Connection: HttpConnection,
});
addCleanupTask(() => client.close());

Expand Down
6 changes: 3 additions & 3 deletions packages/kbn-es-archiver/src/es_archiver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import Fs from 'fs';
import Path from 'path';

import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import { ToolingLog, REPO_ROOT } from '@kbn/dev-utils';
import { KbnClient } from '@kbn/test';

Expand All @@ -23,14 +23,14 @@ import {
} from './actions';

interface Options {
client: KibanaClient;
client: Client;
baseDir?: string;
log: ToolingLog;
kbnClient: KbnClient;
}

export class EsArchiver {
private readonly client: KibanaClient;
private readonly client: Client;
private readonly baseDir: string;
private readonly log: ToolingLog;
private readonly kbnClient: KbnClient;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,8 @@ it('transforms each input index to a stream of docs using scrollSearch helper',
Array [
Object {
"_source": "true",
"body": Object {
"query": undefined,
},
"index": "bar",
"query": undefined,
"rest_total_hits_as_int": true,
"scroll": "1m",
"size": 1000,
Expand All @@ -116,10 +114,8 @@ it('transforms each input index to a stream of docs using scrollSearch helper',
Array [
Object {
"_source": "true",
"body": Object {
"query": undefined,
},
"index": "foo",
"query": undefined,
"rest_total_hits_as_int": true,
"scroll": "1m",
"size": 1000,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
*/

import { Transform } from 'stream';
import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import { Stats } from '../stats';
import { Progress } from '../progress';
import { ES_CLIENT_HEADERS } from '../../client_headers';
Expand All @@ -21,7 +21,7 @@ export function createGenerateDocRecordsStream({
progress,
query,
}: {
client: KibanaClient;
client: Client;
stats: Stats;
progress: Progress;
query?: Record<string, any>;
Expand All @@ -37,9 +37,7 @@ export function createGenerateDocRecordsStream({
scroll: SCROLL_TIMEOUT,
size: SCROLL_SIZE,
_source: 'true',
body: {
query,
},
query,
rest_total_hits_as_int: true,
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@
* Side Public License, v 1.
*/

import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import AggregateError from 'aggregate-error';
import { Writable } from 'stream';
import { Stats } from '../stats';
import { Progress } from '../progress';
import { ES_CLIENT_HEADERS } from '../../client_headers';

export function createIndexDocRecordsStream(
client: KibanaClient,
client: Client,
stats: Stats,
progress: Progress,
useCreate: boolean = false
Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-es-archiver/src/lib/indices/__mocks__/stubs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
* Side Public License, v 1.
*/

import type { KibanaClient } from '@elastic/elasticsearch/api/kibana';
import type { Client } from '@elastic/elasticsearch';
import sinon from 'sinon';
import { ToolingLog } from '@kbn/dev-utils';
import { Stats } from '../../stats';
Expand Down Expand Up @@ -67,7 +67,7 @@ const createEsClientError = (errorType: string) => {
const indexAlias = (aliases: Record<string, any>, index: string) =>
Object.keys(aliases).find((k) => aliases[k] === index);

type StubClient = KibanaClient;
type StubClient = Client;

export const createStubClient = (
existingIndices: string[] = [],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ describe('esArchiver: createCreateIndexStream()', () => {
"ignore": Array [
404,
],
"meta": true,
},
],
]
Expand Down
Loading

0 comments on commit 3c8fa52

Please sign in to comment.