Improve data plugin performance

[mshustov]

The data plugin, being one of the most fundamental plugins in the Kibana ecosystem, is the critical part of improving the overall Kibana performance.

Network

If you run Kibana with APM enabled (see instruction), you can notice that the plugin performs some amount of duplicated network requests, which can be optimized:

Note: I've got these numbers running Kibana and Elasticsearch locally - the network latency might be even higher on Cloud.

Code

• re-use API created by the core.
  

  kibana/src/plugins/data/server/search/search_service.ts

  Lines 243 to 255 in b3430e3

  	const esClient = elasticsearch.client.asScoped(request);
  	const savedObjectsClient = savedObjects.getScopedClient(request);
  	const scopedIndexPatterns = await indexPatterns.indexPatternsServiceFactory(
  	savedObjectsClient,
  	esClient.asCurrentUser
  	);
  	const uiSettingsClient = uiSettings.asScopedToClient(savedObjectsClient);
  	// cache ui settings, only including items which are explicitly needed by SearchSource
  	const uiSettingsCache = pick(
  	await uiSettingsClient.getAll(),
  	searchSourceRequiredUiSettings
  	);

  here RequestHandlerContext can be used.
• extend RequestHandlerContext with lazy instantiated API

  kibana/src/core/server/core_route_handler_context.ts

  Lines 32 to 58 in bc8a1da

  	class CoreElasticsearchRouteHandlerContext {
  	#client?: IScopedClusterClient;
  	#legacy?: {
  	client: Pick<LegacyScopedClusterClient, 'callAsInternalUser' | 'callAsCurrentUser'>;
  	};
  	constructor(
  	private readonly elasticsearchStart: InternalElasticsearchServiceStart,
  	private readonly request: KibanaRequest
  	) {}
  	public get client() {
  	if (this.#client == null) {
  	this.#client = this.elasticsearchStart.client.asScoped(this.request);
  	}
  	return this.#client;
  	}
  	public get legacy() {
  	if (this.#legacy == null) {
  	this.#legacy = {
  	client: this.elasticsearchStart.legacy.client.asScoped(this.request),
  	};
  	}
  	return this.#legacy;
  	}
  	}

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[lukeelmers]

here RequestHandlerContext can be used

Awhile back there was a discussion around avoiding using RequestHandlerContext in cases like this because of some of the complexities it creates for folks authenticating via API keys (for example alerting would need to not only generate a fake KibanaRequest to create an alert that relies on the search service, but now also a fake RequestHandlerContext).

However, performance did not come up as a potential issue in that discussion.

@streamich has suggested a caching layer in the data plugin's services. Another option would be to go down the path of restructuring everything to accept scoped services, which was an alternative approach proposed in that thread.

[lukeelmers]

Also on the same topic of data plugin performance, there's this related issue about whether we can further decrease the page load bundle size: #84788

[lizozom]

@lukeelmers @mshustov

This issue has been without activity for a while now.
Is it still relevant? Can it be made more actionable?
If it is releveant, we can add it to the Kibana Performance project and groom it.

[mshustov]

Is it still relevant? Can it be made more actionable?

It's on AppServices team to answer these questions as they own data plugin

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)