[Stack Monitoring] Improve error message for getKibanaInfo

[simianhacker]

The current error message from the getKibanaInfo() when Elasticsearch returns ZERO results is unhelpful; it will return this error message:

Monitoring Request Error Missing required parameter of field: timestamp: Missing required parameter of field: timestamp

This error doesn't indicate the real problem which is the query returned ZERO results. We should add a check to ensure there are results and when it fails we should return a meaningful error message like:

Monitoring Request Error: The query for Kibana state using "${clusterUuid}" cluster UUID and "${kibanaUuid}" Kibana UUID returned zero results. There needs to be at least one record returned for this query."

The offending code lives here:

kibana/x-pack/plugins/monitoring/server/lib/kibana/get_kibana_info.ts

Lines 19 to 24 in 4d47cb7

	const legacySource = hit?._source.kibana_stats;
	const mbSource = hit?._source.kibana?.stats;
	const availabilityTimestamp = hit?._source['@timestamp'] ?? legacySource?.timestamp;
	if (!availabilityTimestamp) {
	throw new MissingRequiredError('timestamp');
	}

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[lucabelluccini]

The problem affects any get_<product>_info. E.g. Logstash (

kibana/x-pack/plugins/monitoring/server/lib/logstash/get_node_info.ts

Line 20 in 4d47cb7

const availabilityTimestamp = mbStats?.timestamp ?? legacyStats?.timestamp;

).

The problem can be triggered by the presence of metricbeat-* indices with incorrect mappings, as Stack Monitoring queries .monitoring-*,metricbeat-* by default (monitoring.ui.metricbeat.indices allows to customize the set of Metricbeat indices to use).

To identify the problematic indices, one could run GET .monitoring-*,metricbeat-*/_mapping/field/timestamp and check if there are indices with such field and the type is not date.