Hosts+ 5000 hosts but summary shows 4939

[pjbertels]

** Cluster **
ess_deployment_id: 896c3c84f4505413154dd5767c30f4fc
ess_cloud_id: dev-oblt:dXMtd2VzdDIuZ2NwLmVsYXN0aWMtY2xvdWQuY29tOjQ0MyQwZDUwZDcyMDEwNGM0NjNiOWMxYWZmNjJhMmYyNzg1ZCRiYmUyOWNlMmFmNmQ0ZjE0ODViZDg1NTdjNjU0YzIwMA==
stack_mode: ess
stack_version: 8.14.2-SNAPSHOT
stack_target: production
stack_template: observability
k8s_default_namespace: default
k8s_project: elastic-observability
k8s_provider: gcp
k8s_region: us-central1-c

Use oblt-cli --disable-banner cluster secrets cluster-state --cluster-name dev-oblt to get login credentials.

Kibana version:
: 8.14.2-SNAPSHOT
Elasticsearch version:
: 8.14.2-SNAPSHOT
Server OS version:

Browser version:
Chrome Version 125.0.6422.142 (Official Build) (64-bit)

Browser OS version:
Windows 11

Original install method (e.g. download page, yum, from source, etc.):
dev-oblt obli-cil installed

Describe the bug:
I added 5000 fake hostnames, I added 10,000 docs (5000 system_cpu, 5000 system_memory)and discovery confirms they are there as does the response to my _bulk update.
Hosts reports 4939 hosts not 5000 as I would expect.

Steps to reproduce:

1. Add 5000 hosts
2. Check discovery
3. Confirm with Hosts +

Expected behavior:
5000 hosts in the summary pane

Screenshots (if relevant):

Discover confirms the hosts I added 10,000 docs (5000 system_cpu, 5000 system_memory)

Why 4939 hosts?

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[pjbertels]

Link to the Hosts page: https://dev-oblt.kb.us-west2.gcp.elastic-cloud.com/app/metrics/hosts?_a=(dateRange:(from:now-120m,to:now),filters:!(),limit:500,panelFilters:!(),query:(language:kuery,query:%27agent.ephemeral_id%20:%20deadbeef*%20and%20host.name%20:%20gke-test06*%20%27))&controlPanels=(cloud.provider:(explicitInput:(fieldName:cloud.provider,id:cloud.provider,title:%27Cloud%20Provider%27),grow:!f,order:1,type:optionsListControl,width:medium),host.os.name:(explicitInput:(fieldName:host.os.name,id:host.os.name,title:%27Operating%20System%27),grow:!f,order:0,type:optionsListControl,width:medium),service.name:(explicitInput:(fieldName:service.name,id:service.name,title:%27Service%20Name%27),grow:!f,order:2,type:optionsListControl,width:medium))

[pjbertels]

Inventory says there are 5000 ...

[pjbertels]

_bulk update also confirms there are 5000 docs added to each datastream approximately every 60 seconds.

b'2024-06-24T17:15:46.995Z\n'
/home/pbertels/generous/workspaces/system/system_cpu_rt.json
/home/pbertels/generous//bin/synthesize.sh metrics-system.cpu-default system_cpu_rt /tmp/test_synth/pump NOFILE
/home/pbertels/generous/workspaces/system/system_memory_rt.json
/home/pbertels/generous//bin/synthesize.sh metrics-system.memory-default system_memory_rt /tmp/test_synth/pump NOFILE
PWD=/tmp/test_synth/gen
OLDPWD=/home/pbertels/generous
/tmp/test_synth/gen
synth generate /tmp/test_synth/pump --random --collection system_cpu_rt --to jsonl: | sed 's/^/{"create":{"_index":"metrics-system.cpu-default"}}\n/' | tsbs_load_elasticsearch --batch-size 30000 --workers 4 > /tmp/test_synth/pump/results/LOAD_system_cpu_rt 2>&1
Summary:
loaded 5000 metrics in 9.798sec with 4 workers (mean rate 510.32 metrics/sec)
PWD=/tmp/test_synth/gen
OLDPWD=/home/pbertels/generous
/tmp/test_synth/gen
synth generate /tmp/test_synth/pump --random --collection system_memory_rt --to jsonl: | sed 's/^/{"create":{"_index":"metrics-system.memory-default"}}\n/' | tsbs_load_elasticsearch --batch-size 30000 --workers 4 > /tmp/test_synth/pump/results/LOAD_system_memory_rt 2>&1
Summary:
loaded 5000 metrics in 8.628sec with 4 workers (mean rate 579.51 metrics/sec)

[pjbertels]

Based on recent testing of 500, 5000 and 10K generated hosts, this issue can be closed. I had an issue in data generation using generous where duplicate host names could be created. The issue was only apparent at bulk insert time. I have resolved this issue in generous and have not found this count discrepancy when testing against the dev-oblt cluster.