APM - apm-source-map index, cluster in yellow state

[mciricean]

Kibana version: 8.10.4

Elasticsearch version: 8.10.4

Server OS version: rockylinux8.7

Browser version: chrome 121.0.6167.85

Browser OS version: ubuntu 22.04

Original install method (e.g. download page, yum, from source, etc.):
docker

Describe the bug:

when installing kibana the es cluster is in yellow state, because of .apm-source-map index uses 2 replicas and one shard cannot be allocated

my cluster has 2 servers (VMs) each with 2 es nodes running on them.

ip        heap.percent ram.percent ram.current cpu load_15m node.role master name       version port disk.used disk.avail query_cache.hit_count query_cache.miss_count
10.9.8.11           50          96       7.2gb   7     1.05 dim       *      server1-x0 8.10.4  9300    38.8gb      1.1gb                     0                      0
10.9.8.11           30          96       7.2gb   7     1.05 dit       -      server1-d1 8.10.4  9301     1.4gb     25.6gb                     0                      0
10.9.8.12           45          97       7.3gb  24     1.78 dim       -      server2-x0 8.10.4  9300    36.8gb      3.1gb                     0                      0
10.9.8.12           20          97       7.3gb  24     1.78 dit       -      server2-d1 8.10.4  9301     1.4gb     25.6gb                     0                      0

I have cluster.routing.allocation.same_shard.host: true to replicate the shards on the other hosts (or using cluster.routing.allocation.awareness.attributes: rack_id i get same behavior ) and i cannot have 2 replicas for this index in this case.

GET _cat/shards?v&s=index

index                                                         shard prirep state      docs   store ip        node
.apm-agent-configuration                                      0     r      STARTED       0    248b 10.9.8.11 server1-d1
.apm-agent-configuration                                      0     p      STARTED       0    248b 10.9.8.12 server2-d1
.apm-custom-link                                              0     p      STARTED       0    248b 10.9.8.11 server1-d1
.apm-custom-link                                              0     r      STARTED       0    248b 10.9.8.12 server2-d1
.apm-source-map                                               0     p      STARTED       0    248b 10.9.8.11 server1-d1
.apm-source-map                                               0     r      STARTED       0    248b 10.9.8.12 server2-d1
.apm-source-map                                               0     r      UNASSIGNED                        
.ds-.kibana-event-log-8.10.4-2024.01.28-000001                0     p      STARTED       1   6.2kb 10.9.8.11 server1-d1
.ds-.kibana-event-log-8.10.4-2024.01.28-000001                0     r      STARTED       1   6.3kb 10.9.8.12 server2-d1
...

I've noticed the auto_expand_replicas is not correct for this case "auto_expand_replicas": "0-2" ; as a workaround I'm just resetting this index config.
PUT /.apm-source-map/_settings
{
"index": {
"auto_expand_replicas": "0-1"
}
}

is there a kibana.yml config that i can use to set the value auto_expand_replicas for apm-source-map index, during installation ? or avoid creating this index /disable APM ?

Steps to reproduce:
see above

1. install es cluster with 2 hosts, 2 nodes each
2. enable cluster.routing.allocation.same_shard.host: true
3. install/start kibana
4. check cluster health, will be in yellow state, check shard allocation status for apm-source-map index

Expected behavior:

New install with this setup should work without updating the index config

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[yomduf]

Thanks for reporting it
This is fixed in 8.13 => #175317