[BUG] Dashboards-maps: documents size limit

[cbeaujoin-stellar]

What is the bug?

As described here: #141

Update default query document size limit to 1000, the max is from index.max_result_window size,

But in fact limitation is set to 10000 and is not aligned with index.max_result_window:

dashboards-maps/public/components/layer_config/documents_config/document_layer_source.tsx

Line 212 in 811e8b0

setHasInvalidRequestNumber(

How can one reproduce the bug?

Create a map based on an index with number of documents and index.max_result_window greather than 10000.
You can't set a Number of Document greater than 10k.

What is the expected behavior?

UI should be aligned with the spec:

• Limited to the index index.max_result_window

Idealy, maybe the maps plugin should use the scroll API when index.max_result_window > 10k

What is your host/environment?

docker, v2.11.1

Do you have any screenshots?

Do you have any additional context?

https://forum.opensearch.org/t/dashboards-maps-documents-size-limit/17983

[junqiu-lei]

@cbeaujoin-stellar Thanks reporting, DOCUMENTS_DEFAULT_REQUEST_NUMBER was by designed to set to 1000 by default so avoid large number of data render issue.

@BionIT When we support multi data source, do you think if it's possible to read the value from index.max_result_window?

[cbeaujoin-stellar]

Hi @junqiu-lei,
is this planned for 2.14.0 (May 14th, 2024) ?

[junqiu-lei]

Assign to @vamshin to decide the priority.

[vamshin]

@cbeaujoin-stellar we are looking to have this delivered for 2.15.

[cbeaujoin-stellar]

Do not forget that for next release, it's still very boring to manualy export edit import Maps objects Json file.

[junqiu-lei]

@cbeaujoin-stellar After we looked into this issue, rendering issues can occur when large number of documents are visualized same time, which is why we have set the default document size to 1000. Additionally, the Maps app uses index patterns instead of single indices for data layers. An index pattern can encompass multiple indices, making it impractical to apply a single index's max_result_window setting in this context.

[cbeaujoin-stellar]

I'm running it with 4 indices with 500k documents each, it's working fine.

[cbeaujoin-stellar]

It should at least be configurable according to management/opensearch-dashboards/settings > Custom vector map size

[cbeaujoin-stellar]

Any update ?