[Security Solution] [Bug] A few entities fail to be assigned the values every time for asset criticality via bulk upload

[muskangulati-qasource]

Describe the bug
A few entities fail to be assigned the values every time for asset criticality via bulk upload

Kibana/Elasticsearch Stack version

VERSION: 8.16.0
BUILD: 79314
COMMIT: 5575428dd3aef69366cddb4ccf07a2a26d30ce48

Steps

1. Kibana version 8.16.0 or above should exist
2. Navigate to Stack Management >> Alerts and Insights >> Entity Store
3. Add a file which has multiple entries atleast 21k
4. Validate it always shows error that a few entries are not assigned the value. However, the reason is not specified

Expected Result
No entities should fail to be assigned the values every time for asset criticality via bulk upload

Screen Recording

Entity.Store.mp4

Test file used
user.txt

Error from network logs

 "errors": [
        {
            "message": "[host.name:ip-172-31-83-81]: version conflict, required seqNo [27862], primary term [2]. current document has seqNo [27871] and primary term [2]"
        },
        {
            "message": "[user.name:root]: version conflict, required seqNo [27880], primary term [2]. current document has seqNo [27882] and primary term [2]"
        },
        {
            "message": "[user.name:root]: version conflict, required seqNo [27897], primary term [2]. current document has seqNo [27902] and primary term [2]"
        },
        {
            "message": "[user.name:root]: version conflict, required seqNo [27908], primary term [2]. current document has seqNo [27912] and primary term [2]"
        },

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[muskangulati-qasource]

@amolnater-qasource please review!

[amolnater-qasource]

Reviewed & assigned to @MadameSheema

[elasticmachine]

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

[jaredburgettelastic]

@muskangulati-qasource could you please share with us the file that you used to upload?

[muskangulati-qasource]

Hi @jaredburgettelastic,

We have used below file to upload via bulk upload for asset criticality:

user.txt

Please let us know if anything else is required from our end.

Thank you!

[machadoum]

I was able to reproduce it. It is an extreme edge case. The document assigns criticality thousands of times for the same entity. Because the server side is asynchronous, some assignments for the same entity conflict.

But we can make some improvements in this scenario:

1 It is hard to find the line with an error when uploading a file with thousands of lines. Maybe we should show two EuiCodeBlocks: one with the successful assignment and one with the failed assignment and error messages for every line. But then we lose the line number.

2 We could validate duplicated entities in the server and return an error. But that requires storing every entity name in a map, which costs memory. @hop-dev Do you think the feature is worth the performance impact?
Here is simplified example of how it could be implemented: d4454f8

[hop-dev]

@machadoum I think it is likely worth it, I would be interested to compare the performance of a large file before and after the change to double check.

[jaredburgettelastic]

Fixing this bug will also fix a bug where the server-based line number is off-by-one.

[machadoum]

PR: #199651

[muskangulati-qasource]

Hi @MadameSheema,

We have tested this ticket on the latest 8.16.1 BC2 and found the issue is now fixed.

Please find below the testing details:

Build details

VERSION: 8.16.1
BUILD: 79724
COMMIT: c8b46e87c4d61de4fe046ce5ea0a0b68aad5acf9

Screen Recording

• When duplicate entries exists:
  File used: user (1).txt

2.mp4

• When non duplicate entries are used:
  File used: userNew.txt

3.mp4

Hence, we are closing this issue and marking it as 'QA validated'.

Thanks!!