Incorrect order of messages in the Matrix when using concurrency in the script

[drygdryg]

If the value of the CONCURRENCY_LIMIT .env variable is set to more than 1, messages in the Matrix are displayed in the wrong order (not in chronological order).
To keep the correct message order, I have to set CONCURRENCY_LIMIT=1 and pass the --sort {ts:1} argument to each mongoexport command.

[HerHde]

This is odd, as they should be sorted by the timestamp. Could you maybe send me the source code of two messages RC and corresponding Matrix, please?

[drygdryg]

For testing purposes, I have set up a clean instance of Rocket.Chat (version 3.18.0) and created 7 users. I wrote a script that sequentially sent messages from these users using Rocket.Chat API.

Screenshot of Rocket.Chat source room

After the migration, I get a broken message order.

Screenshot of migrated room in Element Desktop demonstrating a broken message order

I used CONCURRENCY_LIMIT=70.

The commands I used to export:

mongoexport --collection=rocketchat_message --db=rocketchat --out=rocketchat_message.json
mongoexport --collection=rocketchat_room --db=rocketchat --out=rocketchat_room.json
mongoexport --collection=users --db=rocketchat --out=users.json

Exported RC data:
mongodb-export.zip

[desto12]

I can confirm that there is issue with sorting messages by timestamp, I tried tu run 3 separate node apps with different messages.json. I think that should be handled on the matrix side :(

[flying-scorpio]

As explained here, sorting on the matrix server side is not done using the timestamps (which are "just" metadata).
The warning talks about a future implementation to permit server-side modifications of the ordering, but the spec discussion seems to show this is not going anywhere.

They also talk about the Gitter migration that has been done using timestamp massaging, which is exactly what this script does.

So my understanding is that we are forced to send messages in the correct sequence to have the server respect the order, and then the timestamp just makes it so that older messages are not dated to now().

[HerHde]

So my understanding is that we are forced to send messages in the correct sequence to have the server respect the order

So do I understand it