idempotence tests are weak

[dwlehman]

One option for better idempotence tests would be to save the data we collect for verifications and compare the data for the two runs instead of (or in addition to) doing the normal verification.

[AbdelAzizMohamedMousa]

Yes, saving the data and comparing it between runs can be a good option for improving idempotence tests. By saving the data collected during the first run and comparing it to the data collected during subsequent runs, you can verify that the system behaves idempotently.

This approach can be particularly useful in cases where the normal verification method may not be sufficient or may not provide enough confidence in the system's idempotent behavior. By comparing the data, you can be more confident that the system is behaving as expected.

However, it's important to keep in mind that this approach may require additional resources for storing and comparing the data. You'll also need to ensure that the data being saved and compared is sufficient to verify idempotence accurately.

Another approach that can be used in addition to saving and comparing data is to randomize inputs to the system. By randomizing the inputs, you can test the system's ability to handle different scenarios and still behave idempotently.