Consenus mode #42
Consenus mode #42
Conversation
| if consensus_mode: | ||
| original_diff_effects_fn = globals.harness_ctx.diff_effect_fn | ||
|
|
||
| def diff_effect_wrapper(a, b): |
There was a problem hiding this comment.
Does this take into account scenarios where there are multiple fields that need to be normalized? For example, for InstrEffects there's error codes and custom error codes that should both be ignored
There was a problem hiding this comment.
Nope, this stuff is hardcoded unfortunately. Didn't see any output in custom_err so I kinda ignored it for now.
Wasn't really sure of how to deal with different effects having different fields to ignore in a consensus mode run. The fact that we want to modify things in place for output also complicates things quite a bit. Perhaps we can define a separate diff_effects_consensus_fn as a part of the interface?
There was a problem hiding this comment.
How about we make globals.harness_ctx.result_field_names a list and iterate over it with a for loop within this function? Then for example if someone wanted to find the passing cases if you ignore, for example, CU's, then they could just add that to the list. You can keep the list empty as default, and someone who wants to modify the diff behavior can just add the ignored fields to the list
There was a problem hiding this comment.
So I don't exactly just ignore the result fields themselves, just that if they both have error codes. So a generic "ignore fields" list won't apply
Adds a "consensus mode" flag which, among other things, normalizes error codes between effects. This way, tests do not fail in scenarios that don't break consensus (i.e., different error codes)
Also did some cleanup by removing
check_consistency_in_resultsin multiprocessing_utils.. We have a lot more cleaning up to do