Towards Correctness Checking of MPI Partitioned Communication in MUST - Artifact

This repository contains the artifact with the test cases for the paper Correctness Checking of MPI+OpenMP Applications Using Vector Clocks in MUST. The test cases used are in the tests/ folder and are grouped by MPI-CorrBench (COBE) and our own test suite, as well as correct and erroneous test cases.

Flaky Test Cases

Some test cases are not uniquely classified. For the counter-based MPI-Serialized analysis, the error has to manifest to be detected, meaning any of the erroneous thread-level test cases could be classified as TP or FN. This includes the four test cases in our own test suite:

• ThreadLevel_serialized_error.cpp
• ThreadLevel_serialized_few_tasks_error.cpp
• ThreadLevel_serialized_many_tasks_error.cpp
• ThreadLevel_serialized_sections_error.cpp

And the single test case from MPI-CorrBench:

• wrong_threading_level_3.c

The ThreadLevel_serialized_many_tasks_error.cpp test case performs many concurrent MPI calls and is more consistently classified as TP. The other four test cases are not consistently classified as TP or FN.

False Positive Test Case'

The test case own_checks/ConComm/correct/concurrent_communicator_transitive.cpp is a correct test for the concurrent collectives that use the same communicator error class, that uses MPI synchronization to synchronize the threads of the first process transitively. Since the clock-based approach currently does not track the MPI synchronization it also does not recognize this transitive synchronization and reports this test case as erroneous even though the execution of the collectives that use the same communicator are not concurrent, and hence clock-based approach classifies this test case as (FP). MUST does not have a counter-based approach for this error class and hence classifies this test case as TN.

Classification Results

• TP: True Positive (correct-alert)
• FP: False Positive (false-alert)
• TN: True Negative (error-free)
• FN: False Negative (missing-alert)
• A: Accuracy

Analyses	Test Suite	Total	TP	FP	TN	FN	A
Counter-based	Own	30	1	0	17	12	0.60
Clock-based	Own	30	13	1	16	0	0.97
Counter-based	COBE	20	0	0	10	10	0.50
Clock-based	COBE	20	10	0	10	0	1.00

Detailed Results

Test Suite	Test	Error class	Type	Counter-based	Clock-based
COBE	deadlock_probe_2	Probe	Correct	TN	TN
COBE	probe	Probe	Correct	TN	TN
COBE	two_collectives	Collective	Correct	TN	TN
COBE	two_collectives_2	Collective	Correct	TN	TN
COBE	two_collectives_3	Collective	Correct	TN	TN
COBE	two_collectives_4	Collective	Correct	TN	TN
COBE	two_collectives_5	Collective	Correct	TN	TN
COBE	two_collectives_6	Collective	Correct	TN	TN
COBE	two_collectives_7	Collective	Correct	TN	TN
COBE	two_collectives	Thread-serialized	Correct	TN	TN
COBE	deadlock_probe_2	Probe	Erroneous	FN	TP
COBE	probe	Probe	Erroneous	FN	TP
COBE	two_collectives	Collectives	Erroneous	FN	TP
COBE	two_collectives_2	Collectives	Erroneous	FN	TP
COBE	two_collectives_3	Collectives	Erroneous	FN	TP
COBE	two_collectives_4	Collectives	Erroneous	FN	TP
COBE	two_collectives_5	Collectives	Erroneous	FN	TP
COBE	two_collectives_6	Collectives	Erroneous	FN	TP
COBE	two_collectives_7	Collectives	Erroneous	FN	TP
COBE	wrong_threading_level_3	Thread-serialized	Erroneous	FN	TP
Own	concurrent_recv	Receive	Correct	TN	TN
Own	concurrent_communicator	Collective	Correct	TN	TN
Own	concurrent_communicator_for_dynamic	Collective	Correct	TN	TN
Own	concurrent_communicator_for_nowait	Collective	Correct	TN	TN
Own	concurrent_communicator_sections	Collective	Correct	TN	TN
Own	concurrent_communicator_single	Collective	Correct	TN	TN
Own	concurrent_communicator_task_nesting	Collective	Correct	TN	TN
Own	concurrent_communicator_tasking_depend	Collective	Correct	TN	TN
Own	concurrent_communicator_tasking_locks	Collective	Correct	TN	TN
Own	concurrent_communicator_transitive	Collective	Correct	TN	FP
Own	concurrent_recv_probe	Probe	Correct	TN	TN
Own	concurrent_recv_probe_any_tag	Probe	Correct	TN	TN
Own	ThreadLevel_serialized	Thread-serialized	Correct	TN	TN
Own	ThreadLevel_serialized_single	Thread-serialized	Correct	TN	TN
Own	ThreadLevel_serialized_tasking_depend	Thread-serialized	Correct	TN	TN
Own	ThreadLevel_serialized_tasking_locks	Thread-serialized	Correct	TN	TN
Own	ThreadLevel_serialized_taskwait	Thread-serialized	Correct	TN	TN
Own	concurrent_recv_any_tag_error	Receive	Erroneous	FN	TP
Own	concurrent_recv_error	Receive	Erroneous	FN	TP
Own	concurrent_communicator_error	Collective	Erroneous	FN	TP
Own	concurrent_communicator_for_dynamic_nowait_error	Collective	Erroneous	FN	TP
Own	concurrent_communicator_sections_error	Collective	Erroneous	FN	TP
Own	concurrent_communicator_task_nesting_error	Collective	Erroneous	FN	TP
Own	concurrent_communicator_tasking_error	Collective	Erroneous	FN	TP
Own	concurrent_recv_probe_any_tag_error	Probe	Erroneous	FN	TP
Own	concurrent_recv_probe_error	Probe	Erroneous	FN	TP
Own	ThreadLevel_serialized_error	Thread-serialized	Erroneous	Flaky (mostly FN)	TP
Own	ThreadLevel_serialized_few_tasks_error	Thread-serialized	Erroneous	Flaky (mostly FN)	TP
Own	ThreadLevel_serialized_many_tasks_error	Thread-serialized	Erroneous	TP (inherently flaky)	TP
Own	ThreadLevel_serialized_sections_error	Thread-serialized	Erroneous	Flaky (mostly FN)	TP

Building Docker Container

The docker container installs MPICH v4.0.2, Clang v15.0.6, MUST, and LLVM lit. To build the container, run in the root directory:

docker build -t must-omp-clock .

Running Tests

To run all test cases for the counter-based and clock-based analyses, run:

docker run -it must-omp-clock ./tests/run_tests.sh

(Make sure that run_tests.sh uses LF line endings and not CRLF, otherwise Linux will provide the error message "No such file or directory")

To start a specific test case, run:

docker run -e LIT_MUST_VERSION=<analysis-type> -it must-omp-clock lit -j1 -a <path-to-test>