Parallelization Test Plan

Parallelization Test Plan

Parallelization Use Cases

This is a high level description of the parallelization use cases that need to be covered by automated tests. The test plan needs to cover all four modes of parallel execution with and without dependencies. Use cases are coarse-grained artifacts. There are numerous scenarios that need to be covered for each of the items listed below: Data providers, factories, sequential suite execution, parallel suite execution and various flavors of dependency conditions.

Use Case ID	Description
PTP-UC-1	Parallel by methods without dependencies.
PTP-UC-2	Parallel by methods with dependencies.
PTP-UC-3	Parallel by tests without dependencies.
PTP-UC-4	Parallel by tests with dependencies.
PTP-UC-5	Parallel by classes without dependencies.
PTP-UC-6	Parallel by classes with dependencies.
PTP-UC-7	Parallel by instances without dependencies.
PTP-UC-8	Parallel by instances with dependencies.

Parallelization Test Matrix

This test matrix breaks down each of the parallelization use cases into fine-grained test cases. The test matrix is a comprehensive statement what must be tested. It does not address how to implement the test cases, nor does it cover what the inputs are or what the expected outcomes are.

PTP-UC-1: Parallel by methods without dependencies.

Test Case ID	Suite execution mode	Data Provider (Non-Parallel)	Factory	Dependencies
PTP-TC-1	Sequential	No	No	N/A
PTP-TC-2	Parallel	No	No	N/A
PTP-TC-3	Sequential	Yes	No	N/A
PTP-TC-4	Parallel	Yes	No	N/A
PTP-TC-5	Sequential	No	Yes	N/A
PTP-TC-6	Parallel	No	Yes	N/A
PTP-TC-7	Sequential	Yes	Yes	N/A
PTP-TC-8	Parallel	Yes	Yes	N/A

PTP-UC-2: Parallel by methods with dependencies

Test Case ID	Suite execution mode	Data Provider (Non-Parallel)	Factory	Dependencies
PTP-TC-9	Sequential	No	No	Method
PTP-TC-10	Sequential	No	No	Intra-class group
PTP-TC-11	Sequential	No	No	Inter-class group
PTP-TC-12	Sequential	No	No	Intra-class meta group
PTP-TC-13	Sequential	No	No	Inter-class meta group
PTP-TC-14	Sequential	No	No	Inter-suite group
PTP-TC-15	Sequential	No	No	Inter-suite meta group
PTP-TC-16	Sequential	No	No	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)
PTP-TC-17	Parallel	No	No	Method
PTP-TC-18	Parallel	No	No	Intra-class group
PTP-TC-19	Parallel	No	No	Inter-class group
PTP-TC-20	Parallel	No	No	Intra-class meta group
PTP-TC-21	Parallel	No	No	Inter-class meta group
PTP-TC-22	Parallel	No	No	Inter-suite group
PTP-TC-23	Parallel	No	No	Inter-suite meta group
PTP-TC-24	Parallel	No	No	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)
PTP-TC-25	Sequential	Yes	No	Method
PTP-TC-26	Sequential	Yes	No	Intra-class group
PTP-TC-27	Sequential	Yes	No	Inter-class group
PTP-TC-28	Sequential	Yes	No	Intra-class meta group
PTP-TC-29	Sequential	Yes	No	Inter-class meta group
PTP-TC-30	Sequential	Yes	No	Inter-suite group
PTP-TC-31	Sequential	Yes	No	Inter-suite meta group
PTP-TC-32	Sequential	Yes	No	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)
PTP-TC-33	Parallel	Yes	No	Method
PTP-TC-34	Parallel	Yes	No	Intra-class group
PTP-TC-35	Parallel	Yes	No	Inter-class group
PTP-TC-36	Parallel	Yes	No	Intra-class meta group
PTP-TC-37	Parallel	Yes	No	Inter-class meta group
PTP-TC-38	Parallel	Yes	No	Inter-suite group
PTP-TC-39	Parallel	Yes	No	Inter-suite meta group
PTP-TC-32	Parallel	Yes	No	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)
PTP-TC-33	Sequential	No	Yes	Method
PTP-TC-34	Sequential	No	Yes	Intra-class group
PTP-TC-35	Sequential	No	Yes	Inter-class group
PTP-TC-36	Sequential	No	Yes	Intra-class meta group
PTP-TC-37	Sequential	No	Yes	Inter-class meta group
PTP-TC-38	Sequential	No	Yes	Inter-suite group
PTP-TC-39	Sequential	No	Yes	Inter-suite meta group
PTP-TC-40	Sequential	No	Yes	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)
PTP-TC-41	Parallel	No	Yes	Method
PTP-TC-42	Parallel	No	Yes	Intra-class group
PTP-TC-43	Parallel	No	Yes	Inter-class group
PTP-TC-44	Parallel	No	Yes	Intra-class meta group
PTP-TC-45	Parallel	No	Yes	Inter-class meta group
PTP-TC-46	Parallel	No	Yes	Inter-suite group
PTP-TC-47	Parallel	No	Yes	Inter-suite meta group
PTP-TC-48	Parallel	No	Yes	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)
PTP-TC-49	Sequential	Yes	Yes	Method
PTP-TC-50	Sequential	Yes	Yes	Intra-class group
PTP-TC-51	Sequential	Yes	Yes	Inter-class group
PTP-TC-52	Sequential	Yes	Yes	Intra-class meta group
PTP-TC-53	Sequential	Yes	Yes	Inter-class meta group
PTP-TC-54	Sequential	Yes	Yes	Inter-suite group
PTP-TC-55	Sequential	Yes	Yes	Inter-suite meta group
PTP-TC-56	Sequential	Yes	Yes	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)
PTP-TC-57	Parallel	Yes	Yes	Method
PTP-TC-58	Parallel	Yes	Yes	Intra-class group
PTP-TC-59	Parallel	Yes	Yes	Inter-class group
PTP-TC-60	Parallel	Yes	Yes	Intra-class meta group
PTP-TC-61	Parallel	Yes	Yes	Inter-class meta group
PTP-TC-62	Parallel	Yes	Yes	Inter-suite group
PTP-TC-63	Parallel	Yes	Yes	Inter-suite meta group
PTP-TC-64	Parallel	Yes	Yes	Method, group (intra-class, inter-class, intra-suite, and inter-suite), meta group (Intra-class, inter-class, intra-suite, and inter-suite)

Testing methodology

There are some general areas of concern for a parallelization test. They must cover the following bases:

• That parallel threads are spawned according to the parallelization mode specified.
• That parallel threads are spawned in accordance with the thread pool limitations specified -- suite thread pool size, data provider thread pool size, and thread counts.
• That true parallelization is actually taking place. It is possible for multiple threads to be spawned when parallelization is enabled, but they do not actually run in parallel, but rather one after another in sequential fashion.
• When there are dependencies involved, that topological sorting is preserved so that methods execute only after groups and methods they depend on have executed and passed.

In order to successfully verify all the above, significant bookkeeping is required to accurately track the state of the model execution run:

• The start and end times of each suite (DONE)
• The start and end times of each defined 'test' within the suites (DONE)
• The start and end time of each test method (DONE)
• The thread pool ID and thread IDs for test method (PARTIAL -- Thread ID is now logged)
• The order of execution of the test methods (DONE)
• When there are multiple instances of test classes, the methods in the class also need to be mapped to the instance to which they belong (DONE. NEEDS TESTING.)
• When a new thread is spawned, the thread count is recorded. (TBD)
• When a thread completes execution, the thread count is recorded. (TBD)

There is an execution state tracker which provides an interface to a synchronized data structure that holds all this information: test.thread.parallelization.TestNgRunStateTracker.

There is a listener class implementation for the suite, test and test method listeners interfaces that writes state information using the execution state tracker: test.thread.parallelization.TestNgRunStateListener.

Parallelization Test Cases

PTP-TC-1: Parallel by methods mode with sequential test suites, no dependencies and no factories or data providers.

VERIFIES:

1. That all methods in one suite are executed before any methods in subsequent suites are executed
2. When the number of threads reaches the specified thread count, that no additional threads are spawned until the thread count drops
3. That when thread count drops below the maximum and there are still methods in the queue, a new method begins execution.
4. That suite-level and test-level events run in the same thread
5. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
6. That different test methods and their associated listener events are executed in separate threads
7. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order

SCENARIOS:

1. Single suite with a single test consisting of a single test class with five methods

   • Thread count and parallel mode are specified at the suite level
   • The thread count is equal to the number of test methods, ensuring that no method should have to wait for any other method to complete execution
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

2. Three suites with 1, 2 and 3 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class.

   • For one of the suites, the thread count and parallel mode are specified at the suite level
   • For one of the suites, the thread count and parallel mode are specified at the test level
   • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
   • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
   • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

PTP-TC-2: Parallel by methods mode with parallel test suites, no dependencies and no factories or data providers.

VERIFIES:

1. That the suites are run in parallel, but that their tests are run sequentially
2. That the test methods in the tests for each suite are executed in parallel fashion according to their individual thread count limitations
3. That when the number of suites executing in parallel reaches the specified suite thread pool size, that no additional suite-level threads are spawned until the number of suites executing in parallel drops below the specified suite thread pool size
4. When the number of methods executing in parallel for a given suite reaches the specified thread count, that no additional method-level threads are spawned until the number of methods executing for that suite drops below its specified thread count
5. That when the number of suites executing in parallel drops below the suite thread pool size and there are still suites in the queue, a new suite begins execution.
6. That when thread count drops below the maximum for a given suite and there are still methods in the queue for that suite, a new method begins execution
7. That suite-level and test-level events run in the same thread for each suite executing in parallel
8. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
9. That different test methods and their associated listener events are executing in different threads
10. That separate threads in which suites execute are run in true parallel fashion and not in sequential order
11. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order

SCENARIOS:

1. Two suites with 1 and 2 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class.

   • The suite thread pool is 2, so neither suite should have to wait for the other to complete execution
   • For both suites, the thread count and parallel mode are specified at the suite level
   • The thread count is less than the number of test methods in a test for one of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
   • The thread count is more than the number of test methods for the rest of the tests, ensuring that none of their test methods should have to wait for any other method to complete execution
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

2. Three suites with 1, 2 and 3 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class.

   • The suite thread pool is 2, so one suite will have to wait for one of the others to complete execution before it can begin execution
   • For one of the suites, the thread count and parallel mode are specified at the suite level
   • For one of the suites, the thread count and parallel mode are specified at the test level
   • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
   • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
   • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

PTP-TC-3: Parallel by methods mode with sequential test suites using a non-parallel data provider but no dependencies and no factories.

VERIFIES:

1. That all methods in one suite are executed before any methods in subsequent suites are executed
2. When the number of threads reaches the specified thread count, that no additional threads are spawned until the thread count drops
3. That when thread count drops below the maximum and there are still methods in the queue, a new method begins execution.
4. That suite-level and test-level events run in the same thread
5. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
6. That different test methods and their associated listener events are executed in separate threads
7. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order
8. That the methods are run for each data set given to them by the data provider in accordance with the thread count specified in parallel by methods mode. The data provider is NOT parallel, so each invocation of the method should in in the same thread.

SCENARIOS:

1. Single suite with a single test consisting of a single test class with five methods with a data provider specifying 3 sets of data

   • Thread count and parallel mode are specified at the suite level
   • The thread count is equal to the number of test methods times 3, the number of times each method will be invoked with a data set from the data provider. Expectation is that only 5 threads will be spawned, one for each of the methods and that in this thread, the method will be invoked 3 times, once for each set of data from the data provider.
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

2. Two suites with 1 and 2 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class. Each test class has some methods with use a data provider and some which do not. Two data providers are used: one which provides two sets of data, one which provide three sets of data.

   • For both suites, the thread count and parallel mode are specified at the suite level
   • The thread count is less than the number of test methods for all tests in both suites, so methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution. The expectation is that a thread for a method does not terminate until the method has been invoked for all sets of data if it uses a data provider.
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

PTP-TC-4: Parallel by methods mode with parallel test suites using a non-parallel data provider but no dependencies and no factories.

1. That the suites are run in parallel, but that their tests are run sequentially
2. That the test methods in the tests for each suite are executed in parallel fashion according to their individual thread count limitations
3. That when the number of suites executing in parallel reaches the specified suite thread pool size, that no additional suite-level threads are spawned until the number of suites executing in parallel drops below the specified suite thread pool size
4. When the number of methods executing in parallel for a given suite reaches the specified thread count, that no additional method-level threads are spawned until the number of methods executing for that suite drops below its specified thread count
5. That when the number of suites executing in parallel drops below the suite thread pool size and there are still suites in the queue, a new suite begins execution.
6. That when thread count drops below the maximum for a given suite and there are still methods in the queue for that suite, a new method begins execution
7. That suite-level and test-level events run in the same thread for each suite executing in parallel
8. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
9. That different test methods and their associated listener events are executing in different threads
10. That separate threads in which suites execute are run in true parallel fashion and not in sequential order
11. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order
12. That the methods are run for each data set given to them by the data provider in accordance with the thread count specified in parallel by methods mode. The data provider is NOT parallel, so each invocation of the method should in in the same thread.

SCENARIOS:

1. Three suites with 1, 2 and 3 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class. Some test classes have a mixture of some methods that use a data provider and some which do not. The data providers provide data sets of varying sizes.

   • The suite thread pool is 2, so one suite will have to wait for one of the others to complete execution before it can begin execution
   • For one of the suites, the thread count and parallel mode are specified at the suite level
   • For one of the suites, the thread count and parallel mode are specified at the test level
   • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
   • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution. The expectation is that a thread for a method does not terminate until the method has been invoked for all sets of data if it uses a data provider.
   • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution. The expectation is that a thread for a method does not terminate until the method has been invoked for all sets of data if it uses a data provider.
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

PTP-TC-5: Parallel by methods mode with sequential test suites using factories but no dependencies and no data providers.

VERIFIES:

1. That all methods in one suite are executed before any methods in subsequent suites are executed
2. When the number of threads reaches the specified thread count, that no additional threads are spawned until the thread count drops
3. That when thread count drops below the maximum and there are still methods in the queue, a new method begins execution.
4. That suite-level and test-level events run in the same thread
5. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
6. That different test methods and their associated listener events are executed in separate threads
7. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order
8. That the methods are run for each test class instance created by a factory in accordance with the thread count specified in parallel by methods mode, and that methods in one instance do not wait for methods in another instance to complete before they execute if the thread count is less than the maximum.

SCENARIOS:

1. Single suite with a single test consisting of a factory that provides two instances of a single test class with five methods

   • Thread count and parallel mode are specified at the suite level
   • The thread count is equal to the number of test methods times 2, the number of times each method will be invoked because there are two instances of the test class. Expectation is that 10 threads will be spawned because each method will run once for each of the two instances and that each instance of a test method will run in separate threads.
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

2. Two suites with 1 and 2 tests respectively. One suite with two tests which have a test for a suite shall consist of a single test class without a factory while the other shall consist of factories which provide multiple instances of multiple test classes. One suite shall consist of a single test with multiple test classes which uses factories.

   • For both suites, the thread count and parallel mode are specified at the suite level
   • The thread count is less than the number of test methods for all tests in both suites, so methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution. The expectation is that threads will be spawned for each test method that executes just as they would if there were no factories and test suites simply used the default mechanism for instantiating single instances of the test classes.
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

PTP-TC-6: Parallel by methods mode with parallel test suites using factories but no dependencies and no data providers.

VERIFIES:

1. That the suites are run in parallel, but that their tests are run sequentially
2. That the test methods in the tests for each suite are executed in parallel fashion according to their individual thread count limitations
3. That when the number of suites executing in parallel reaches the specified suite thread pool size, that no additional suite-level threads are spawned until the number of suites executing in parallel drops below the specified suite thread pool size
4. When the number of methods executing in parallel for a given suite reaches the specified thread count, that no additional method-level threads are spawned until the number of methods executing for that suite drops below its specified thread count
5. That when the number of suites executing in parallel drops below the suite thread pool size and there are still suites in the queue, a new suite begins execution.
6. That when thread count drops below the maximum for a given suite and there are still methods in the queue for that suite, a new method begins execution
7. That suite-level and test-level events run in the same thread for each suite executing in parallel
8. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
9. That different test methods and their associated listener events are executing in different threads
10. That separate threads in which suites execute are run in true parallel fashion and not in sequential order
11. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order
12. That the methods are run for each test class instance created by a factory in accordance with the thread count specified in parallel by methods mode, and that methods in one instance do not wait for methods in another instance to complete before they execute if the thread count is less than the maximum.

SCENARIOS:

1. Three suites with 1, 2 and 3 tests respectively. One suite with two tests which have a test for a suite shall consist of a single test class without a factory while the other shall consist of factories which provide multiple instances of multiple test classes. One suite shall consist of a single test with multiple test classes which uses factories. One suite shall have multiple tests with multiple classes, none of which use a factory.

   • The suite thread pool is 2, so one suite will have to wait for one of the others to complete execution before it can begin execution
   • For one of the suites, the thread count and parallel mode are specified at the suite level
   • For one of the suites, the thread count and parallel mode are specified at the test level
   • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
   • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
   • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution
   • The expectation is that threads will be spawned for each test method that executes just as they would if there were no factories and test suites simply used the default mechanism for instantiating single instances of the test classes.
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions

PTP-TC-7: Parallel by methods mode with sequential test suites using a factory with uses a non-parallel data provider and there are no dependencies.

VERIFIES:

1. That all methods in one suite are executed before any methods in subsequent suites are executed
2. When the number of threads reaches the specified thread count, that no additional threads are spawned until the thread count drops
3. That when thread count drops below the maximum and there are still methods in the queue, a new method begins execution.
4. That suite-level and test-level events run in the same thread
5. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
6. That different test methods and their associated listener events are executed in separate threads
7. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order
8. That the methods are run for each test class instance created by a factory for each data set given by the data provider in accordance with the thread count specified in parallel by methods mode
9. That methods in one instance do not wait for methods in another instance to complete before they execute if the thread count is less than the maximum.

SCENARIOS:

1. Single suite with a single test consisting of a single test class with five methods with a factory method using a data provider specifying 3 sets of data

   • Thread count and parallel mode are specified at the suite level
   • The thread count is equal to the number of test methods times 3, the number of times each method will be invoked with a data set from the data provider. Expectation is that only 15 threads will be spawned at once. There will be three instances of the test class, one for each of the three data sets and each instance will have the five test methods defined on the class, for a total of 15 test method invocations.
   • No test method invocation should be queued because the thread count is sufficient to service all the test method invocations.
   • There are NO configuration methods
   • All test methods pass
   • NO ordering is specified
   • group-by-instances is NOT set
   • There are no method exclusions