Merge pull request #532 from myteron/pyDoc2Github_multiprocessing_mul…

…tithreading

PyDoc2GitHub - Introduction to Multithreading and Multiprocessing in …

docs/Secure-Coding-Guide-for-Python/CWE-664/CWE-400/README.md

@@ -2,6 +2,10 @@
 
 Canceling the task in a thread pool only prevents it from being executed if it has not started yet. For the task to be interruptible, it must handle the `threading.Event` flag.
 
+> [!NOTE]
+> Prerequisite to understand this page:
+> [Intro to multiprocessing and multithreading](../../Intro_to_multiprocessing_and_multithreading/readme.md)
+
 ## Non-Compliant Code Example
 
 Tasks can be submitted to the ThreadPoolExecutor by calling `submit()`. Submitted tasks can be canceled by calling `cancel()` on the Future object returned by `submit()`. Calling this method will return True and stop the task from being executed if it has not started yet. However, if its execution has already started, calling `cancel()` will instead return  False and will not stop the task [[Python 3.10.4 docs on threading.Event]](https://docs.python.org/3/library/threading.html#event-objects).

docs/Secure-Coding-Guide-for-Python/CWE-664/CWE-410/README.md

@@ -8,6 +8,10 @@ Thread pools combine:
 * Protecting the input queue from being overloaded by providing optional time-out functions
 * Handling of failures that occur in a thread without impacting the re-usability of threads [Brownlee 2022]
 
+> [!NOTE]
+> Prerequisite to understand this page:
+> [Intro to multiprocessing and multithreading](../../Intro_to_multiprocessing_and_multithreading/readme.md)
+
 ## Non-Compliant Code Example (Thread-Per-Message)
 
 The `noncompliant01.py` code example demonstrates the Thread-Per-Message design pattern. Each request sent to MessageAPI results in a creation of a new thread.

docs/Secure-Coding-Guide-for-Python/CWE-664/CWE-665/README.md

@@ -0,0 +1,10 @@
+# CWE-665: Improper Initialization
+
+> [!NOTE]
+> Prerequisite to understand this page:
+> [Intro to multiprocessing and multithreading](../../Intro_to_multiprocessing_and_multithreading/readme.md)
+
+## Introduction
+
+> [!NOTE]
+> Placeholder page for links, content still need to be moved into github

docs/Secure-Coding-Guide-for-Python/CWE-664/CWE-833/README.md

@@ -2,6 +2,10 @@
 
 The number of threads that can simultaneously run within a `ThreadPoolExecutor` is limited by the `_max_workers` parameter. Submitting tasks whose execution is dependent on other tasks submitted to the same `ThreadPoolExecutor` may result in a *thread-starvation* deadlock. An example of this type of deadlock is shown in the following article [Brownlee, 2021], which describes it as "Deadlock 1: Submit and Wait for a Task Within a Task". Submitting a task will add it to an internal `ThreadPoolExecutor` queue. The task will be removed from the queue when one of the worker threads becomes available, i.e., after finishing its current task. If all workers are busy and all their current tasks are waiting for results from tasks that are waiting in the queue, the program will run indefinitely as no worker will be able to complete its task and take one of the blocking tasks from the queue.
 
+> [!NOTE]
+> Prerequisite to understand this page:
+> [Intro to multiprocessing and multithreading](../../Intro_to_multiprocessing_and_multithreading/readme.md)
+
 ## Non-Compliant Code Example (Interdependent Subtasks)
 
 The `noncompliant01.py` code example shows how a thread-starvation deadlock could happen when one task depends on the results of other, nested tasks. The `ReportTableGenerator` class creates a table out of the input data by building each row concurrently. The client can call the `generate_string_table()` method by providing a list of String arguments. Each argument is parsed in a separate thread that executes the inner method `_create_table_row()`. Creation of the row may consist of multiple steps, such as reformating the string, which could be performed in separate sub-threads. The `_reformat_string()` method is submitted within `_create_table_row()` and uses the same executor. Before the row is built, all of the building threads must return the results.

docs/Secure-Coding-Guide-for-Python/CWE-703/CWE-392/README.md

@@ -2,6 +2,10 @@
 
 Failure to provide a mechanism for reporting that tasks in a thread pool failed as a result of an exceptional condition can make it difficult or impossible to diagnose the problem.
 
+> [!NOTE]
+> Prerequisite to understand this page:
+> [Intro to multiprocessing and multithreading](../../Intro_to_multiprocessing_and_multithreading/readme.md)
+
 **ThreadPoolExecutor** from the `concurrent.futures` module is used in Python to asynchronously execute callable tasks [[Python Docs - concurrent.futures]](https://docs.python.org/3/library/concurrent.futures.html). The `ThreadPoolExecutor` suppresses exceptions raised by tasks to ensure that the failure of a task does not result in the failure of a worker thread in the thread pool. Otherwise, worker threads would close whenever an exception is raised, negatively impacting the performance of the `ThreadPoolExecutor`.
 
 ## Non-Compliant Code Example (submit)

docs/Secure-Coding-Guide-for-Python/Intro_to_multiprocessing_and_multithreading/compliant01.py

@@ -0,0 +1,27 @@
+# SPDX-FileCopyrightText: OpenSSF project contributors  
+# SPDX-License-Identifier: MIT
+""" Compliant Code Example """
+
+import time
+from multiprocessing import Process
+
+
+def waste_time(t: int):
+    for _ in range(t):
+        _ += 1
+
+
+BIG_NUMBER = 100000000
+start = time.time()
+waste_time(BIG_NUMBER)
+end = time.time()
+print(f"Time taken when executing sequentially (in seconds): {end - start}")
+p1 = Process(target=waste_time, args=(BIG_NUMBER // 2,))
+p2 = Process(target=waste_time, args=(BIG_NUMBER // 2,))
+start = time.time()
+p1.start()
+p2.start()
+p1.join()
+p2.join()
+end = time.time()
+print(f"Time taken when executing in 2 processes (in seconds): {end - start}")

docs/Secure-Coding-Guide-for-Python/Intro_to_multiprocessing_and_multithreading/compliant02.py

@@ -0,0 +1,17 @@
+# SPDX-FileCopyrightText: OpenSSF project contributors  
+# SPDX-License-Identifier: MIT
+""" Compliant Code Example """
+import asyncio
+
+
+async def func(x: int):
+    print(f"{x} - start")
+    await asyncio.sleep(1)
+    print(f"{x} - end")
+
+
+async def run_async():
+    await asyncio.gather(func(1), func(2), func(3))
+
+
+asyncio.run(run_async())

docs/Secure-Coding-Guide-for-Python/Intro_to_multiprocessing_and_multithreading/example01.py

@@ -0,0 +1,23 @@
+# SPDX-FileCopyrightText: OpenSSF project contributors  
+# SPDX-License-Identifier: MIT
+""" Non-compliant Code Example """
+
+import threading
+
+
+def critical_func(x: str, l: threading.Lock):
+    print(f"{x}: performing regular operations")
+    with l:
+        print(f"{x}: entered critical section")
+        i = 0
+        for _ in range(100000):
+            i += 1
+        print(f"{x}: exiting critical section")
+    print(f"{x}: finished")
+
+
+lock = threading.Lock()
+t1 = threading.Thread(target=critical_func, args=("A", lock))
+t2 = threading.Thread(target=critical_func, args=("B", lock))
+t1.start()
+t2.start()

docs/Secure-Coding-Guide-for-Python/Intro_to_multiprocessing_and_multithreading/noncompliant01.py

@@ -0,0 +1,17 @@
+# SPDX-FileCopyrightText: OpenSSF project contributors  
+# SPDX-License-Identifier: MIT
+""" Non-compliant Code Example """
+import threading
+
+
+def wait_for_other(x: str, other: threading.Thread):
+    print(f"{x}: waiting for other")
+    other.join()
+    print(f"{x}: finished waiting")
+
+
+t = threading.Thread(target=wait_for_other, args=("A", threading.current_thread()))
+t.start()
+print("B: waiting for other")
+t.join()
+print("B: finished waiting")

docs/Secure-Coding-Guide-for-Python/Intro_to_multiprocessing_and_multithreading/noncompliant02.py

@@ -0,0 +1,26 @@
+# SPDX-FileCopyrightText: OpenSSF project contributors  
+# SPDX-License-Identifier: MIT
+""" Non-compliant Code Example """
+import time
+from threading import Thread
+
+
+def waste_time(t: int):
+    for _ in range(t):
+        _ += 1
+
+
+BIG_NUMBER = 100000000
+start = time.time()
+waste_time(BIG_NUMBER)
+end = time.time()
+print(f"Time taken when executing sequentially (in seconds): {end - start}")
+t1 = Thread(target=waste_time, args=(BIG_NUMBER // 2,))
+t2 = Thread(target=waste_time, args=(BIG_NUMBER // 2,))
+start = time.time()
+t1.start()
+t2.start()
+t1.join()
+t2.join()
+end = time.time()
+print(f"Time taken when executing in 2 threads (in seconds): {end - start}")

docs/Secure-Coding-Guide-for-Python/Intro_to_multiprocessing_and_multithreading/noncompliant03.py

@@ -0,0 +1,25 @@
+# SPDX-FileCopyrightText: OpenSSF project contributors  
+# SPDX-License-Identifier: MIT
+""" Non-compliant Code Example """
+import time
+from threading import Thread
+
+
+def waste_time(t: float):
+    time.sleep(t)
+
+
+WAIT_TIME = 4
+start = time.time()
+waste_time(WAIT_TIME)
+end = time.time()
+print(f"Time taken when executing sequentially (in seconds): {end - start}")
+t1 = Thread(target=waste_time, args=(WAIT_TIME // 2,))
+t2 = Thread(target=waste_time, args=(WAIT_TIME // 2,))
+start = time.time()
+t1.start()
+t2.start()
+t1.join()
+t2.join()
+end = time.time()
+print(f"Time taken when executing in 2 threads (in seconds): {end - start}")