From 02b2506b125c72f6917b9402868192af4eef7cfe Mon Sep 17 00:00:00 2001 From: webreflection Date: Mon, 13 May 2024 11:15:51 +0200 Subject: [PATCH 1/5] Fix #97 - Explain the deadlock case --- docs/faq.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/docs/faq.md b/docs/faq.md index 665e965..1a22b0c 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -376,6 +376,68 @@ obsolete standards. While such legacy code exists, be aware that JavaScript code may require special care. +### Possible deadlock + +There are cases where users might encounter an error similar to the following one: + +> 💀🔒 - Possible deadlock if proxy.xyz(...args) is awaited + +#### When + +Let's assume a worker script contains the following *Python* code: + +```python title="worker: a deadlock example" +from pyscript import sync + +sync.worker_task = lambda: print('🔥 this is fine 🔥') + +# deadlock 💀🔒 +sync.main_task() +``` + +On the *main* thread, let's instead assume this code: +```html title="main: a deadlock example" + +``` + +When the worker bootstraps and locks itself until `main_task()` is completed, it cannot respond to anything at all: it's literally locked. + +If the *awaited* task is then asking for *worker* tasks to execute, we are in a clear [deadlock](https://en.wikipedia.org/wiki/Deadlock) situation. + +#### Workaround + +Beside trying to avoid deadlocks by all mean is the obvious suggestion and solution, the *blocking* part of the equation is what makes the presented exchange not possible. + +However, if the *main* task does not need to block the *worker* while executing, we can rewrite that code as such: + +On the *main* thread, let's instead assume this code: + +```html title="main: avoiding deadlocks" + +``` + +This way it's still possible to consume *worker* exposed utilities within *main* exposed tasks and the worker can happily unlock itself and react to any scheduled task in the meantime. + + ## Helpful hints This section contains common hacks or hints to make using PyScript easier. From b926968b358e609047d0112fef788c0330ef9bd9 Mon Sep 17 00:00:00 2001 From: "Nicholas H.Tollervey" Date: Mon, 13 May 2024 10:57:16 +0100 Subject: [PATCH 2/5] Update deadlock section in the FAQ. --- docs/faq.md | 45 +++++++++++++++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 1a22b0c..f37de2b 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -378,13 +378,23 @@ special care. ### Possible deadlock -There are cases where users might encounter an error similar to the following one: +Users may encounter an error message similar to the following: -> 💀🔒 - Possible deadlock if proxy.xyz(...args) is awaited +!!! failure + + ``` + 💀🔒 - Possible deadlock if proxy.xyz(...args) is awaited + ``` #### When -Let's assume a worker script contains the following *Python* code: +This error happens when your code on a worker and in the main thread are +[in a deadlock](https://en.wikipedia.org/wiki/Deadlock). Put simply, neither +fragment of code can proceed without waiting for the other. + +#### Why + +Let's assume a worker script contains the following Python code: ```python title="worker: a deadlock example" from pyscript import sync @@ -395,7 +405,8 @@ sync.worker_task = lambda: print('🔥 this is fine 🔥') sync.main_task() ``` -On the *main* thread, let's instead assume this code: +On the main thread, let's instead assume this code: + ```html title="main: a deadlock example" ``` -When the worker bootstraps and locks itself until `main_task()` is completed, it cannot respond to anything at all: it's literally locked. - -If the *awaited* task is then asking for *worker* tasks to execute, we are in a clear [deadlock](https://en.wikipedia.org/wiki/Deadlock) situation. +When the worker bootstraps and calls `sync.main_task()` on the main thread, it +blocks until the result of this call is returned. Hence it cannot respond to +anything at all. However, in the code on the main thread, the +`sync.worker_task()` in the worker is called, but the worker is blocked! Now +the code on both the main thread and worker are mutually blocked and waiting +on each other. We are in a classic +[deadlock](https://en.wikipedia.org/wiki/Deadlock) situation. -#### Workaround +The moral of the story? Don't create such circular deadlocks! -Beside trying to avoid deadlocks by all mean is the obvious suggestion and solution, the *blocking* part of the equation is what makes the presented exchange not possible. +How? -However, if the *main* task does not need to block the *worker* while executing, we can rewrite that code as such: +The mutually blocking calls cause the deadlock, so simply don't block. -On the *main* thread, let's instead assume this code: +For example, on the main thread, let's instead assume this code: ```html title="main: avoiding deadlocks" ``` -This way it's still possible to consume *worker* exposed utilities within *main* exposed tasks and the worker can happily unlock itself and react to any scheduled task in the meantime. - +By scheduling the call to the worker (rather than awaiting it), it's possible +for the main thread to call functions defined in the worker in a non-blocking +manner, thus allowing the worker to also work in an unblocked manner and react +to such calls. We have resolved the mutual deadlock. ## Helpful hints From 4422cac5f5ea3cf3460db304ea047e84b6544616 Mon Sep 17 00:00:00 2001 From: "Nicholas H.Tollervey" Date: Mon, 13 May 2024 11:07:18 +0100 Subject: [PATCH 3/5] A more complete README. --- README.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 8af4f1b..5824a7f 100644 --- a/README.md +++ b/README.md @@ -2,21 +2,27 @@ Welcome to the PyScript documentation repository. +This source code becomes the official PyScript documentation hosted here: + +[https://docs.pyscript.net](https://docs.pyscript.net/) + Contribute prose and participate in discussions about the written support of PyScript and related topics. ## Getting started Before you start contributing to the documentation, it's worthwhile to -take a look at the general contributing guidelines for the PyScript project. You can find these guidelines here +take a look at the general contributing guidelines for the PyScript project. +You can find these guidelines here [Contributing Guidelines](https://github.com/pyscript/pyscript/blob/main/CONTRIBUTING.md) ## Setup The `docs` directory in the pyscript repository contains a -[Material for MkDocs](https://squidfunk.github.io/mkdocs-material/) documentation project. Material is a system -that takes plaintext files containing documentation written in Markdown, along with -static files like templates and themes, to build the static end result. +[Material for MkDocs](https://squidfunk.github.io/mkdocs-material/) +documentation project. Material is a system that takes plaintext files +containing documentation written in Markdown, along with static files like +templates and themes, to build the static end result. To setup the documentation development environment simply create a new virtual environment, then `pip install -r requirements.txt` (from in the root @@ -35,19 +41,22 @@ Simply run `mkdocs serve` or `./bin/mkdocs serve`. ## Cross-referencing -You can link to other pages in the documentation by using the `{doc}` role. For example, to link to the `docs/README.md` file, you would use: +Link to other pages in the documentation by using the `{doc}` role. For +example, to link to the `docs/README.md` file, you would use: ```markdown {doc}`docs/README.md` ``` -You can also cross-reference the python glossary by using the `{term}` role. For example, to link to the `iterable` term, you would use: +Cross-reference the Python glossary by using the `{term}` role. For example, to +link to the `iterable` term, you would use: ```markdown {term}`iterable` ``` -You can also cross-reference functions, methods or data attributes by using the `{attr}` for example: +Cross-reference functions, methods or data attributes by using the `{attr}` for +example: ```markdown {py:func}`repr` From 86edd34f74c0b69b217b266538fe8b58694d0be5 Mon Sep 17 00:00:00 2001 From: "Nicholas H.Tollervey" Date: Mon, 13 May 2024 11:29:36 +0100 Subject: [PATCH 4/5] Document sync_main_only flag and use cases. --- docs/faq.md | 2 +- docs/user-guide/configuration.md | 80 +++++++++++++++++++++++--------- docs/user-guide/workers.md | 14 ++++++ 3 files changed, 72 insertions(+), 24 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index f37de2b..5b75b94 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -91,7 +91,7 @@ solution. (These requirements are explored workers are limited to one way calls from the main thread to methods exposed by workers. -If `sync_main_only = True`, the following caveats apply: +If `sync_main_only = true`, the following caveats apply: * It is not possible to manipulate the DOM or do anything meaningful on the main thread **from a worker**. This is because Atomics cannot guarantee diff --git a/docs/user-guide/configuration.md b/docs/user-guide/configuration.md index 0fe8c1c..e1174a5 100644 --- a/docs/user-guide/configuration.md +++ b/docs/user-guide/configuration.md @@ -35,11 +35,11 @@ ensure the packages [arrr](https://arrr.readthedocs.io/en/latest/) and [numberwang](https://numberwang.readthedocs.io/en/latest/) are installed from PyPI (the [Python Packaging Index](https://pypi.org/)): -```TOML title="Configuration via TOML" +```TOML title="Configuration via TOML." packages = ["arrr", "numberwang" ] ``` -```JSON title="Configuration via JSON" +```JSON title="Configuration via JSON." { "packages": ["arrr", "numberwang"] } @@ -56,7 +56,7 @@ reference it from the tag used to specify the Python code: If you use JSON, you can make it the value of the `config` attribute: -```HTML title="JSON as the value of the config attribute" +```HTML title="JSON as the value of the config attribute." ``` @@ -64,7 +64,7 @@ For historical and convenience reasons we still support the inline specification of configuration information via a _single_ `` or `` tag in your HTML document: -```HTML title="Inline configuration via the <py-config> tag" +```HTML title="Inline configuration via the <py-config> tag." { "packages": ["arrr", "numberwang" ] @@ -79,10 +79,10 @@ specification of configuration information via a _single_ `` or ## Options -There are four core options ([`interpreter`](#interpreter), [`files`](#files), -[`packages`](#packages), and -[`js_modules`](#javascript-modules)) and an experimental flag -([experimental_create_proxy](#experimental_create_proxy)) that can be used in +There are five core options ([`interpreter`](#interpreter), [`files`](#files), +[`packages`](#packages), [`js_modules`](#javascript-modules) and +[`sync_main_only`](#sync_main_only)) and an experimental flag +([`experimental_create_proxy`](#experimental_create_proxy)) that can be used in the configuration of PyScript. The user is also free to define arbitrary additional configuration options that plugins or an app may require for their own reasons. @@ -100,11 +100,11 @@ a custom version of the interpreter. The following two examples are equivalent: -```TOML title="Specify the interpreter version in TOML" +```TOML title="Specify the interpreter version in TOML." interpreter = "0.23.4" ``` -```JSON title="Specify the interpreter version in JSON" +```JSON title="Specify the interpreter version in JSON." { "interpreter": "0.23.4" } @@ -113,7 +113,7 @@ interpreter = "0.23.4" The following JSON fragment uses a fully qualified URL to point to the same version of Pyodide as specified in the previous examples: -```JSON title="Specify the interpreter via a fully qualified URL" +```JSON title="Specify the interpreter via a fully qualified URL." { "interpreter": "https://cdn.jsdelivr.net/pyodide/v0.23.4/full/pyodide.mjs" } @@ -127,7 +127,7 @@ destination filesystem path. The following JSON and TOML are equivalent: -```json title="Fetch files onto the filesystem with JSON" +```json title="Fetch files onto the filesystem with JSON." { "files": { "https://example.com/data.csv": "./data.csv", @@ -136,7 +136,7 @@ The following JSON and TOML are equivalent: } ``` -```toml title="Fetch files onto the filesystem with TOML" +```toml title="Fetch files onto the filesystem with TOML." [files] "https://example.com/data.csv" = "./data.csv" "/code.py" = "./subdir/code.py" @@ -147,7 +147,7 @@ URL becomes the destination filename, in the root of the filesystem, to which the content is copied. As a result, the `data.csv` entry from the previous examples could be equivalently re-written as: -```json title="JSON implied filename in the root directory" +```json title="JSON implied filename in the root directory." { "files": { "https://example.com/data.csv": "", @@ -156,7 +156,7 @@ examples could be equivalently re-written as: } ``` -```toml title="TOML implied filename in the root directory" +```toml title="TOML implied filename in the root directory." [files] "https://example.com/data.csv" = "" ... etc ... @@ -202,7 +202,7 @@ their name is replaced with their associated value. The following JSON and TOML are equivalent: -```json title="Using the template language in JSON" +```json title="Using the template language in JSON." { "files": { "{DOMAIN}": "https://my-server.com", @@ -218,7 +218,7 @@ The following JSON and TOML are equivalent: } ``` -```toml title="Using the template language in TOML" +```toml title="Using the template language in TOML." [files] "{DOMAIN}" = "https://my-server.com" "{PATH}" = "a/path" @@ -287,11 +287,11 @@ to be installed onto the Python path. The following two examples are equivalent: -```TOML title="A packages list in TOML" +```TOML title="A packages list in TOML." packages = ["arrr", "numberwang", "snowballstemmer>=2.2.0" ] ``` -```JSON title="A packages list in JSON" +```JSON title="A packages list in JSON." { "packages": ["arrr", "numberwang", "snowballstemmer>=2.2.0" ] } @@ -338,13 +338,13 @@ To specify such modules, simply provide a list of source/module name pairs. For example, to use the excellent [Leaflet](https://leafletjs.com/) JavaScript module for creating interactive maps you'd add the following lines: -```TOML title="JavaScript main thread modules defined in TOML" +```TOML title="JavaScript main thread modules defined in TOML." [js_modules.main] "https://cdn.jsdelivr.net/npm/leaflet@1.9.4/dist/leaflet-src.esm.js" = "leaflet" "https://cdn.jsdelivr.net/npm/leaflet@1.9.4/dist/leaflet.css" = "leaflet" # CSS ``` -```JSON title="JavaScript main thread modules defined in JSON" +```JSON title="JavaScript main thread modules defined in JSON." { "js_modules": { "main": { @@ -386,12 +386,12 @@ Some JavaScript modules (such as access to the DOM and, for efficiency reasons, can be included in the worker context: -```TOML title="A JavaScript worker module defined in TOML" +```TOML title="A JavaScript worker module defined in TOML." [js_modules.worker] "https://cdn.jsdelivr.net/npm/html-escaper" = "html_escaper" ``` -```JSON title="A JavaScript worker module defined in JSON" +```JSON title="A JavaScript worker module defined in JSON." { "js_modules": { "worker": { @@ -404,6 +404,40 @@ context: However, `from pyscript.js_modules import html_escaper` would then only work within the context of Python code **running on a worker**. +### sync_main_only + +Sometimes you just want to start an expensive computation on a web worker +without the need for the worker to interact with the main thread. You're simply +awaiting the result of a method exposed from a worker. + +This has the advantage of not requiring the use of `SharedArrayBuffer` and +[associated CORS related header configuration](../workers/#http-headers). + +If the `sync_main_only` flag is set, then **interactions between the main thread +and workers are limited to one way calls from the main thread to methods +exposed by the workers**. + +```TOML title="Setting the sync_main_only flag in TOML." +sync_main_only = true +``` + +```JSON title="Setting the sync_main_only flag in JSON." +{ + "sync_main_only": true +} +``` + +If `sync_main_only` is set, the following caveats apply: + +* It is not possible to manipulate the DOM or do anything meaningful on the + main thread **from a worker**. This is because Atomics cannot guarantee + sync-like locks between a worker and the main thread. +* Only a worker's `pyscript.sync` methods are exposed, and **they can only be + awaited from the main thread**. +* The worker can only `await` main thread references one after the other, so + developer experience is degraded when one needs to interact with the + main thread. + ### experimental_create_proxy Knowing when to use the `pyscript.ffi.create_proxy` method when using Pyodide diff --git a/docs/user-guide/workers.md b/docs/user-guide/workers.md index 3de35bf..2e54c68 100644 --- a/docs/user-guide/workers.md +++ b/docs/user-guide/workers.md @@ -11,6 +11,20 @@ You don't need to know about Atomics to use web workers, but the underlying [coincident library](http://localhost:8000/user-guide/architecture/#coincident) uses it under the hood. +!!! info + + Sometimes you only need to `await` in the main thread the result of a call + to a method exposed in a worker. + + In such a limited case, and on the understanding that **code in the worker + will not be able to reach back into the main thread**, you should + use the [`sync_main_only` flag](../configuration/#sync_main_only) in your + configuration. + + While this eliminates the need for the Atomics related header configuration + (see below), the only possible use case is to **return a serialisable + result from the method called on the worker**. + ## HTTP headers For Atomics to work **you must ensure your web server enables the following From 0b5c59302eec71e3753091ac07eda96878281f9b Mon Sep 17 00:00:00 2001 From: "Nicholas H.Tollervey" Date: Mon, 13 May 2024 11:31:05 +0100 Subject: [PATCH 5/5] Bump version to 2024.5.2 --- docs/beginning-pyscript.md | 8 ++++---- docs/user-guide/first-steps.md | 4 ++-- docs/user-guide/plugins.md | 10 +++++----- version.json | 2 +- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/beginning-pyscript.md b/docs/beginning-pyscript.md index 3d9611e..de6098b 100644 --- a/docs/beginning-pyscript.md +++ b/docs/beginning-pyscript.md @@ -112,8 +112,8 @@ module in the document's `` tag: 🦜 Polyglot - Piratical PyScript - - + + @@ -163,8 +163,8 @@ In the end, our HTML should look like this: 🦜 Polyglot - Piratical PyScript - - + +

Polyglot 🦜 💬 🇬🇧 ➡️ 🏴‍☠️

diff --git a/docs/user-guide/first-steps.md b/docs/user-guide/first-steps.md index a23f774..601b274 100644 --- a/docs/user-guide/first-steps.md +++ b/docs/user-guide/first-steps.md @@ -20,9 +20,9 @@ CSS: - + - + diff --git a/docs/user-guide/plugins.md b/docs/user-guide/plugins.md index 09223ce..84ee985 100644 --- a/docs/user-guide/plugins.md +++ b/docs/user-guide/plugins.md @@ -99,7 +99,7 @@ For example, this will work because all references are contained within the registered function: ```js -import { hooks } from "https://pyscript.net/releases/2024.5.1/core.js"; +import { hooks } from "https://pyscript.net/releases/2024.5.2/core.js"; hooks.worker.onReady.add(() => { // NOT suggested, just an example! @@ -113,7 +113,7 @@ hooks.worker.onReady.add(() => { However, due to the outer reference to the variable `i`, this will fail: ```js -import { hooks } from "https://pyscript.net/releases/2024.5.1/core.js"; +import { hooks } from "https://pyscript.net/releases/2024.5.2/core.js"; // NO NO NO NO NO! ☠️ let i = 0; @@ -146,7 +146,7 @@ the page. ```js title="log.js - a plugin that simply logs to the console." // import the hooks from PyScript first... -import { hooks } from "https://pyscript.net/releases/2024.5.1/core.js"; +import { hooks } from "https://pyscript.net/releases/2024.5.2/core.js"; // The `hooks.main` attribute defines plugins that run on the main thread. hooks.main.onReady.add((wrap, element) => { @@ -196,8 +196,8 @@ hooks.worker.onAfterRun.add(() => { - - + +