Fixes of the afternoon

blog/2024-04-18-useful-python-scripts/index.md

@@ -39,7 +39,7 @@ To run Python scripts in Windmill, you first need access to Windmill (free on ou
 
 A Python script in Windmill consists of two parts: the code and the settings that include metadata and configurations. The code must have at least a main function. Once the Python environment is set up, you can proceed to write your script.
 
-Windmill automatically manages [dependencies](/docs/advanced/imports) for you. When you import libraries in your Python script, Windmill parses these imports upon saving the script and automatically generates a list of dependencies. It then spawns a dependency job to associate these PyPi packages with a lockfile, ensuring that the same version of the script is always executed with the same versions of its dependencies.
+Windmill automatically manages [dependencies](/docs/advanced/imports) for you. When you import libraries in your Python script, Windmill parses these imports upon saving the script and automatically generates a list of dependencies. It then spawns a dependency job to associate these PyPI packages with a lockfile, ensuring that the same version of the script is always executed with the same versions of its dependencies.
 
 Here is a simple example of a Python script in Windmill that performs sentiment analysis:
 

docs/advanced/14_dependencies_in_typescript/index.mdx

@@ -12,7 +12,7 @@ There are however methods to have more control on your dependencies:
 - Overriding dependencies [providing a package.json](#lockfile-per-script-inferred-from-a-packagejson).
 - [Bundling](#bundle-per-script-built-by-cli) per script with CLI, more powerful and local only.
 
-Moreover, there are two other tricks, compatible with the methodologies mentioned above:
+Moreover, there are other tricks, compatible with the methodologies mentioned above:
 
 - [Sharing common logic with Relative Imports](#sharing-common-logic-with-relative-imports-when-not-using-bundles) when not using Bundles.
 - [Private npm Registry & Private npm Packages](#private-npm-registry--private-npm-packages).

docs/advanced/15_dependencies_in_python/index.mdx

@@ -9,10 +9,11 @@ There are however methods to have more control on your dependencies:
 - Leveraging [standard mode](#lockfile-per-script-inferred-from-imports-standard) on [web IDE](#web-ide) or [locally](#cli).
 - Overriding dependencies [providing a requirements.txt](#lockfile-per-script-inferred-from-a-requirementstxt).
 
-Moreover, there are two other tricks, compatible with the methodologies mentioned above:
+Moreover, there are other tricks, compatible with the methodologies mentioned above:
 - [Sharing common logic with Relative Imports](#sharing-common-logic-with-relative-imports).
-- [Pinning Dependencies and Requirements](#pinning-dependencies-and-requirements).
-- [Private PyPi Repository](#private-pypi-repository).
+- [Pinning dependencies and requirements](#pinning-dependencies-and-requirements).
+- [Private PyPI Repository](#private-pypi-repository).
+- [Python runtime settings](#python-runtime-settings).
 
 To learn more about how dependencies from other languages are handled, see [Dependency management & imports](../../advanced/6_imports/index.mdx).
 
@@ -31,12 +32,12 @@ To learn more about how dependencies from other languages are handled, see [Depe
 	/>
 </div>
 
-## Lockfile per Script inferred from Imports (Standard)
+## Lockfile per script inferred from imports (Standard)
 
 In Windmill, you can run scripts without having to [manage a requirements.txt](#lockfile-per-script-inferred-from-a-requirementstxt) directly. This is achieved by automatically parsing the imports and resolving the dependencies.
 
 In Python, the imports are automatically parsed on saving of the script and a list of imports is generated.
-A dependency job is then spawned to associate that list of PyPi packages with a lockfile, which will lock
+A dependency job is then spawned to associate that list of PyPI packages with a lockfile, which will lock
 the versions. This ensures that the same version of a Python script is always
 executed with the same versions of its dependencies. It also avoids the hassle
 of having to maintain a separate requirements file.
@@ -87,7 +88,7 @@ When a lockfile is present alongside a script at time of deployment by the CLI,
 	/>
 </div>
 
-## Lockfile per Script inferred from a requirements.txt
+## Lockfile per script inferred from a requirements.txt
 
 Although Windmill can [automatically resolve imports](#lockfile-per-script-inferred-from-imports-standard). It is possible to override the dependencies by providing a `requirements.txt` file in the same directory as the script as you would do in a standard Python project, building and maintaining a requirements.txt to declare dependencies.
 
@@ -151,9 +152,9 @@ With this toggle, you can choose to use the metadata lockfile (derived from requ
 
 ## Other
 
-Other tricks can be used: [Sharing common logic with Relative Imports](#sharing-common-logic-with-relative-imports), [Pinning Dependencies and Requirements](#pinning-dependencies-and-requirements) and [Private PyPi Repository](#private-pypi-repository). All are compatible with the methods described above.
+Other tricks can be used: [Sharing common logic with relative imports](#sharing-common-logic-with-relative-imports), [Pinning dependencies and requirements](#pinning-dependencies-and-requirements) and [Private PyPI Repository](#private-pypi-repository). All are compatible with the methods described above.
 
-### Sharing common logic with Relative Imports
+### Sharing common logic with relative imports
 
 If you want to share common logic with Relative Imports, this can be done easily using [relative imports](../5_sharing_common_logic/index.mdx) in both Python and TypeScript.
 
@@ -204,7 +205,7 @@ scripts locally and on Windmill. See [Developing scripts locally](../4_local_dev
 	/>
 </div>
 
-### Pinning Dependencies and Requirements
+### Pinning dependencies and requirements
 
 If the imports are not properly analyzed, there exists an escape hatch to
 override the inferred imports. One needs to head the Script with the `requirements` comment followed by dependencies.
@@ -239,7 +240,7 @@ def main(...):
   ...
 ```
 
-### Private PyPi Repository
+### Private PyPI repository
 
 Environment variables can be set to customize `pip`'s index-url and extra-index-url and certificate.
 This is useful for private repositories.
@@ -257,4 +258,8 @@ windmill_worker:
 
 "Pip Index Url" and "Pip Extra Index Url" are filled through Windmill UI, in [Instance settings](../18_instance_settings/index.mdx#registries) under [Enterprise Edition](/pricing).
 
-![Private PyPi Repository](./private_pip.png 'Private PyPi Repository')
+![Private PyPI Repository](./private_pip.png 'Private PyPI Repository')
+
+### Python runtime settings
+
+For a given [worker group](../../core_concepts/9_worker_groups/index.mdx), you can add Python [runtime specific settings](../../core_concepts/9_worker_groups/index.mdx#python-runtime-settings) like additional Python paths and PIP local dependencies.

docs/core_concepts/2_variables_and_secrets/index.mdx

@@ -14,7 +14,7 @@ All Variables (not just secrets) are encrypted with a workspace specific symmetr
 
 There are two types of Variables in Windmill: user-defined and contextual.
 
-## User-defined Variables
+## User-defined variables
 
 User-defined Variables is essentially a key-value store where every user can
 read, set and share values at given keys as long as they have the privilege to
@@ -69,7 +69,7 @@ offer a selection over the resources of type postgresql in the auto-generated UI
 There is also a concept of [state](..//3_resources_and_types/index.mdx#states) to share values
 across script executions.
 
-## Contextual Variables
+## Contextual variables
 
 Contextual Variables are environment variables automatically set by Windmill. This is how the Deno and Python clients get their implicit
 credentials to interact with the platform.
@@ -100,7 +100,7 @@ You can use them in a Script by clicking on "+Context Var":
 | WM_OBJECT_PATH | u_user_flow_path/u_user_flow_path/c/17[...]196dd | Script or flow step execution unique path, useful for storing results in an external service |
 | WM_OIDC_JWT | eyJh[...]ciOiJIUzI1NiIsInR5 | OIDC JWT token (EE only) |
 
-### Custom Contextual Variables
+### Custom contextual variables
 
 From Variables tab, [admins](../16_roles_and_permissions/index.mdx) can create custom contextual variables that will act as env variables for all jobs within a workspace.
 
@@ -123,7 +123,7 @@ are kept safe in three different ways:
   the <a href="https://app.windmill.dev/audit_logs" rel="nofollow">Audit logs</a>. It means that you can audit who accesses secrets. Additionally you can audit results, logs and
   script code for every script run.
 
-## Add a Variable or Secret
+## Add a variable or secret
 
 You can define variables from the Variables page. Like all objects in
 Windmill, variable ownership is defined by the path - see
@@ -136,7 +136,7 @@ A variable can be made secret. In this case, its value will not be visible outsi
 
 ![Add variable](./add_variable.png.webp)
 
-## Accessing a Variable from a script
+## Accessing a variable from a script
 
 <div className="grid grid-cols-2 gap-6 mb-4">
 	<DocCard
@@ -146,7 +146,7 @@ A variable can be made secret. In this case, its value will not be visible outsi
 	/>
 </div>
 
-### Accessing Contextual Variables from a script
+### Accessing contextual variables from a script
 
 See the `Contextual` tab on the <a href="https://app.windmill.dev/variables" rel="nofollow">Variable page</a> for the list of reserved variables and what they are used for.
 
@@ -156,11 +156,11 @@ You can use them in a Script by clicking on "+Context Var":
 
 Reserved variables are passed to the job as environment variables. For example, the ephemeral token is passed as `WM_TOKEN`.
 
-### Accessing User-defined Variables from a script
+### Accessing user-defined variables from a script
 
 There are 2 main ways variables are used within scripts:
 
-#### Passing Variables as parameters to scripts
+#### Passing variables as parameters to scripts
 
 Variables can be passed as parameters of the script, using the UI-based variable picker. Underneath, the variable is passed as a string of the form: `$var:<variable_path>` and replaced by the worker at time of execution of the script by fetching the value with the job's permissions. So the job will fail if the job's permissions inherited from the caller do not allow access to the variable. This is the same mechanism used for resource, but they use `$res:` instead of `$var:`.
 
@@ -221,4 +221,16 @@ To add a custom environment variable to a script in Windmill, you should follow
 		description="In a self-hosted environment, Windmill allows you to set custom environment variables for your scripts."
 		href="/docs/script_editor/custom_environment_variables"
 	/>
+</div>
+
+## Environment variables passed to jobs
+
+From a [worker group](../9_worker_groups/index.mdx), you can add static and dynamic environment variables that will be [passed to jobs](../9_worker_groups/index.mdx#environment-variables-passed-to-jobs) handled by this worker group. Dynamic environment variable values will be loaded from the worker host environment variables while static environment variables will be set directly from their values below.
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Workers and worker groups"
+		description="Worker Groups allow users to run scripts and flows on different machines with varying specifications."
+		href="/docs/core_concepts/worker_groups"
+	/>
 </div>

docs/core_concepts/40_autoscaling/index.mdx

@@ -6,13 +6,33 @@ Autoscaling is available in the [Enterprise plan](/pricing).
 
 ## Autoscaling configuration
 
-You configure a minimum and maximum number of workers. The autoscaler will adjust the number of workers between the minimum and maximum based on the workload by calling a script which call your underlying infra orchestrator such as Kubernetes, ECS or Nomad. Coming soon, those will be hanlded natively by Windmill without the need for running a job.
+You configure a minimum and maximum number of [workers](../9_worker_groups/index.mdx). The autoscaler will adjust the number of workers between the minimum and maximum based on the workload by calling a script which call your underlying infra orchestrator such as Kubernetes, ECS or Nomad. Coming soon, those will be hanlded natively by Windmill without the need for running a job.
 
-Autoscaling is configured in each worker group config under "Autoscaling". It takes the following configuration:
-![Autoscaling](./png)
+Autoscaling is configured in each [worker group](/core_concepts/9_worker_groups/index.mdx) config under "Autoscaling". It takes the following configuration:
 
+![Autoscaling](./config.png)
 
-When using a custom script, the arguments that are passed to the script are, worker group, desired workers, reason, and event type. For instance, if you are using Kubernetes, you can use the following script:
+### Rules
+
+| Parameter | Description |
+|-----------|-------------|
+| Enabled | Whether autoscaling is enabled for the worker group |
+| Min # of Workers | The minimum number of workers to scale down to |
+| Max # of Workers | The maximum number of workers to scale up to |
+
+### Integration
+
+| Integration Type | Description |
+|-----------------|-------------|
+| Dry run | Test autoscaling behavior without making actual changes |
+| Custom script | Use your own script to handle scaling workers |
+| ECS | Native ECS integration (coming soon) |
+| Nomad | Native Nomad integration (coming soon) |
+| Kubernetes | Native Kubernetes integration (coming soon) |
+
+When using a custom script, you'll need to provide a path to script in the [admins workspace](../../advanced/18_instance_settings/index.mdx#admins-workspace), and optionally a custom [tag](../9_worker_groups/index.mdx#set-tags-to-assign-specific-queues) for executing the script.
+
+The arguments that are passed to the script are: worker group, desired workers, reason, and event type. For instance, if you are using Kubernetes, you can use the following script:
 
 ```bash
 worker_group="$1"
@@ -27,6 +47,21 @@ echo "Applying $event_type of $desired_workers to $worker_group bc $reason"
 kubectl scale deployment windmill-workers-$worker_group --replicas=$desired_workers -n $namespace
 ```
 
+You can directly test scaling from the config.
+
+### Advanced
+
+| Parameter | Description |
+|-----------|-------------|
+| Cooldown seconds after an incremental scale-in/out | Time to wait after an incremental scaling event before allowing another |
+| Cooldown seconds after a full scale out | Time to wait after a full scale out event before allowing another |
+| Num jobs waiting to trigger an incremental scale-out | Number of waiting jobs needed to trigger a gradual scale out |
+| Num jobs waiting to trigger a full scale out | Number of waiting jobs needed to trigger scaling to max workers (Default: max_workers, full scale out = scale out to max workers) |
+| Occupancy rate % threshold to go below to trigger a scale-in (decrease) | Default: 25%. When the average worker occupancy rate across 15s, 5m and 30m intervals falls below this threshold, the system will trigger a scale-in event to reduce the number of workers. This helps prevent having too many idle workers. |
+| Occupancy rate threshold to exceed to trigger an incremental scale-out (increase) | Default: 75%. When the average worker occupancy rate across 15s, 5m and 30m intervals exceeds this threshold, the system will trigger a scale-out event to add more workers. This helps ensure there is enough capacity to handle the workload. |
+| Num workers to scale-in/out by when incremental | Number of workers to add/remove during incremental scaling events (Default: (max_workers - min_workers) / 5) |
+| Custom tags to autoscale on | By default, autoscaling will apply to the tags the worker group is assigned to but you can override this here. |
+
 ## Autoscaling algorithm
 
 The autoscaling algorithm is based on the number of jobs waiting for workers in the queue for the tags that the worker group is listening from (tags can be overriden) and based on the occupancy rate of the workers of that worker group.
@@ -50,10 +85,7 @@ The function uses occupancy rates over different time periods (`occupancy_rate_1
 
 The goal is to have just enough workers to handle the current workload (represented by `queue_counts`) efficiently, without having too many idle workers or too few to handle the jobs.
 
-
-
-
 ## Autoscaling events
 
 Autoscaling events can be viewed under the worker group details:
-![Autoscaling events](./events.png)
+![Autoscaling events](./events.png)