Refresh doc, missing pages, backlinks (#755)

blog/2023-11-21-workers-group/index.mdx

@@ -104,7 +104,7 @@ When writing a script that should be executed on a specific worker group, you ha
 
 <div className="grid grid-cols-2 gap-6 mb-4">
 	<DocCard
-		title="Worker Groups and Tags"
+		title="Worker groups and tags"
 		description="Worker Groups allow users to run scripts and flows on different machines with varying specifications."
 		href="/docs/core_concepts/worker_groups"
 	/>

docs/core_concepts/11_persistent_storage/within_windmill.mdx

@@ -12,7 +12,7 @@ Instead, Windmill is very convenient to use alongside data storage providers to
 
 There are however internal methods to persist data between executions of jobs.
 
-## States and Resources
+## States and resources
 
 Within Windmill, you can use [States](../3_resources_and_types/index.mdx#states) and [Resources](../3_resources_and_types/index.mdx) as a way to store a transient state - that can be represented as small JSON.
 
@@ -98,7 +98,7 @@ _Python_
 	/>
 </div>
 
-### Custom Flow States
+### Custom flow states
 
 Custom flow states are a way to store data across steps in a [flow](../../flows/1_flow_editor.mdx). You can set and retrieve a value given a key from any step of flow and it will be available from within the flow globally. That state will be stored in the flow state itself and thus has the same lifetime as the flow [job](../20_jobs/index.mdx) itself.
 
@@ -176,30 +176,25 @@ In conclusion `setState` and `setResource` are convenient ways to persist json b
 	/>
 </div>
 
-## Shared Directory
+## Shared directory
 
-For heavier ETL processes or sharing data between steps in a flow, Windmill provides a [Shared Directory](../../flows/3_editor_components.mdx#shared-directory) feature.
-
-The Shared Directory allows steps within a flow to share data by storing it in a designated folder.
+For heavier ETL processes or sharing data between steps in a flow, Windmill provides a Shared Directory feature. This allows steps within a flow to share data by storing it in a designated folder at `./shared`.
 
 :::caution
-
-Although Shared Folders are recommended for persisting states within a flow, it's important to note that all steps are executed on the same worker and the data stored in the Shared Directory is strictly ephemeral to the flow execution.
-
+Although Shared Directories are recommended for persisting states within a flow, it's important to note that:
+- All steps are executed on the same worker
+- The data stored in the Shared Directory is strictly ephemeral to the flow execution
+- The contents are not preserved across [suspends](../../flows/11_flow_approval.mdx) and [sleeps](../../flows/15_sleep.md)
 :::
 
-To enable the Shared Directory, follow these steps:
-
-1. Open the `Settings` menu in the Windmill interface.
-2. Go to the `Shared Directory` section.
-3. Toggle on the option for `Shared Directory on './shared'`.
+To enable the Shared Directory:
+1. Open the `Settings` menu in the Windmill interface
+2. Go to the `Shared Directory` section
+3. Toggle on the option for `Shared Directory on './shared'`
 
 ![Flow Shared Directory](../../assets/flows/flow_settings_shared_directory.png.webp)
 
-Once the Shared Directory is enabled, you can use it in your flow by referencing the `./shared` folder. This folder is shared among the steps in the flow, allowing you to store and access data between them.
-
-:::tip
-
-Keep in mind that the contents of the `./shared` folder are not preserved across [suspends](../../flows/11_flow_approval.mdx) and [sleeps](../../flows/15_sleep.md). The directory is temporary and active only during the execution of the flow.
-
-:::
+Once enabled, steps can read and write files to the `./shared` folder to pass data between them. This is particularly useful for:
+- Handling larger datasets that would be impractical to pass as step inputs/outputs
+- Temporary storage of intermediate processing results
+- Sharing files between steps in an ETL pipeline

docs/core_concepts/1_scheduling/index.mdx

@@ -41,6 +41,23 @@ Cron jobs are one of many ways to [trigger workflows](../../getting_started/9_tr
 
 :::
 
+## Cron syntax
+
+Windmill uses [zslayton's cron expression parser](https://github.com/zslayton/cron). This library differs slightly from the Unix library.
+
+Although the syntaxes are similar, there are some notable differences:
+
+| Feature                   | Unix cron               | zslayton's `cron` library                        |
+|---------------------------|-------------------------|--------------------------------------------|
+| **Day of Week Index**        | Sunday = 0 through Saturday = 6 | Shifted by one (Sunday = 1 through Saturday = 7, 0 = Sunday) |
+| **Seconds Field**         | Not included            | Included as the first field                |
+| **Year Field**            | Not included            | Optional, can specify specific years       |
+| **Month Representation**  | Numeric and short names | Numeric, short names, and name ranges      |
+| **List and Range in Fields** | Supports lists and ranges | Supports lists, ranges, and combinations |
+| **Step Values**           | Supported (e.g., `*/2`) | Supported, including complex patterns like `2018/2` |
+
+Anyway, the simplified builder and [Windmill AI](../22_ai_generation/index.mdx) will help you to create the cron expression.
+
 ## Set a schedule
 
 Scripts and flows can have unique [primary schedules](#primary-schedule) and multiple [other schedules](#other-schedules).
@@ -150,22 +167,19 @@ More at:
 	/>
 </div>
 
-## Cron syntax
-
-Windmill uses [zslayton's cron expression parser](https://github.com/zslayton/cron). This library differs slightly from the Unix library.
+## Scheduled polls
 
-Although the syntaxes are similar, there are some notable differences:
+A particular use case for schedules are [Trigger scripts](../../flows/10_flow_trigger.mdx).
 
-| Feature                   | Unix cron               | zslayton's `cron` library                        |
-|---------------------------|-------------------------|--------------------------------------------|
-| **Day of Week Index**        | Sunday = 0 through Saturday = 6 | Shifted by one (Sunday = 1 through Saturday = 7, 0 = Sunday) |
-| **Seconds Field**         | Not included            | Included as the first field                |
-| **Year Field**            | Not included            | Optional, can specify specific years       |
-| **Month Representation**  | Numeric and short names | Numeric, short names, and name ranges      |
-| **List and Range in Fields** | Supports lists and ranges | Supports lists, ranges, and combinations |
-| **Step Values**           | Supported (e.g., `*/2`) | Supported, including complex patterns like `2018/2` |
+ and have it return all of the new items since the last run as scheduled polls, without resorting to external webhooks.
 
-Anyway, the simplified builder and [Windmill AI](../22_ai_generation/index.mdx) will help you to create the cron expression.
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Trigger scripts"
+		description="Trigger scripts are designed to pull data from an external source and return all of the new items since the last run, without resorting to external webhooks."
+		href="/docs/flows/flow_trigger"
+	/>
+</div>
 
 ## Control permissions and errors
 

docs/core_concepts/20_jobs/index.mdx

@@ -126,6 +126,10 @@ You can set a custom retention period for the jobs runs details. The retention p
 	/>
 </div>
 
+## High priority jobs
+
+High priority jobs are jobs that are given a `priority` value between 1 and 100. Jobs with a higher priority value will be given precedence over jobs with a lower priority value in the job queue.
+
 ## Large job logs management
 
 To optimize log storage and performance, Windmill leverages S3 for log management. This approach minimizes database load by treating the database as a temporary buffer for up to 5000 characters of logs per job.

docs/core_concepts/22_ai_generation/index.mdx

@@ -105,7 +105,7 @@ Generate a flow consisting of a sequence of scripts.
 
 ### Trigger Flows
 
-[Trigger flows](../../flows/10_flow_trigger.mdx) are designed to pull data from an external source and return all of the new items since the last run, without resorting to external webhooks. A trigger script is intended to be used with [schedules](../1_scheduling/index.mdx) and [states](../3_resources_and_types/index.mdx#states) (rich objects in JSON, persistent from one run to another) in order to compare the execution to the previous one and process each new item in a [for loop](../../flows/12_flow_loops.md).
+[Trigger flows](../../flows/10_flow_trigger.mdx) are designed to pull data from an external source and return all of the new items since the last run, without resorting to external webhooks. A trigger script is intended to be used as scheduled poll with [schedules](../1_scheduling/index.mdx) and [states](../3_resources_and_types/index.mdx#states) (rich objects in JSON, persistent from one run to another) in order to compare the execution to the previous one and process each new item in a [for loop](../../flows/12_flow_loops.md).
 
 If there are no new items, the flow will be skipped.
 

docs/core_concepts/24_caching/index.md

@@ -28,7 +28,7 @@ In the above example, the result of step the script will be cached for 180 secon
 
 ## Cache Flows
 
-Caching a flow means caching the results of that script for a certain duration. If the flow is triggered with the same flow inputs during the given duration, it will return the cached result.
+Caching a flow means caching the results of that flow for a certain duration. If the flow is triggered with the same flow inputs during the given duration, it will return the cached result.
 
 <video
     className="border-2 rounded-xl object-cover w-full h-full"

docs/core_concepts/40_websocket_triggers/index.mdx

@@ -1,8 +1,8 @@
 # Websocket triggers
 
-Windmill can connect to websocket servers and trigger runnables when a message is received.
+Windmill can connect to [websocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) servers and trigger runnables (scripts, flows) when a message is received.
 Listening is done from the servers, so it doesn't take up any workers.
-Websocket triggers are not available on the Cloud.
+Websocket triggers are not available on the [Cloud](/pricing).
 
 ## How to use
 

docs/core_concepts/5_monitor_past_and_future_runs/index.mdx

@@ -25,7 +25,7 @@ You might also be interested in Audit logs:
 	/>
 </div>
 
-## Aggregated View
+## Aggregated view
 
 The Runs menu in each workspace provides a time series view where you can monitor different time slots.
 The green (respectively, red) dots being the tasks that succeeded (respectively, failed).
@@ -44,7 +44,7 @@ The graph can represent jobs on their Duration (default) or by Concurrency, mean
 
 > Graphical view by concurrent jobs.
 
-## Details per Run
+## Details per run
 
 Clicking on each run in the menu opens a run page where you can view the run's state, inputs, and success/failure reasons.
 
@@ -80,7 +80,7 @@ Example of filters in use:
 
 > Here were filtered successful runs from August 2023 which returned `{"baseNumber": 11}`.
 
-## Jobs Labels
+## Jobs labels
 
 Labels allow to add static or dynamic tags to [jobs](../20_jobs/index.mdx) with property "wm_labels" followed by an array of strings.
 
@@ -94,4 +94,8 @@ export async function main() {
 
 Jobs can be filtered by labels in the Runs menu to monitor specific groups of jobs.
 
-![Run labels](./runs_labels.png 'Run labels')
+![Run labels](./runs_labels.png 'Run labels')
+
+## Invisible runs
+
+When this option is enabled, manual executions of this script or flow are invisible to users other than the user running it, including the owner(s). This setting can be overridden when this script is run manually from the advanced menu.

docs/core_concepts/9_worker_groups/index.mdx

@@ -110,7 +110,9 @@ You can assign groups to flows and flow steps to be executed on specific queues.
 
 <br />
 
-There are 2 worker groups by default: _default_ and _native_.
+There are 2 worker groups by default: [default](#default-worker-group) and [native](#native-worker-group).
+
+#### Default worker group
 
 The tags of _default_ worker group are:
 
@@ -127,6 +129,11 @@ The tags of _default_ worker group are:
 
 Button `Reset to all tags minus native ones` will reset the tags of _default_ worker group to a given worker group.
 
+#### Native worker group
+
+Native workers are workers within the _native_ worker group.
+This group is pre-configured to listen to native jobs tags. Those jobs are executed under a special mode with subworkers for increased throughput.
+
 The tags of _native_ worker group are:
 
 - `nativets`: The default worker group for rest scripts.
@@ -138,6 +145,8 @@ The tags of _native_ worker group are:
 - `bigquery`: The default worker group for bigquery scripts.
 - `mssql`: The default worker group for mssql scripts.
 
+<br/><br/>
+
 If you assign custom worker groups to all your workers, make sure that they cover all tags above, otherwise those jobs will never be executed.
 
 Button `Reset to native tags` will reset the tags of _native_ worker group to a given worker group.
@@ -158,6 +167,8 @@ gpu(workspace+workspace2)
 
 Only 'workspace' and 'workspace2' will be able to use the `gpu` tags.
 
+Jobs within a same job queue can be given a [priority](../20_jobs/index.mdx#high-priority-jobs) between 1 and 100. Jobs with a higher priority value will be given precedence over jobs with a lower priority value in the job queue.
+
 ### How to assign worker tags to a worker group
 
 Use the edit/create config next to the worker group name in Windmill UI:

docs/core_concepts/index.mdx

@@ -77,6 +77,11 @@ On top of its editors to build endpoints, flows and apps, Windmill comes with a
 		description="There are 5 ways to do error handling in Windmill."
 		href="/docs/core_concepts/error_handling"
 	/>
+	<DocCard
+		title="Jobs"
+		description="A job represents a past, present or future `task` or `work` to be executed by a worker."
+		href="/docs/core_concepts/jobs"
+	/>
 	<DocCard
 		title="Jobs runs"
 		description="Get an aggregated view of past and future runs on your workspace."
@@ -127,6 +132,11 @@ On top of its editors to build endpoints, flows and apps, Windmill comes with a
 		description="Trigger scripts and flows from custom HTTP routes."
 		href="/docs/core_concepts/http_routing"
 	/>
+	<DocCard
+		title="Websocket triggers"
+		description="Trigger scripts and flows from websocket servers."
+		href="/docs/core_concepts/websocket_triggers"
+	/>
 	<DocCard
 		title="Caching"
 		description="Caching is used to cache the results of a script, flow, flow step or app inline scripts for a specified number of seconds."
@@ -137,11 +147,6 @@ On top of its editors to build endpoints, flows and apps, Windmill comes with a
 		description="In Windmill, JSON is the primary data format used for representing information. When working with binary data, such as files, they are represented as Base64 encoded strings."
 		href="/docs/core_concepts/files_binary_data"
 	/>
-	<DocCard
-		title="Jobs"
-		description="A job represents a past, present or future `task` or `work` to be executed by a worker."
-		href="/docs/core_concepts/jobs"
-	/>
 	<DocCard
 		title="Service logs"
 		description="View logs from any worker or servers directly within the service logs section of the search modal."

docs/flows/10_flow_trigger.mdx

@@ -2,7 +2,7 @@ import DocCard from '@site/src/components/DocCard';
 
 # Trigger scripts
 
-Trigger scripts are designed to pull data from an external source and return all of the new items since the last run, without resorting to external webhooks. A trigger script is intended to be used with [schedules](../core_concepts/1_scheduling/index.mdx) and [states](../core_concepts/3_resources_and_types/index.mdx#states) (rich objects in JSON, persistent from one run to another) in order to compare the execution to the previous one and process each new item in a [for loop](../flows/12_flow_loops.md). If there are no new items, the flow will be skipped.
+Trigger scripts are designed to pull data from an external source and return all of the new items since the last run, without resorting to external webhooks. A trigger script is intended to be used as scheduled poll with [schedules](../core_concepts/1_scheduling/index.mdx) and [states](../core_concepts/3_resources_and_types/index.mdx#states) (rich objects in JSON, persistent from one run to another) in order to compare the execution to the previous one and process each new item in a [for loop](../flows/12_flow_loops.md). If there are no new items, the flow will be skipped.
 
 By default, adding a trigger will set the schedule to 15 minutes.
 
@@ -33,6 +33,8 @@ Check our pages dedicated to [Scheduling](../core_concepts/1_scheduling/index.md
 	/>
 </div>
 
+## Scheduled polls
+
 :::tip
 
 Think of this as someone who checks the mailbox every day. If there is a new
@@ -50,7 +52,9 @@ activating the schedule as seen in the image below.
 
 Example of a trigger script watching new Slack posts with a given word in a given channel and the flow sending each of them by email in a for loop:
 
-![Example of a schedule script with a for loop](../getting_started/9_trigger_flows/schedule-flow.png 'Example of a schedule script with a for loop')
+![Example of a schedule script with a for loop](../getting_started/9_trigger_flows/schedule-script.png 'Example of a schedule script with a for loop')
+
+![Schedule](../getting_started/9_trigger_flows/schedule-flow.png 'Schedule')
 
 > This flow can be found on [WindmillHub](https://hub.windmill.dev/flows/51/watch-slack-posts-containing-a-given-word-and-send-all-new-ones-per-email).
 

docs/flows/1_flow_editor.mdx

@@ -69,18 +69,18 @@ The Flow editor has the following features which are the subject of specific pag
 		description="Iterate quickly and get control on your flow testing."
 		href="/docs/flows/test_flows"
 	/>
-	<DocCard
-		color="teal"
-		title="AI-generated flows"
-		description="Generate flows from prompts."
-		href="/docs/flows/ai_flows"
-	/>
 	<DocCard
 		color="teal"
 		title="Flow editor components"
 		description="Details on the flow editor's major components."
 		href="/docs/flows/editor_components"
 	/>
+	<DocCard
+		color="teal"
+		title="AI-generated flows"
+		description="Generate flows from prompts."
+		href="/docs/flows/ai_flows"
+	/>
 	<DocCard
 		color="teal"
 		title="Error handling in flows"
@@ -187,7 +187,7 @@ The Flow editor has the following features which are the subject of specific pag
 
 <br />
 
-![Example of a flow](../assets/flows/flow_example.png.webp)
+![Example of a flow](../assets/flows/flow_example.png)
 
 > _Example of a [flow](https://hub.windmill.dev/flows/38/automatically-populate-crm-contact-details-from-simple-email) in Windmill._
 

docs/flows/2_early_stop.md

@@ -2,7 +2,7 @@
 
 If defined, at the end or before a step, the predicate expression will be evaluated to decide if the flow should stop early.
 
-## Early Stop for Step
+## Early stop for step
 
 For each step of the flow, an early stop can be defined. The result of the step will be compared to a previously defined predicate expression, and if the condition is met, the flow will stop after that step.
 
@@ -26,7 +26,7 @@ For each step of the flow, an early stop can be defined. The result of the step
 
 If stop early is run within a forloop, it will just break the for-loop and have it stop at that iteration instead of stopping the whole flow.
 
-## Early Stop for Flow
+## Early stop for flow
 
 To stop early the flow based on the flow inputs, you can set an "Early Stop" from the flow settings. If the inputs meet the predefined condition, the flow will not run.