diff --git a/blog/2023-11-21-workers-group/index.mdx b/blog/2023-11-21-workers-group/index.mdx index 4b0c7d189..d2daf8b35 100644 --- a/blog/2023-11-21-workers-group/index.mdx +++ b/blog/2023-11-21-workers-group/index.mdx @@ -104,7 +104,7 @@ When writing a script that should be executed on a specific worker group, you ha
diff --git a/docs/assets/flows/flow_advanced_settings.png b/docs/assets/flows/flow_advanced_settings.png new file mode 100644 index 000000000..f6bd5d3e4 Binary files /dev/null and b/docs/assets/flows/flow_advanced_settings.png differ diff --git a/docs/assets/flows/flow_example.png b/docs/assets/flows/flow_example.png index 80afdd3c8..46575ce15 100644 Binary files a/docs/assets/flows/flow_example.png and b/docs/assets/flows/flow_example.png differ diff --git a/docs/assets/flows/flow_example.png.webp b/docs/assets/flows/flow_example.png.webp deleted file mode 100644 index 095e114fa..000000000 Binary files a/docs/assets/flows/flow_example.png.webp and /dev/null differ diff --git a/docs/assets/flows/flow_triggers.png b/docs/assets/flows/flow_triggers.png new file mode 100644 index 000000000..2bcb95493 Binary files /dev/null and b/docs/assets/flows/flow_triggers.png differ diff --git a/docs/core_concepts/11_persistent_storage/within_windmill.mdx b/docs/core_concepts/11_persistent_storage/within_windmill.mdx index 44e031672..90dd6cc7a 100644 --- a/docs/core_concepts/11_persistent_storage/within_windmill.mdx +++ b/docs/core_concepts/11_persistent_storage/within_windmill.mdx @@ -12,7 +12,7 @@ Instead, Windmill is very convenient to use alongside data storage providers to There are however internal methods to persist data between executions of jobs. -## States and Resources +## States and resources Within Windmill, you can use [States](../3_resources_and_types/index.mdx#states) and [Resources](../3_resources_and_types/index.mdx) as a way to store a transient state - that can be represented as small JSON. @@ -98,7 +98,7 @@ _Python_ />
-### Custom Flow States +### Custom flow states Custom flow states are a way to store data across steps in a [flow](../../flows/1_flow_editor.mdx). You can set and retrieve a value given a key from any step of flow and it will be available from within the flow globally. That state will be stored in the flow state itself and thus has the same lifetime as the flow [job](../20_jobs/index.mdx) itself. @@ -176,30 +176,25 @@ In conclusion `setState` and `setResource` are convenient ways to persist json b /> -## Shared Directory +## Shared directory -For heavier ETL processes or sharing data between steps in a flow, Windmill provides a [Shared Directory](../../flows/3_editor_components.mdx#shared-directory) feature. - -The Shared Directory allows steps within a flow to share data by storing it in a designated folder. +For heavier ETL processes or sharing data between steps in a flow, Windmill provides a Shared Directory feature. This allows steps within a flow to share data by storing it in a designated folder at `./shared`. :::caution - -Although Shared Folders are recommended for persisting states within a flow, it's important to note that all steps are executed on the same worker and the data stored in the Shared Directory is strictly ephemeral to the flow execution. - +Although Shared Directories are recommended for persisting states within a flow, it's important to note that: +- All steps are executed on the same worker +- The data stored in the Shared Directory is strictly ephemeral to the flow execution +- The contents are not preserved across [suspends](../../flows/11_flow_approval.mdx) and [sleeps](../../flows/15_sleep.md) ::: -To enable the Shared Directory, follow these steps: - -1. Open the `Settings` menu in the Windmill interface. -2. Go to the `Shared Directory` section. -3. Toggle on the option for `Shared Directory on './shared'`. +To enable the Shared Directory: +1. Open the `Settings` menu in the Windmill interface +2. Go to the `Shared Directory` section +3. Toggle on the option for `Shared Directory on './shared'` ![Flow Shared Directory](../../assets/flows/flow_settings_shared_directory.png.webp) -Once the Shared Directory is enabled, you can use it in your flow by referencing the `./shared` folder. This folder is shared among the steps in the flow, allowing you to store and access data between them. - -:::tip - -Keep in mind that the contents of the `./shared` folder are not preserved across [suspends](../../flows/11_flow_approval.mdx) and [sleeps](../../flows/15_sleep.md). The directory is temporary and active only during the execution of the flow. - -::: \ No newline at end of file +Once enabled, steps can read and write files to the `./shared` folder to pass data between them. This is particularly useful for: +- Handling larger datasets that would be impractical to pass as step inputs/outputs +- Temporary storage of intermediate processing results +- Sharing files between steps in an ETL pipeline \ No newline at end of file diff --git a/docs/core_concepts/1_scheduling/index.mdx b/docs/core_concepts/1_scheduling/index.mdx index 80788358e..eb95d46b5 100644 --- a/docs/core_concepts/1_scheduling/index.mdx +++ b/docs/core_concepts/1_scheduling/index.mdx @@ -41,6 +41,23 @@ Cron jobs are one of many ways to [trigger workflows](../../getting_started/9_tr ::: +## Cron syntax + +Windmill uses [zslayton's cron expression parser](https://github.com/zslayton/cron). This library differs slightly from the Unix library. + +Although the syntaxes are similar, there are some notable differences: + +| Feature | Unix cron | zslayton's `cron` library | +|---------------------------|-------------------------|--------------------------------------------| +| **Day of Week Index** | Sunday = 0 through Saturday = 6 | Shifted by one (Sunday = 1 through Saturday = 7, 0 = Sunday) | +| **Seconds Field** | Not included | Included as the first field | +| **Year Field** | Not included | Optional, can specify specific years | +| **Month Representation** | Numeric and short names | Numeric, short names, and name ranges | +| **List and Range in Fields** | Supports lists and ranges | Supports lists, ranges, and combinations | +| **Step Values** | Supported (e.g., `*/2`) | Supported, including complex patterns like `2018/2` | + +Anyway, the simplified builder and [Windmill AI](../22_ai_generation/index.mdx) will help you to create the cron expression. + ## Set a schedule Scripts and flows can have unique [primary schedules](#primary-schedule) and multiple [other schedules](#other-schedules). @@ -150,22 +167,19 @@ More at: /> -## Cron syntax - -Windmill uses [zslayton's cron expression parser](https://github.com/zslayton/cron). This library differs slightly from the Unix library. +## Scheduled polls -Although the syntaxes are similar, there are some notable differences: +A particular use case for schedules are [Trigger scripts](../../flows/10_flow_trigger.mdx). -| Feature | Unix cron | zslayton's `cron` library | -|---------------------------|-------------------------|--------------------------------------------| -| **Day of Week Index** | Sunday = 0 through Saturday = 6 | Shifted by one (Sunday = 1 through Saturday = 7, 0 = Sunday) | -| **Seconds Field** | Not included | Included as the first field | -| **Year Field** | Not included | Optional, can specify specific years | -| **Month Representation** | Numeric and short names | Numeric, short names, and name ranges | -| **List and Range in Fields** | Supports lists and ranges | Supports lists, ranges, and combinations | -| **Step Values** | Supported (e.g., `*/2`) | Supported, including complex patterns like `2018/2` | + and have it return all of the new items since the last run as scheduled polls, without resorting to external webhooks. -Anyway, the simplified builder and [Windmill AI](../22_ai_generation/index.mdx) will help you to create the cron expression. +
+ +
## Control permissions and errors diff --git a/docs/core_concepts/20_jobs/index.mdx b/docs/core_concepts/20_jobs/index.mdx index be942e6e5..4d55f5234 100644 --- a/docs/core_concepts/20_jobs/index.mdx +++ b/docs/core_concepts/20_jobs/index.mdx @@ -126,6 +126,10 @@ You can set a custom retention period for the jobs runs details. The retention p /> +## High priority jobs + +High priority jobs are jobs that are given a `priority` value between 1 and 100. Jobs with a higher priority value will be given precedence over jobs with a lower priority value in the job queue. + ## Large job logs management To optimize log storage and performance, Windmill leverages S3 for log management. This approach minimizes database load by treating the database as a temporary buffer for up to 5000 characters of logs per job. diff --git a/docs/core_concepts/22_ai_generation/index.mdx b/docs/core_concepts/22_ai_generation/index.mdx index 41500d750..016fabb31 100644 --- a/docs/core_concepts/22_ai_generation/index.mdx +++ b/docs/core_concepts/22_ai_generation/index.mdx @@ -105,7 +105,7 @@ Generate a flow consisting of a sequence of scripts. ### Trigger Flows -[Trigger flows](../../flows/10_flow_trigger.mdx) are designed to pull data from an external source and return all of the new items since the last run, without resorting to external webhooks. A trigger script is intended to be used with [schedules](../1_scheduling/index.mdx) and [states](../3_resources_and_types/index.mdx#states) (rich objects in JSON, persistent from one run to another) in order to compare the execution to the previous one and process each new item in a [for loop](../../flows/12_flow_loops.md). +[Trigger flows](../../flows/10_flow_trigger.mdx) are designed to pull data from an external source and return all of the new items since the last run, without resorting to external webhooks. A trigger script is intended to be used as scheduled poll with [schedules](../1_scheduling/index.mdx) and [states](../3_resources_and_types/index.mdx#states) (rich objects in JSON, persistent from one run to another) in order to compare the execution to the previous one and process each new item in a [for loop](../../flows/12_flow_loops.md). If there are no new items, the flow will be skipped. diff --git a/docs/core_concepts/24_caching/index.md b/docs/core_concepts/24_caching/index.md index e35df8a86..ef4661a52 100644 --- a/docs/core_concepts/24_caching/index.md +++ b/docs/core_concepts/24_caching/index.md @@ -28,7 +28,7 @@ In the above example, the result of step the script will be cached for 180 secon ## Cache Flows -Caching a flow means caching the results of that script for a certain duration. If the flow is triggered with the same flow inputs during the given duration, it will return the cached result. +Caching a flow means caching the results of that flow for a certain duration. If the flow is triggered with the same flow inputs during the given duration, it will return the cached result.