Skip to content

Commit

Permalink
New changelogs (#290)
Browse files Browse the repository at this point in the history
* New changelogs

* Up to date with new changelogs
  • Loading branch information
hcourdent authored Sep 5, 2023
1 parent 3d9b9ec commit be65950
Show file tree
Hide file tree
Showing 61 changed files with 331 additions and 107 deletions.
31 changes: 31 additions & 0 deletions docs/apps/3_app-runnable-panel.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import DocCard from '@site/src/components/DocCard';

# Runnable Guide

Windmill Apps can be powered by code, with scripts linked to components, background runnables that are not linked to a component but run in the background, frontend scripts that can be used to manipulate the client app state, and evals.
Expand Down Expand Up @@ -293,6 +295,35 @@ Other component can trigger a component to recompute. For example, a button can

Button and Form components can trigger other components to recompute. For example, a button can trigger a table to recompute. When the button is clicked, the table is reloaded.

## Cache app inline scripts

Caching an app inline script means caching the results of that script for a certain duration. If the script is triggered with the same inputs during the given duration, it will return the cached result.

<video
className="border-2 rounded-xl object-cover w-full h-full"
controls
src="/videos/caching_app.mp4"
/>

<br/>

You can enable caching for an app inline script directly its editor settings. Here's how you can do it:

1. **Settings**: From the Code Editor, go to the top bar and pick the `Cache` tab.

2. **Enable Caching**: To enable caching, toggle on "Cache the results for each possible inputs" and specify the desired duration for caching results (in seconds.)

In the above example, the result of step the script will be cached for 5 minutes. If `Inline Script 0` is re-triggered with the same input within this period, Windmill will immediately return the cached result.

<div class="grid grid-cols-2 gap-6 mb-4">
<DocCard
color="orange"
title="Caching in Windmill"
description="Caching is used to cache the results of a script, flow, flow step or app inline scripts for a specified number of seconds."
href="/docs/core_concepts/caching"
/>
</div>

## Policy

A viewer of the app will execute the runnables of the app on behalf of the publisher avoiding the risk that a resource or script would not be available to the viewer. To guarantee tight security, a policy is computed at time of saving of the app which only allow the scripts/flows referred to in the app to be called on behalf of. Furthermore, static parameters are not overridable. Hence, users will only be able to use the app as intended by the publisher without risk for leaking resources not used in the app.
Binary file added docs/assets/flows/concurrency_flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/assets/flows/flow_for_loop.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/assets/flows/flow_for_loop.png.avif
Binary file not shown.
Binary file removed docs/assets/flows/flow_for_loop.png.webp
Binary file not shown.
Binary file added docs/assets/flows/iter_value_index.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/assets/integrations/psql-1-resources.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/assets/integrations/psql-1-resources.png.avif
Binary file not shown.
Binary file removed docs/assets/integrations/psql-1-resources.png.webp
Binary file not shown.
Binary file modified docs/assets/integrations/psql-2-postgres-rt.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/assets/integrations/psql-2-postgres-rt.png.avif
Binary file not shown.
Binary file removed docs/assets/integrations/psql-2-postgres-rt.png.webp
Binary file not shown.
22 changes: 18 additions & 4 deletions docs/core_concepts/21_concurrency_limits/index.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,27 @@
# Concurrency Limits

The Concurrency Limits feature allows you to define concurrency limits for scripts and inline scripts within flows. Its primary goal is to prevent exceeding the API Limit of the targeted API, eliminating the need for complex workarounds using worker groups.
The Concurrency Limits feature allows you to define concurrency limits for scripts, flows and inline scripts within flows. Its primary goal is to prevent exceeding the API Limit of the targeted API, eliminating the need for complex workarounds using worker groups.

The Concurrency Limit operates globally and across flow runs. It involves two key parameters: "Maximum number of runs" and the "Per time window (seconds)."

When jobs reach the concurrency limit, they are automatically queued for execution at the next available optimal slot given the time window.

## Concurrency limit of Script

Concurrency limit can be set from the metadata menu. Pick "Concurrency" and define a time window and max number of executions of the flow within that time window.

![Concurrency Limit](../../assets/code_editor/concurrency_limit.png.webp)

## How it Works
## Concurrency limit of flow

From the Flow Settings menu, pick "Concurrency" and define a time window and max number of executions of the flow within that time window.

![Concurrency limit of flow](../../assets/flows/concurrency_flow.png "Concurrency limit of flow")

## Concurrency limit of scripts within flow

The Concurrency Limit operates globally and across flow runs. It involves two key parameters: "Maximum number of runs" and the "Per time window (seconds)."

Concurrency limit can be set from the metadata menu.
Concurrency limit can be set for each step of a flow in the `Advanced` menu, on tab "Concurrency".

When jobs reach the concurrency limit, they are automatically queued for execution at the next available optimal slot given the time window.
![Concurrency Limit Scripts within Flow](../../assets/code_editor/concurrency_limit_flow.png.webp "Concurrency Limit Scripts within Flow")
101 changes: 101 additions & 0 deletions docs/core_concepts/24_caching/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Caching

Caching is used to cache the results of a script, flow, flow step or app inline scripts for a specified number of seconds, thereby reducing the need for redundant computations when re-running the same step with identical input.

When you configure caching, Windmill stores the result in a cache for the duration you specify. If the same runnable is re-triggered with the same input within this duration, Windmill instantly retrieves the cached result instead of re-computing it.

This feature can significantly improve the performance of your scripts & flows, especially for steps that are computationally demanding or dependent on external resources, such as APIs or databases.

## Cache scripts

Caching a script means caching the results of that script for a certain duration. If the script is triggered with the same inputs during the given duration, it will return the cached result.

<video
className="border-2 rounded-xl object-cover w-full h-full"
controls
src="/videos/caching_script.mp4"
/>

<br/>

You can enable caching for a script directly in the script metadata. Here's how you can do it:

1. **Metadata**: From the Script Editor, pick the "Metdata" menu an scroll to "Cache".

2. **Enable Caching**: To enable caching, toggle on "Cache the results for each possible inputs" and specify the desired duration for caching results (in seconds.)

In the above example, the result of step the script will be cached for 180 seconds (4 minutes). If `u/henri/slow_function___base_11` is re-triggered with the same input within this period, Windmill will immediately return the cached result.

## Cache flows

Caching a flow means caching the results of that script for a certain duration. If the flow is triggered with the same flow inputs during the given duration, it will return the cached result.

<video
className="border-2 rounded-xl object-cover w-full h-full"
controls
src="/videos/caching_flow.mp4"
/>

<br/>

You can enable caching for a flow directly in the flow settings. Here's how you can do it:

1. **Settings**: From the Flow Editor, go to the "Settings" menu and pick the `Cache` tab.

2. **Enable Caching**: To enable caching, toggle on "Cache the results for each possible inputs" and specify the desired duration for caching results (in seconds.)

In the above example, the result of step the flow will be cached for 60 seconds. If `u/henri/example_flow_quickstart_no_slack` is re-triggered with the same input within this period, Windmill will immediately return the cached result.

## Cache flow steps

Caching a flow step means caching the results of that step for a certain duration. If the step is triggered with the same inputs during the given duration, it will return the cached result.

<video
className="border-2 rounded-xl object-cover w-full h-full"
controls
src="/videos/cache_for_steps.mp4"
/>

<br/>

You can enable caching for a step directly in the step configuration. Here's how you can do it:

1. **Select a step**: From the Flow Editor, select the step for which you want to cache the results.

2. **Enable Caching**: To enable caching, navigate to the `Advanced` menu and select `Cache`. Toggle it on and specify the desired duration for caching results (in seconds.)

![Caching result example](../../assets/flows/cache_steps.gif)

In the above example, the result of step `a` will be cached for 86400 seconds (1 day). If `a` is re-triggered with the same input within this period, Windmill will immediately return the cached result.

:::tip Step Mocking

[Step mocking](../../flows/5_step_mocking.md) allows faster iteration. When a step is mocked, it will immediately return the mocked value without performing any computation.

:::

## Cache app inline scripts

Caching an app inline script means caching the results of that script for a certain duration. If the script is triggered with the same inputs during the given duration, it will return the cached result.

<video
className="border-2 rounded-xl object-cover w-full h-full"
controls
src="/videos/caching_app.mp4"
/>

<br/>

You can enable caching for an app inline script directly its editor settings. Here's how you can do it:

1. **Settings**: From the Code Editor, go to the top bar and pick the `Cache` tab.

2. **Enable Caching**: To enable caching, toggle on "Cache the results for each possible inputs" and specify the desired duration for caching results (in seconds.)

In the above example, the result of step the script will be cached for 5 minutes. If `Inline Script 0` is re-triggered with the same input within this period, Windmill will immediately return the cached result.

## Conclusion

Caching is a powerful tool that can optimize your workflows in Windmill. By caching the results, you can eliminate unnecessary computation and accelerate your workflows.

However, it's essential to use caching judiciously: while it can greatly enhance efficiency, overusing the cache or using it for unsuitable scripts or flows can result in outdated results or unwarranted memory usage. Always consider the nature of your step and the data it processes when deciding whether to enable caching.
Binary file modified docs/core_concepts/3_resources_and_types/add_resource.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified docs/core_concepts/3_resources_and_types/add_resource_type.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 4 additions & 4 deletions docs/core_concepts/3_resources_and_types/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ To create a resource using an existing type, go to the
[Resources](https://app.windmill.dev/resources) page and click "Add a
resource/API".

![Add a resource](./add_resource.png.webp)
![Add a resource](./add_resource.png)

Just like most objects in Windmill, Resources have a path that define their
permissions - see [ownership path prefix](../16_roles_and_permissions/index.mdx).
Expand All @@ -49,7 +49,7 @@ Each **Resource** has a **Resource Type**, that defines what fields that
resource contains. Select one from the list and check the schema to see what
fields are present.

![Add resource example](./add_resource_postgresql.png.webp)
![Add resource example](./add_sf_resource.png)

Resources commonly need to access secrets or re-use
[Variables](../2_variables_and_secrets/index.mdx), for example passwords or API
Expand All @@ -76,13 +76,13 @@ Windmill comes preloaded with some common Resource Types, such as databases, app
can also add custom Resource Types by clicking "Add a resource type" on the
[Resources](https://app.windmill.dev/resources) page.

![Create resource type](./add_resource_type.png.webp)
![Create resource type](./add_resource_type.png)

Use the "Add Property" button to add a field to the resource type. You can
specify constraints for the field (a type, making it mandatory, specifying a
default, etc). You can also view the schema by toggling the "As JSON" option:

![Resource type schema view](./resource_type_json.png.webp)
![Resource type schema view](./resource_type_json.png)

## States

Expand Down
Binary file modified docs/core_concepts/3_resources_and_types/resource_type_json.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Binary file modified docs/core_concepts/5_monitor_past_and_future_runs/4-filters.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
40 changes: 20 additions & 20 deletions docs/core_concepts/5_monitor_past_and_future_runs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ Each workspace has a dedicated Runs menu that allows you to visualise all past a

<br />

You might also be intereted in Audit Logs:

<div class="grid grid-cols-2 gap-6 mb-4">
<DocCard
title="Audit Logs"
Expand All @@ -25,46 +27,44 @@ Each workspace has a dedicated Runs menu that allows you to visualise all past a

## Aggregated View

The Runs menu in each workspace provides a time series view where you can monitor different time slots. The green (respectively, red) dots being the tasks that succeeded (respectively, failed). You can adjust the level of details by picking "All", "Runs", "Previews" or "Dependencies".
The Runs menu in each workspace provides a time series view where you can monitor different time slots.
The green (respectively, red) dots being the tasks that succeeded (respectively, failed).

![Time series](./1-runs-menu.png.webp 'Time series')
![Time series](./1-runs-menu.png 'Time series')

> All past and future runs of the workspace are visible from the menu
> All past and future runs of the workspace are visible from the menu.
## Details per Run

Clicking on each run in the menu opens a run page where you can view the run's state, inputs, and success/failure reasons.

View of a past run:

![Details per run](./2-detail-per-run.png.webp 'Details per run')
![Details per run](./2-detail-per-run.png 'Details per run')

> Check the details of each runs
> View of a past run.
<br />

You can also view [scheduled](../1_scheduling/index.md) runs from the run menu, which provides information on the arguments used and the next trigger.

View of a scheduled run:

![Schedule run](./3-scheduled-run.png.webp)
![Schedule run](./3-scheduled-run.png)

> Although scheduled scripts and flows are visible on their dedicated tab, the run menu helps you see the next scheduled one
> Although scheduled scripts and flows are visible on their [dedicated tab](../1_scheduling/index.md), the run menu helps you see the next scheduled one.
## Filters

There are multiple ways to filter the Runs menu:

- by Datetime
- Success / Failure
- Skipped / Not skipped
- Runs / Previews / Dependencies / All
- by [Path](../16_roles_and_permissions/index.mdx#path)
- by Arguments
- by Result
You can adjust the level of details by picking playing with filters on:
- **Datetime**
- **Metadata**: [Path](../16_roles_and_permissions/index.mdx#path) / User / Folder
- **Kind**: All / Runs / Previews / Dependencies
- **Status**: All / Success / Failure
- **Skipped flows**
- **Arguments**
- **Results**

Example of filters in use:

![Filters](./4-filters.png.webp 'Filters')
![Filters](./4-filters.png 'Filters')

> Here were filtered successful runs which returned `{"message": "Created record in table"}`
> Here were filtered successful runs from August 2023 which returned `{"baseNumber": 11}`.
7 changes: 6 additions & 1 deletion docs/core_concepts/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,14 @@ On top of its editors to build endpoints, flows and apps, Windmill comes with a
/>
<DocCard
title="Concurrency Limits"
description="The Concurrency Limits feature allows you to define concurrency limits for scripts and inline scripts within flows."
description="The Concurrency Limits feature allows you to define concurrency limits for scripts, flows and inline scripts within flows."
href="/docs/core_concepts/concurrency_limits"
/>
<DocCard
title="Caching in Windmill"
description="Caching is used to cache the results of a script, flow, flow step or app inline scripts for a specified number of seconds."
href="/docs/core_concepts/caching"
/>
<DocCard
title="Draft and Deploy"
description="Develop and cooperate in a structured way."
Expand Down
13 changes: 10 additions & 3 deletions docs/flows/12_flow_loops.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,17 @@ For loops is a special type of steps that allows you to iterate over a list of i
<br/>

Clicking on the `For loop` step on the mini-map, it will open the `For loop` step editor.
There are three configuration options:
There are 4 configuration options:

- **Iterator expression**: the expression that will be evaluated to get the list of items to iterate over. You can also connect with a previous result that contain several items, it will iterate over all of them.
- **Iterator expression**: the [JS expression](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Expressions_and_Operators) that will be evaluated to get the list of items to iterate over. You can also [connect with a previous result](./16_architecture.md) that contain several items, it will iterate over all of them.
- **Skip failure**: if set to `true`, the loop will continue to the next item even if the current item failed.
- **Run in parallel**: if set to `true`, all iterations will be run in parallel.
- **Parallelism**: assign a given number of parallel branches to control huge for-loops.

![For loop step](../assets/flows/flow_for_loop.png.webp)
![For loop step](../assets/flows/flow_for_loop.png "For loop step")

<br/>

Steps within the flow can use both the iteration index and value. For example in `["Paris", "Lausanne", "Lille"]`, for iteration index "1", "Lausanne" is the value.

![Iter value & index](../assets/flows/iter_value_index.png "Iter value & index")
25 changes: 17 additions & 8 deletions docs/flows/2_early_stop.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,19 @@
# Early Stop / Break

If defined, at the end of the step, the predicate expression will be evaluated to decide if the flow should stop early.
If defined, at the end or before a step, the predicate expression will be evaluated to decide if the flow should stop early.

## Early Stop for Step

For each step of the flow, an early stop can be defined. The result of the step will be compared to a previously defined predicate expression, and if the condition is met, the flow will stop after that step.

<video
className="border-2 rounded-xl object-cover w-full h-full dark:border-gray-800"
autoPlay
loop
controls
id="main-video"
src="/videos/early_stop.mp4"
/>

<br/>

If stop early is run within a forloop, it will just break the for-loop and have it stop at that iteration instead of stopping the whole flow.

## Add Early Stop

- Pick the step you want to stop after.

- Go to `Advanced`, then `Early stop/Break`.
Expand All @@ -26,3 +23,15 @@ If stop early is run within a forloop, it will just break the for-loop and have
- Write the condition you want to stop on, based on the step's results.

- Optionally, toggle on "Label flow as "skipped" if stopped". Skipped flows are just a label useful to not see them in the [runs](../core_concepts/5_monitor_past_and_future_runs/index.mdx) page.

If stop early is run within a forloop, it will just break the for-loop and have it stop at that iteration instead of stopping the whole flow.

## Early Stop for Flow

To stop early the flow based on the flow inputs, you can set an "Early Stop" from the flow settings. If the inputs meet the predefined condition, the flow will not run.

<video
className="border-2 rounded-xl object-cover w-full h-full dark:border-gray-800"
controls
src="/videos/early_stop_flow.mp4"
/>
4 changes: 2 additions & 2 deletions docs/flows/3_editor_components.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -330,8 +330,8 @@ The header is composed of:
- **Summary**: edit the summary of the script.
- **Shortcuts**: shortcut to advanced configuration.
1. [Retries](./14_retries.md): configure the number of retries and the delay between each retry.
2. [Concurrency limit](./6_concurrency_limit.md): set concurrency limits to prevent exceeding the API Limit of the targeted API.
3. [Cache](./4_cache.md): cache the results of a step for a specified time.
2. [Concurrency limit](./6_concurrency_limit.mdx): set concurrency limits to prevent exceeding the API Limit of the targeted API.
3. [Cache](./4_cache.mdx): cache the results of a step for a specified time.
4. [Early stop/Break](./2_early_stop.md): if defined, at the end of the step, the predicate expression will be evaluated to decide if the flow should stop early. Skipped flows are just a label useful to not see them in the runs page. If stop early is run within a forloop, it will just break the for-loop and have it stop at that iteration instead of stopping the whole flow.
5. [Suspend](./11_flow_approval.md): if defined, at the end of the step, the flow will be suspended until it receives external requests to be resumed or canceled. This is most useful to implement approval steps but can be used flexibly for other purpose. To get the resume urls, use `wmill.getResumeUrls()` in Typescript, or `wmill.get_resume_urls()` in Python.
6. [Sleep](./15_sleep.md): if defined, at the end of the step, the flow will sleep for a number of seconds before scheduling the next job (if any, no effect if the step is the last one). Sleeping is passive and does not consume any resources.
Expand Down
Loading

0 comments on commit be65950

Please sign in to comment.