Skip to content

Commit

Permalink
update worker groups
Browse files Browse the repository at this point in the history
  • Loading branch information
rubenfiszel committed Sep 13, 2023
1 parent 775b013 commit 26f98e3
Show file tree
Hide file tree
Showing 5 changed files with 40 additions and 32 deletions.
24 changes: 11 additions & 13 deletions docs/core_concepts/25_dedicated_workers/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,26 +9,24 @@ For now it only work with bun scripts that have been pre-deployed.

The scripts can be used within flows.

To setup, pass:
To use:

```
DEDICATED_WORKER=<workspace>:<path>
```
1. Assign one worker group to execute that script

to the dedicated worker that will be tasked with executing that script.
![Worker group config](./worker_group_config.png)

or use the dedicatedWorkers section in the values.yaml of the helm chart.
The worker group will restart (assuming the pods/restart are set to restart automatically) and will now wait for step 2. below to happen

You will also need to toggle the "Dedicated Workers" option for that script in the metadata:
2. toggle the "Dedicated Workers" option for that script in the metadata:

![Dedicated Workers](./dedicated_workers.png)

You might also want to look for [Worker Group Tag](../9_worker_groups/index.mdx), which allows to assign custom worker groups to scripts and flows in Windmill for efficient execution on different machines with varying specifications:

<div class="grid grid-cols-2 gap-6 mb-4">
<DocCard
title="Workers and Worker Groups"
description="Worker Groups allow users to run scripts and flows on different machines with varying specifications."
href="/docs/core_concepts/worker_groups"
/>
</div>
<DocCard
title="Workers and Worker Groups"
description="Worker Groups allow users to run scripts and flows on different machines with varying specifications."
href="/docs/core_concepts/worker_groups"
/>
</div>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
48 changes: 29 additions & 19 deletions docs/core_concepts/9_worker_groups/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,31 +36,41 @@ Workers can belong to multiple worker groups simultaneously. The default worker

If you assign custom worker groups to all your workers, make sure that they cover all worker groups above, otherwise those jobs will never be executed.

You can customize the worker group of a worker by setting the WORKER_TAGS environment variable.
To make custom tags available from the UI, go to your workspace settings and click on the "Assignable Tags" button:

![Worker Group Assignable Tags](./worker_group_ui_2.png)

It is possible to restrict some tags to specific workspace using the following syntax:

```
WORKER_TAGS=deno,light,gpu
gpu(workspace+workspace2)
```

By default, the `WORKER_TAGS` of workers include `deno, python3, bash, go, flow, hub, dependency`.
Only 'workspace' and 'workspace2' will be able to use the `gpu` tags.

The configuration above will make the workers accept only deno jobs and jobs with the `light` tag.
## How to have a worker join a worker group

To make custom tags available from the UI and be authorized to be used, you need to pass the following env variable to the Windmill server:
Simply pass the worker group as the env variable `WORKER_GROUP`:

```
CUSTOM_TAGS=light,gpu
WORKER_GROUP=worker_group_name
```

You can restrict some tags to specific workspace using the following syntax:
The worker will automatically join the worker group on start and be displayed on the Workers page.

## How to assign worker tags to a worker group

Use the edit/create config next to the worker group name in windmill UI:

![Worker Group edit config](./worker_group_ui.png)

**Note**: The worker group management UI is an enterprise only feature. It is still possible to use worker groups with the community edition by passing to each worker the env variable `WORKER_TAGS`:

```
CUSTOM_TAGS=light,gpu(workspace+workspace2)
WORKER_TAGS=tag1,tag2
```

Only workspace and workspace2 will be able to use the `gpu` tags.

## How to assign a custom worker group
## How to assign a custom worker group to a script or flow

For scripts saved on the script editor, select the corresponding worker group tag in the metadata section.

Expand All @@ -80,10 +90,10 @@ You can assign a worker group to an entire flow in the flow's settings:

If a workspace tag contains the substring `$workspace`, it will be replaced by the workspace id corresponding to the job. This is especially useful to have the same script deployed to different workspace and have them run on different workers.

In the following setup:
With the following assignable tag:

```
CUSTOM_TAGS=normal-$workspace
normal-$workspace
```

the workspaces, `dev`, `staging`, `prod` and the worker groups: `normal-dev`, `normal-staging`, `normal-prod`. The same script wih the tag `normal-$workspace` will run on the corresponding worker group depending on the workspace it is deployed to. This enable to share the same control plane but use workers with different network restrictions for tighter security.
Expand All @@ -98,9 +108,9 @@ They are as fast as running the same logic in a forloop, but keep the benefit of
Dedicated Workers / Native Performance is an [Enterprise & Team Edition](../../misc/7_plans_details/index.mdx) feature.

<div class="grid grid-cols-2 gap-6 mb-4">
<DocCard
title="Dedicated Workers / Native Performance"
description="Dedicated Workers are workers that are dedicated to a particular script."
href="/docs/core_concepts/dedicated_workers"
/>
</div>
<DocCard
title="Dedicated Workers / Native Performance"
description="Dedicated Workers are workers that are dedicated to a particular script."
href="/docs/core_concepts/dedicated_workers"
/>
</div>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 26f98e3

Please sign in to comment.