Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for choosing between multiprocess and inprocess executors via cli flag #2895

Merged
merged 16 commits into from
Dec 1, 2023
Merged

Add support for choosing between multiprocess and inprocess executors via cli flag #2895

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
436bd83
Add support for running in-process executor.
rousik Sep 25, 2023
514cb51
Switch to --dagster-workers flags and refactor.
rousik Sep 26, 2023
6ac8b9d
Merge remote-tracking branch 'origin/dev' into inprocess
rousik Sep 26, 2023
9d776ab
Merge branch 'dev' into inprocess
rousik Oct 3, 2023
5aff2ee
update help command for the --dagster-workers flag
rousik Oct 3, 2023
f4195c1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 3, 2023
b874c19
Merge branch 'dev' into inprocess
rousik Oct 6, 2023
3007276
Merge branch 'dev' into inprocess
rousik Oct 10, 2023
eb92a84
Merge branch 'dev' into inprocess
rousik Oct 18, 2023
4d0d9f4
Merge branch 'dev' into inprocess
zaneselvans Oct 19, 2023
33254b5
Fix the type of --dagster-workers argument.
rousik Oct 21, 2023
2728ad6
Merge branch 'dev' into inprocess
rousik Nov 30, 2023
2c4c829
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 30, 2023
2190545
Add default --gcs-cache-path to simplify code below.
rousik Nov 30, 2023
7abe68a
Merge remote-tracking branch 'origin/dev' into inprocess
rousik Dec 1, 2023
cc81e2b
Update conda-lock.yml and rendered conda environment files.
rousik Dec 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions environments/conda-linux-64.lock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ dependencies:
- colorama=0.4.6=pyhd8ed1ab_0
- crashtest=0.4.1=pyhd8ed1ab_0
- cycler=0.12.1=pyhd8ed1ab_0
- dagster-pipes=1.5.9=pyhd8ed1ab_0
- dagster-pipes=1.5.10=pyhd8ed1ab_0
- dataclasses=0.8=pyhc8e2a94_3
- dbus=1.13.6=h5008d03_3
- debugpy=1.8.0=py311hb755f60_1
Expand Down Expand Up @@ -492,7 +492,7 @@ dependencies:
- aws-sdk-cpp=1.11.182=h8beafcf_7
- boto3=1.33.5=pyhd8ed1ab_0
- cachecontrol-with-filecache=0.13.1=pyhd8ed1ab_0
- dagster=1.5.9=pyhd8ed1ab_0
- dagster=1.5.10=pyhd8ed1ab_0
- datasette=0.64.4=pyhd8ed1ab_1
- doc8=1.1.1=pyhd8ed1ab_0
- email-validator=2.1.0.post1=pyhd8ed1ab_0
Expand All @@ -517,8 +517,8 @@ dependencies:
- timezonefinder=6.2.0=py311h459d7ec_2
- catalystcoop.ferc_xbrl_extractor=1.3.1=pyhd8ed1ab_0
- conda-lock=2.5.1=pyhd8ed1ab_0
- dagster-graphql=1.5.9=pyhd8ed1ab_0
- dagster-postgres=0.21.9=pyhd8ed1ab_1
- dagster-graphql=1.5.10=pyhd8ed1ab_0
- dagster-postgres=0.21.10=pyhd8ed1ab_0
- fiona=1.9.5=py311hf8e0aa6_1
- google-api-core=2.14.0=pyhd8ed1ab_0
- google-auth-oauthlib=1.1.0=pyhd8ed1ab_0
Expand All @@ -531,7 +531,7 @@ dependencies:
- qtconsole-base=5.5.1=pyha770c72_0
- recordlinkage=0.16=pyhd8ed1ab_0
- tabulator=1.53.5=pyhd8ed1ab_0
- dagster-webserver=1.5.9=pyhd8ed1ab_0
- dagster-webserver=1.5.10=pyhd8ed1ab_0
- geopandas=0.14.1=pyhd8ed1ab_0
- google-cloud-core=2.3.3=pyhd8ed1ab_0
- libarrow-acero=14.0.1=h59595ed_3_cpu
Expand Down
150 changes: 75 additions & 75 deletions environments/conda-lock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3792,15 +3792,15 @@ package:
category: main
optional: false
- name: dagster
version: 1.5.9
version: 1.5.10
manager: conda
platform: linux-64
dependencies:
alembic: ">=1.2.1,!=1.6.3,!=1.7.0,!=1.11.0"
click: ">=5.0"
coloredlogs: ">=6.1,<=14.0"
croniter: ">=0.3.34"
dagster-pipes: ">=1.5.9,<1.5.10.0a0"
dagster-pipes: ">=1.5.10,<1.5.11.0a0"
docstring_parser: ""
grpcio: ">=1.44.0"
grpcio-health-checking: ">=1.44.0"
Expand All @@ -3826,14 +3826,14 @@ package:
typing_extensions: ">=4.4.0"
universal_pathlib: ""
watchdog: ">=0.8.3"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-1.5.9-pyhd8ed1ab_0.conda
url: https://conda.anaconda.org/conda-forge/noarch/dagster-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: d8ab27112f82687ffcd456a3b88092e5
sha256: 238b08bf9afbc98405cb0c8c9845514da7b4b21aac5817c2b5f0de04e3a19b1b
md5: bcf42b675edb2999669116bc3b1ed789
sha256: dcee8473cbd1823005285f5fc82fbb16d9d1b2b46bcdf8371ee4fe2fdb0ad50c
category: main
optional: false
- name: dagster
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-64
dependencies:
Expand Down Expand Up @@ -3866,15 +3866,15 @@ package:
alembic: ">=1.2.1,!=1.6.3,!=1.7.0,!=1.11.0"
pydantic: ">1.10.0,!=1.10.7"
pendulum: <3
dagster-pipes: ">=1.5.9,<1.5.10.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-1.5.9-pyhd8ed1ab_0.conda
dagster-pipes: ">=1.5.10,<1.5.11.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: d8ab27112f82687ffcd456a3b88092e5
sha256: 238b08bf9afbc98405cb0c8c9845514da7b4b21aac5817c2b5f0de04e3a19b1b
md5: bcf42b675edb2999669116bc3b1ed789
sha256: dcee8473cbd1823005285f5fc82fbb16d9d1b2b46bcdf8371ee4fe2fdb0ad50c
category: main
optional: false
- name: dagster
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-arm64
dependencies:
Expand Down Expand Up @@ -3907,32 +3907,32 @@ package:
alembic: ">=1.2.1,!=1.6.3,!=1.7.0,!=1.11.0"
pydantic: ">1.10.0,!=1.10.7"
pendulum: <3
dagster-pipes: ">=1.5.9,<1.5.10.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-1.5.9-pyhd8ed1ab_0.conda
dagster-pipes: ">=1.5.10,<1.5.11.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: d8ab27112f82687ffcd456a3b88092e5
sha256: 238b08bf9afbc98405cb0c8c9845514da7b4b21aac5817c2b5f0de04e3a19b1b
md5: bcf42b675edb2999669116bc3b1ed789
sha256: dcee8473cbd1823005285f5fc82fbb16d9d1b2b46bcdf8371ee4fe2fdb0ad50c
category: main
optional: false
- name: dagster-graphql
version: 1.5.9
version: 1.5.10
manager: conda
platform: linux-64
dependencies:
dagster: ">=1.5.9,<1.5.10.0a0"
dagster: ">=1.5.10,<1.5.11.0a0"
gql-with-requests: ">=3.0.0"
graphene: ">=3"
python: ">=3.8"
requests: ""
starlette: ""
url: https://conda.anaconda.org/conda-forge/noarch/dagster-graphql-1.5.9-pyhd8ed1ab_0.conda
url: https://conda.anaconda.org/conda-forge/noarch/dagster-graphql-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 7dcd105a5451f9800aa6de278d86db72
sha256: 8484c6b0db1a3505fc7d16e83e0da75b9c886ae3d497266fd06f72fcd3246786
md5: fd3569582db65a0c88fbc7d1bb803853
sha256: a142f082da2b7905713b09e0ad19eef971a5d4f66063ba7a2b19247e3932e129
category: dev
optional: true
- name: dagster-graphql
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-64
dependencies:
Expand All @@ -3941,15 +3941,15 @@ package:
python: ">=3.8"
graphene: ">=3"
gql-with-requests: ">=3.0.0"
dagster: ">=1.5.9,<1.5.10.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-graphql-1.5.9-pyhd8ed1ab_0.conda
dagster: ">=1.5.10,<1.5.11.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-graphql-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 7dcd105a5451f9800aa6de278d86db72
sha256: 8484c6b0db1a3505fc7d16e83e0da75b9c886ae3d497266fd06f72fcd3246786
md5: fd3569582db65a0c88fbc7d1bb803853
sha256: a142f082da2b7905713b09e0ad19eef971a5d4f66063ba7a2b19247e3932e129
category: dev
optional: true
- name: dagster-graphql
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-arm64
dependencies:
Expand All @@ -3958,140 +3958,140 @@ package:
python: ">=3.8"
graphene: ">=3"
gql-with-requests: ">=3.0.0"
dagster: ">=1.5.9,<1.5.10.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-graphql-1.5.9-pyhd8ed1ab_0.conda
dagster: ">=1.5.10,<1.5.11.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-graphql-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 7dcd105a5451f9800aa6de278d86db72
sha256: 8484c6b0db1a3505fc7d16e83e0da75b9c886ae3d497266fd06f72fcd3246786
md5: fd3569582db65a0c88fbc7d1bb803853
sha256: a142f082da2b7905713b09e0ad19eef971a5d4f66063ba7a2b19247e3932e129
category: dev
optional: true
- name: dagster-pipes
version: 1.5.9
version: 1.5.10
manager: conda
platform: linux-64
dependencies:
python: ">=3.8"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-pipes-1.5.9-pyhd8ed1ab_0.conda
url: https://conda.anaconda.org/conda-forge/noarch/dagster-pipes-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 0a9787859365c4d2425e589ac53c462b
sha256: eebc7dca517350678ebfb8b3fff7ec47c60aff62dae2e69b8c4845b6080ec3e8
md5: 69600c68efc23fb84c05c2e9c1c05947
sha256: a98a8b2501af4bc112d5b2f5e3edea6c22a084651cc720786c405877b8507630
category: main
optional: false
- name: dagster-pipes
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-64
dependencies:
python: ">=3.8"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-pipes-1.5.9-pyhd8ed1ab_0.conda
url: https://conda.anaconda.org/conda-forge/noarch/dagster-pipes-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 0a9787859365c4d2425e589ac53c462b
sha256: eebc7dca517350678ebfb8b3fff7ec47c60aff62dae2e69b8c4845b6080ec3e8
md5: 69600c68efc23fb84c05c2e9c1c05947
sha256: a98a8b2501af4bc112d5b2f5e3edea6c22a084651cc720786c405877b8507630
category: main
optional: false
- name: dagster-pipes
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-arm64
dependencies:
python: ">=3.8"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-pipes-1.5.9-pyhd8ed1ab_0.conda
url: https://conda.anaconda.org/conda-forge/noarch/dagster-pipes-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 0a9787859365c4d2425e589ac53c462b
sha256: eebc7dca517350678ebfb8b3fff7ec47c60aff62dae2e69b8c4845b6080ec3e8
md5: 69600c68efc23fb84c05c2e9c1c05947
sha256: a98a8b2501af4bc112d5b2f5e3edea6c22a084651cc720786c405877b8507630
category: main
optional: false
- name: dagster-postgres
version: 0.21.9
version: 0.21.10
manager: conda
platform: linux-64
dependencies:
dagster: 1.5.9.*
dagster: 1.5.10.*
psycopg2-binary: ""
python: ">=3.8"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-postgres-0.21.9-pyhd8ed1ab_1.conda
url: https://conda.anaconda.org/conda-forge/noarch/dagster-postgres-0.21.10-pyhd8ed1ab_0.conda
hash:
md5: 8c1a941fe77b920b1c7933a7a0c6bf2e
sha256: 0e947f376d6878bd8e505932277e84c373da492a38d2c4ef9d96fc25f5327845
md5: ec155b3a7172590ccbc89f461427b5aa
sha256: 4eb986655ef547d4dc72cd34b60687c9c3c390806493c15187c4d26d89d58fc0
category: main
optional: false
- name: dagster-postgres
version: 0.21.9
version: 0.21.10
manager: conda
platform: osx-64
dependencies:
psycopg2-binary: ""
python: ">=3.8"
dagster: 1.5.9.*
url: https://conda.anaconda.org/conda-forge/noarch/dagster-postgres-0.21.9-pyhd8ed1ab_1.conda
dagster: 1.5.10.*
url: https://conda.anaconda.org/conda-forge/noarch/dagster-postgres-0.21.10-pyhd8ed1ab_0.conda
hash:
md5: 8c1a941fe77b920b1c7933a7a0c6bf2e
sha256: 0e947f376d6878bd8e505932277e84c373da492a38d2c4ef9d96fc25f5327845
md5: ec155b3a7172590ccbc89f461427b5aa
sha256: 4eb986655ef547d4dc72cd34b60687c9c3c390806493c15187c4d26d89d58fc0
category: main
optional: false
- name: dagster-postgres
version: 0.21.9
version: 0.21.10
manager: conda
platform: osx-arm64
dependencies:
psycopg2-binary: ""
python: ">=3.8"
dagster: 1.5.9.*
url: https://conda.anaconda.org/conda-forge/noarch/dagster-postgres-0.21.9-pyhd8ed1ab_1.conda
dagster: 1.5.10.*
url: https://conda.anaconda.org/conda-forge/noarch/dagster-postgres-0.21.10-pyhd8ed1ab_0.conda
hash:
md5: 8c1a941fe77b920b1c7933a7a0c6bf2e
sha256: 0e947f376d6878bd8e505932277e84c373da492a38d2c4ef9d96fc25f5327845
md5: ec155b3a7172590ccbc89f461427b5aa
sha256: 4eb986655ef547d4dc72cd34b60687c9c3c390806493c15187c4d26d89d58fc0
category: main
optional: false
- name: dagster-webserver
version: 1.5.9
version: 1.5.10
manager: conda
platform: linux-64
dependencies:
click: ">=7.0,<9.0"
dagster: ">=1.5.9,<1.5.10.0a0"
dagster-graphql: ">=1.5.9,<1.5.10.0a0"
dagster: ">=1.5.10,<1.5.11.0a0"
dagster-graphql: ">=1.5.10,<1.5.11.0a0"
python: ">=3.8"
starlette: ""
uvicorn-standard: ""
url: https://conda.anaconda.org/conda-forge/noarch/dagster-webserver-1.5.9-pyhd8ed1ab_0.conda
url: https://conda.anaconda.org/conda-forge/noarch/dagster-webserver-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 880fa7acdbf3494cef45759bb866bb63
sha256: 2fce08b607d97f72d7452350a0c917d96419074381bf8791ebe116ec3a57b8f4
md5: 6161623733c03c21098dce0af904ea6b
sha256: 975173f6a39f40d5fa505354007200741689714bb1eaf1cba8e52fab1a2bfc88
category: dev
optional: true
- name: dagster-webserver
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-64
dependencies:
starlette: ""
uvicorn-standard: ""
python: ">=3.8"
click: ">=7.0,<9.0"
dagster: ">=1.5.9,<1.5.10.0a0"
dagster-graphql: ">=1.5.9,<1.5.10.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-webserver-1.5.9-pyhd8ed1ab_0.conda
dagster: ">=1.5.10,<1.5.11.0a0"
dagster-graphql: ">=1.5.10,<1.5.11.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-webserver-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 880fa7acdbf3494cef45759bb866bb63
sha256: 2fce08b607d97f72d7452350a0c917d96419074381bf8791ebe116ec3a57b8f4
md5: 6161623733c03c21098dce0af904ea6b
sha256: 975173f6a39f40d5fa505354007200741689714bb1eaf1cba8e52fab1a2bfc88
category: dev
optional: true
- name: dagster-webserver
version: 1.5.9
version: 1.5.10
manager: conda
platform: osx-arm64
dependencies:
starlette: ""
uvicorn-standard: ""
python: ">=3.8"
click: ">=7.0,<9.0"
dagster: ">=1.5.9,<1.5.10.0a0"
dagster-graphql: ">=1.5.9,<1.5.10.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-webserver-1.5.9-pyhd8ed1ab_0.conda
dagster: ">=1.5.10,<1.5.11.0a0"
dagster-graphql: ">=1.5.10,<1.5.11.0a0"
url: https://conda.anaconda.org/conda-forge/noarch/dagster-webserver-1.5.10-pyhd8ed1ab_0.conda
hash:
md5: 880fa7acdbf3494cef45759bb866bb63
sha256: 2fce08b607d97f72d7452350a0c917d96419074381bf8791ebe116ec3a57b8f4
md5: 6161623733c03c21098dce0af904ea6b
sha256: 975173f6a39f40d5fa505354007200741689714bb1eaf1cba8e52fab1a2bfc88
category: dev
optional: true
- name: dask-core
Expand Down
10 changes: 5 additions & 5 deletions environments/conda-osx-64.lock.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ dependencies:
- colorama=0.4.6=pyhd8ed1ab_0
- crashtest=0.4.1=pyhd8ed1ab_0
- cycler=0.12.1=pyhd8ed1ab_0
- dagster-pipes=1.5.9=pyhd8ed1ab_0
- dagster-pipes=1.5.10=pyhd8ed1ab_0
- dataclasses=0.8=pyhc8e2a94_3
- debugpy=1.8.0=py311hdf8f085_1
- decorator=5.1.1=pyhd8ed1ab_0
Expand Down Expand Up @@ -473,7 +473,7 @@ dependencies:
- uvicorn-standard=0.24.0.post1=h6eed73b_0
- boto3=1.33.5=pyhd8ed1ab_0
- cachecontrol-with-filecache=0.13.1=pyhd8ed1ab_0
- dagster=1.5.9=pyhd8ed1ab_0
- dagster=1.5.10=pyhd8ed1ab_0
- datasette=0.64.4=pyhd8ed1ab_1
- doc8=1.1.1=pyhd8ed1ab_0
- email-validator=2.1.0.post1=pyhd8ed1ab_0
Expand All @@ -497,8 +497,8 @@ dependencies:
- timezonefinder=6.2.0=py311he705e18_2
- catalystcoop.ferc_xbrl_extractor=1.3.1=pyhd8ed1ab_0
- conda-lock=2.5.1=pyhd8ed1ab_0
- dagster-graphql=1.5.9=pyhd8ed1ab_0
- dagster-postgres=0.21.9=pyhd8ed1ab_1
- dagster-graphql=1.5.10=pyhd8ed1ab_0
- dagster-postgres=0.21.10=pyhd8ed1ab_0
- fiona=1.9.5=py311h809632c_1
- google-api-core=2.14.0=pyhd8ed1ab_0
- google-auth-oauthlib=1.1.0=pyhd8ed1ab_0
Expand All @@ -514,7 +514,7 @@ dependencies:
- qtconsole-base=5.5.1=pyha770c72_0
- recordlinkage=0.16=pyhd8ed1ab_0
- tabulator=1.53.5=pyhd8ed1ab_0
- dagster-webserver=1.5.9=pyhd8ed1ab_0
- dagster-webserver=1.5.10=pyhd8ed1ab_0
- geopandas=0.14.1=pyhd8ed1ab_0
- google-cloud-core=2.3.3=pyhd8ed1ab_0
- libarrow-dataset=14.0.1=hc222712_3_cpu
Expand Down
Loading
Loading