Skip to content

Commit

Permalink
Updates to deployment docs (dask#10778)
Browse files Browse the repository at this point in the history
  • Loading branch information
scharlottej13 authored Jan 16, 2024
1 parent 69d209f commit cdcddb6
Showing 1 changed file with 36 additions and 28 deletions.
64 changes: 36 additions & 28 deletions docs/source/deploying.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,16 @@ Deploy Dask Clusters
:columns: 12 12 5 5

Dask works well at many scales ranging from a single machine to clusters of
many machines. This section describes the many ways to deploy and run Dask,
including the following:
many machines. This section describes the many ways to deploy and run Dask, including the following:

- :doc:`deploying-python`
- :doc:`deploying-hpc`
- :doc:`deploying-kubernetes`
- :doc:`deploying-cloud`

.. toctree::
:maxdepth: 1
:hidden:

deploying-python.rst
deploying-cli.rst
Expand Down Expand Up @@ -75,69 +80,72 @@ cluster managers, and so it's easy to swap out when you're ready to scale up.

The following resources explain how to set up Dask on a variety of local and distributed hardware.

Manual deployments (not recommended)
------------------------------------

You can set up Dask clusters by hand, or with tools like SSH.

- :doc:`Manual Setup <deploying-cli>`
The command line interface to set up ``dask-scheduler`` and ``dask-worker`` processes.
- :doc:`deploying-ssh`
Use SSH to set up Dask across an un-managed cluster.
- :doc:`Python API (advanced) <deploying-python-advanced>`
Create ``Scheduler`` and ``Worker`` objects from Python as part of a distributed Tornado TCP application.

However, we don't recommend this path. Instead, we recommend that you use
some common resource manager to help you manage your machines, and then deploy
Dask on that system. Those options are described below.

High Performance Computing
--------------------------

See :doc:`deploying-hpc` for more details.

- `Dask-Jobqueue <https://jobqueue.dask.org>`_
Provides cluster managers for PBS, SLURM, LSF, SGE and other resource managers.
- `Dask-MPI <http://mpi.dask.org/en/latest/>`_
Deploy Dask from within an existing MPI environment.
- `Dask Gateway for Jobqueue <https://gateway.dask.org/install-jobqueue.html>`_
Multi-tenant, secure clusters. Once configured, users can launch clusters without direct access to the underlying HPC backend.

See :doc:`deploying-hpc` for more details.

Kubernetes
----------

See :doc:`deploying-kubernetes` for more details.

- `Dask Kubernetes Operator <https://kubernetes.dask.org/en/latest/operator.html>`_
For native Kubernetes integration for fast moving or ephemeral deployments.
- `Dask Gateway for Kubernetes <https://gateway.dask.org/install-kube.html>`_
Multi-tenant, secure clusters. Once configured, users can launch clusters without direct access to the underlying Kubernetes backend.
- `Single Cluster Helm Chart <https://artifacthub.io/packages/helm/dask/dask>`_
Single Dask cluster and (optionally) Jupyter on deployed with Helm.

See :doc:`deploying-kubernetes` for more details.

Cloud
-----

See :doc:`deploying-cloud` for more details.

- `Dask-Yarn <https://yarn.dask.org>`_
Deploy Dask on YARN clusters, such as are found in traditional Hadoop installations.
- `Dask Cloud Provider <https://cloudprovider.dask.org/en/latest/>`_
Constructing and managing ephemeral Dask clusters on AWS, DigitalOcean, Google Cloud, Azure, and Hetzner
- `Coiled <https://coiled.io?utm_source=dask-docs&utm_medium=deploying>`_
Commercial Dask deployment option, which handles the creation and management of Dask clusters on cloud computing environments (AWS and GCP).

See :doc:`deploying-cloud` for more details.

.. _managed-cluster-solutions:

Managed Solutions
-----------------

- `Coiled <https://coiled.io?utm_source=dask-docs&utm_medium=deploying>`_ manages the creation and management of Dask clusters on cloud computing environments (AWS and GCP).
- `Domino Data Lab <https://www.dominodatalab.com/>`_ lets users create Dask clusters in a hosted platform.
- `Saturn Cloud <https://saturncloud.io/>`_ lets users create Dask clusters in a hosted platform or within their own AWS accounts.
- `Coiled <https://coiled.io?utm_source=dask-docs&utm_medium=deploying>`_
Manages the creation and management of Dask clusters on cloud computing environments (AWS and GCP).
- `Domino Data Lab <https://www.dominodatalab.com/>`_
Lets users create Dask clusters in a hosted platform.
- `Saturn Cloud <https://saturncloud.io/>`_
Lets users create Dask clusters in a hosted platform or within their own AWS accounts.

Manual deployments (not recommended)
------------------------------------

You can set up Dask clusters by hand, or with tools like SSH.

- :doc:`Manual Setup <deploying-cli>`
The command line interface to set up ``dask-scheduler`` and ``dask-worker`` processes.
- :doc:`deploying-ssh`
Use SSH to set up Dask across an un-managed cluster.
- :doc:`Python API (advanced) <deploying-python-advanced>`
Create ``Scheduler`` and ``Worker`` objects from Python as part of a distributed Tornado TCP application.

However, we don't recommend this path. Instead, we recommend that you use
some common resource manager to help you manage your machines, and then deploy
Dask on that system. Those options are described above.

Advanced Understanding
----------------------

There are additional concepts to understand if you want to improve your
deployment.
deployment. :doc:`This guide <deployment-considerations>` covers the main topics to consider in addition to running Dask.

0 comments on commit cdcddb6

Please sign in to comment.