-
Notifications
You must be signed in to change notification settings - Fork 730
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #4164 from hhunter-ms/issue_3915
Jobs [Distributed Scheduler] API docs, pt.1
- Loading branch information
Showing
19 changed files
with
307 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| --- | ||
| type: docs | ||
| title: "Dapr Scheduler control plane service overview" | ||
| linkTitle: "Scheduler" | ||
| description: "Overview of the Dapr scheduler service" | ||
| --- | ||
|
|
||
| The Dapr Scheduler service is used to schedule jobs, running in [self-hosted mode]({{< ref self-hosted >}}) or on [Kubernetes]({{< ref kubernetes >}}). | ||
|
|
||
| The diagram below shows how the Scheduler service is used via the jobs API when called from your application. All the jobs that are tracked by the Scheduler service are stored in an embedded Etcd database. | ||
|
|
||
| <img src="/images/scheduler/scheduler-architecture.png" alt="Diagram showing the Scheduler control plane service and the jobs API"> | ||
|
|
||
| ## Self-hosted mode | ||
|
|
||
| The Scheduler service Docker container is started automatically as part of `dapr init`. It can also be run manually as a process if you are running in [slim-init mode]({{< ref self-hosted-no-docker.md >}}). | ||
|
|
||
| ## Kubernetes mode | ||
|
|
||
| The Scheduler service is deployed as part of `dapr init -k`, or via the Dapr Helm charts. For more information on running Dapr on Kubernetes, visit the [Kubernetes hosting page]({{< ref kubernetes >}}). | ||
|
|
||
| ## Related links | ||
|
|
||
| [Learn more about the Jobs API.]({{< ref jobs_api.md >}}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7 changes: 7 additions & 0 deletions
7
daprdocs/content/en/developing-applications/building-blocks/jobs/_index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| type: docs | ||
| title: "Jobs" | ||
| linkTitle: "Jobs" | ||
| weight: 120 | ||
| description: "Manage the scheduling and orchestration of jobs" | ||
| --- |
30 changes: 30 additions & 0 deletions
30
.../content/en/developing-applications/building-blocks/jobs/howto-schedule-jobs.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| type: docs | ||
| title: "How-To: Schedule jobs" | ||
| linkTitle: "How-To: Schedule jobs" | ||
| weight: 2000 | ||
| description: "Learn how to use the jobs API to schedule jobs" | ||
| --- | ||
|
|
||
| Now that you've learned what the [jobs building block]({{< ref jobs-overview.md >}}) provides, let's look at an example of how to use the API. The code example below describes an application that schedules and orchestrates ***TBD***. | ||
|
|
||
| <!-- | ||
| Include a diagram or image, if possible. | ||
| --> | ||
|
|
||
| <!-- | ||
| Make sure the how-to includes examples for multiple programming languages, OS, or deployment targets, if applicable. | ||
| --> | ||
|
|
||
| ## Configure jobs | ||
|
|
||
|
|
||
| <!-- | ||
| Each H2 step should start with a verb/action word. | ||
| --> | ||
|
|
||
|
|
||
| ## Next steps | ||
|
|
||
| - [Learn more about the Scheduler control plane service]({{< ref "concepts/dapr-services/scheduler.md" >}}) | ||
| - [Jobs API reference]({{< ref jobs_api.md >}}) |
62 changes: 62 additions & 0 deletions
62
daprdocs/content/en/developing-applications/building-blocks/jobs/jobs-overview.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| type: docs | ||
| title: "Jobs overview" | ||
| linkTitle: "Overview" | ||
| weight: 1000 | ||
| description: "Overview of the jobs API building block" | ||
| --- | ||
|
|
||
| Many applications require job scheduling, the need to take an action in the future. The jobs API is an orchestrator for scheduling these jobs in the future, either at a specific time or a specific interval. | ||
| Some typically example scenarios include; | ||
| - **Automated Database Backups**: Ensure a database is backed up daily to prevent data loss. Schedule a backup script to run every night at 2 AM, which will create a backup of the database and store it in a secure location. | ||
| - **Regular Data Processing and ETL (Extract, Transform, Load)**: Process and transform raw data from various sources and load it into a data warehouse. Schedule ETL jobs to run at specific times (for example: hourly, daily) to fetch new data, process it, and update the data warehouse with the latest information. | ||
| **Email Notifications and Reports**: Receive daily sales reports and weekly performance summaries via email. Schedule a job that generates the required reports and sends them via email at 6 AM every day for daily reports and 8 AM every Monday for weekly summaries. | ||
| **Maintenance Tasks and System Updates**: Perform regular maintenance tasks such as clearing temporary files, updating software, and checking system health. Schedule various maintenance scripts to run at off-peak hours, such as weekends or late nights, to minimize disruption to users. | ||
| **Batch Processing for Financial Transactions**: Processes a large number of transactions that need to be batched and settled at the end of each business day. Schedule batch processing jobs to run at 5 PM every business day, aggregating the day’s transactions and performing necessary settlements and reconciliations. | ||
| Using the jobs API in these scenarios ensures that tasks are performed consistently and reliably without manual intervention, improving efficiency and reducing the risk of errors. The jobs API helps you with scheduling jobs, and internally it is also used by Dapr to schedule actor reminders. | ||
|
|
||
| ### Delayed pub/sub | ||
|
|
||
| Use jobs to delay your pub/sub messaging. You can publish a message in a future specific time -- for example, a week from today, or a specific UTC date/time. | ||
|
|
||
| ### Scheduled service invocation | ||
|
|
||
| Use jobs with [service invocation]({{< ref service-invocation-overview.md >}}) to schedules method calls between applications. | ||
|
|
||
|
|
||
|
|
||
| Jobs consist of: | ||
| - The jobs API building block | ||
| - [The Scheduler control plane service]({{< ref "concepts/dapr-services/scheduler.md" >}}) | ||
|
|
||
| <img src="/images/scheduler/scheduler-architecture.png" alt="Diagram showing the Scheduler control plane service and the jobs API"> | ||
|
|
||
| ## How it works | ||
|
|
||
| The jobs API is a job scheduler, not the executor which runs the job. The design guarantees *at least once* job execution with a bias towards durability and horizontal scaling over precision. This means: | ||
| - **Guaranteed:** A job is never invoked *before* the schedule time is due. | ||
| - **Not guaranteed:** A ceiling time on when the job is invoked *after* the due time is reached. | ||
|
|
||
| All job details and user-associated data for scheduled jobs are stored in an embedded Etcd database in the Scheduler service. | ||
|
|
||
| ## Features | ||
|
|
||
| The jobs API provides several features to make it easy for you to schedule jobs. | ||
|
|
||
| ### Schedule jobs across multiple replicas | ||
|
|
||
| The Scheduler service enables the scheduling of jobs to scale across multiple replicas, while guaranteeing that a job is only triggered by 1 scheduler service instance. | ||
|
|
||
| ### Actor reminders | ||
|
|
||
| Actors have actor reminders, but present some limitations involving scalability using the Placement service implementation. You can make reminders more scalable by using [`SchedulerReminders`]({{< ref support-preview-features.md >}}). This is set in the configuration for your actor application. | ||
|
|
||
| ## Try out the jobs API | ||
|
|
||
| You can try out the jobs API in your application. After [Dapr is installed]({{< ref install-dapr-cli.md >}}), you can begin using the jobs API, starting with [the How-to: Schedule jobs guide]({{< ref howto-schedule-jobs.md >}}). | ||
|
|
||
| ## Next steps | ||
|
|
||
| - [Learn how to use the jobs API]({{< ref howto-schedule-jobs.md >}}) | ||
| - [Learn more about the Scheduler control plane service]({{< ref "concepts/dapr-services/scheduler.md" >}}) | ||
| - [Jobs API reference]({{< ref jobs_api.md >}}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| --- | ||
| type: docs | ||
| title: "Jobs API reference" | ||
| linkTitle: "Jobs API" | ||
| description: "Detailed documentation on the jobs API" | ||
| weight: 1300 | ||
| --- | ||
|
|
||
| {{% alert title="Note" color="primary" %}} | ||
| The jobs API is currently in alpha. | ||
| {{% /alert %}} | ||
|
|
||
| With the jobs API, you can schedule jobs and tasks in the future. | ||
|
|
||
| ## Schedule a job | ||
|
|
||
| Schedule a job with a name. | ||
|
|
||
| ``` | ||
| POST http://localhost:3500/v1.0-alpha1/jobs/<name> | ||
| ``` | ||
|
|
||
| ### URL parameters | ||
|
|
||
| Parameter | Description | ||
| --------- | ----------- | ||
| `name` | Name of the job you're scheduling | ||
| `data` | A string value and can be any related content. Content is returned when the reminder expires. For example, this may be useful for returning a URL or anything related to the content. | ||
| `dueTime` | Specifies the time after which this job is invoked. Its format should be [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) | ||
|
|
||
| ### Request body | ||
|
|
||
| ```json | ||
| { | ||
| "job": { | ||
| "data": { | ||
| "@type": "type.googleapis.com/google.type.Expr", | ||
| "expression": "<expression>" | ||
| }, | ||
| "dueTime": "30s" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### HTTP response codes | ||
|
|
||
| Code | Description | ||
| ---- | ----------- | ||
| `202` | Accepted | ||
| `400` | Request was malformed | ||
| `500` | Request formatted correctly, error in dapr code or Scheduler control plane service | ||
|
|
||
| ### Response content | ||
|
|
||
| The following example curl command creates a job, naming the job `jobforjabba` and specifying the `dueTime` and the `data`. | ||
|
|
||
| ```bash | ||
| $ curl -X POST \ | ||
| http://localhost:3500/v1.0-alpha1/jobs/jobforjabba \ | ||
| -H "Content-Type: application/json" | ||
| -d '{ | ||
| "job": { | ||
| "data": { | ||
| "HanSolo": "Running spice" | ||
| }, | ||
| "dueTime": "30s" | ||
| } | ||
| }' | ||
| ``` | ||
|
|
||
|
|
||
| ## Get job data | ||
|
|
||
| Get a job from its name. | ||
|
|
||
| ``` | ||
| GET http://localhost:3500/v1.0-alpha1/jobs/<name> | ||
| ``` | ||
|
|
||
| ### URL parameters | ||
|
|
||
| Parameter | Description | ||
| --------- | ----------- | ||
| `name` | Name of the scheduled job you're retrieving | ||
|
|
||
| ### HTTP response codes | ||
|
|
||
| Code | Description | ||
| ---- | ----------- | ||
| `202` | Accepted | ||
| `400` | Request was malformed | ||
| `500` | Request formatted correctly, error in dapr code or Scheduler control plane service | ||
|
|
||
| ### Response content | ||
|
|
||
| After running the following example curl command, the returned response is JSON containing the `name` of the job, the `dueTime`, and the `data`. | ||
|
|
||
| ```bash | ||
| $ curl -X GET http://localhost:3500/v1.0-alpha1/jobs/jobforjabba -H "Content-Type: application/json" | ||
| ``` | ||
|
|
||
| ```json | ||
| { | ||
| "name":"test1", | ||
| "dueTime":"30s", | ||
| "data": { | ||
| "HanSolo": "Running spice" | ||
| } | ||
| } | ||
| ``` | ||
| ## Delete a job | ||
|
|
||
| Delete a named job. | ||
|
|
||
| ``` | ||
| DELETE http://localhost:3500/v1.0-alpha1/jobs/<name> | ||
| ``` | ||
|
|
||
| ### URL parameters | ||
|
|
||
| Parameter | Description | ||
| --------- | ----------- | ||
| `name` | Name of the job you're deleting | ||
|
|
||
| ### HTTP response codes | ||
|
|
||
| Code | Description | ||
| ---- | ----------- | ||
| `202` | Accepted | ||
| `400` | Request was malformed | ||
| `500` | Request formatted correctly, error in dapr code or Scheduler control plane service | ||
|
|
||
| ### Response content | ||
|
|
||
| In the following example curl command, the job named `test1` with app-id `sub` will be deleted | ||
|
|
||
| ```bash | ||
| $ curl -X DELETE http://localhost:3500/v1.0-alpha1/jobs/jobforjabba -H "Content-Type: application/json" | ||
| ``` | ||
|
|
||
|
|
||
| ## Next steps | ||
|
|
||
| [Jobs API overview]({{< ref jobs-overview.md >}}) |
Oops, something went wrong.