Skip to content

Commit

Permalink
Merge branch 'release/0.9.3'
Browse files Browse the repository at this point in the history
  • Loading branch information
s3rius committed Oct 11, 2023
2 parents e0b4b05 + 8cbbc6d commit 3e9ff9d
Show file tree
Hide file tree
Showing 20 changed files with 371 additions and 65 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.9"
python-version: "3.11"
- name: Install deps
run: poetry install
- name: Release package
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.9"
python-version: "3.11"
cache: "poetry"
- name: Install deps
run: poetry install
Expand Down
73 changes: 70 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,16 @@
<hr/>
</div>

Documentation: https://taskiq-python.github.io/

## What is taskiq?

Taskiq is an asynchronous distributed task queue for python.
This project takes inspiration from big projects such as [Celery](https://docs.celeryq.dev) and [Dramatiq](https://dramatiq.io/).
But taskiq can send and run both the sync and async functions.
Also, we use [PEP-612](https://peps.python.org/pep-0612/) to provide the best autosuggestions possible. But since it's a new PEP, I encourage you to use taskiq with VS code because Pylance understands all types correctly.
But taskiq can send and run both the sync and async functions, has
integration with popular async frameworks, such as [FastAPI](https://fastapi.tiangolo.com/) and [AioHTTP](https://docs.aiohttp.org/en/stable/).

Also, we use [PEP-612](https://peps.python.org/pep-0612/) to provide the best autosuggestions possible. All code is type-hinted.

# Installation

Expand All @@ -25,8 +31,69 @@ Or it can be installed directly from git:
pip install git+https://github.com/taskiq-python/taskiq
```

You can read more about how to use it in our docs: https://taskiq-python.github.io/.
# Usage

At first you need to create a broker. Broker is an object that can communicate to workers using distributed queues.

We have differet brokers for different queue backends. For example, we have a broker for [NATS](https://github.com/taskiq-python/taskiq-nats), [Redis](https://github.com/taskiq-python/taskiq-redis), [RabbitMQ](https://github.com/taskiq-python/taskiq-aio-pika), [Kafka](https://github.com/taskiq-python/taskiq-aio-kafka) and even more. Choose the one that fits you and create an instance.

```python
from taskiq_nats import JetStreamBroker

broker = JetStreamBroker("nats://localhost:4222", queue="my_queue")
```

Declaring tasks is as easy as declaring a function. Just add a decorator to your function and you are ready to go.

```python
import asyncio

from taskiq_nats import JetStreamBroker

broker = JetStreamBroker("nats://localhost:4222", queue="my_queue2")


@broker.task
async def my_task(a: int, b: int) -> None:
print("AB", a + b)


async def main():
await broker.startup()

await my_task.kiq(1, 2)

await broker.shutdown()


if __name__ == "__main__":
asyncio.run(main())


```

The message is going to be sent to the broker and then to the worker. The worker will execute the function. To start worker processes, just run the following command:

```bash
taskiq worker path.to.the.module:broker
```

Where `path.to.the.module` is the path to the module where the broker is defined and `broker` is the name of the broker variable.

If you have tasks in different modules, you can ask taskiq to automatically import them by passing the `--fs-discover` flag:

```bash
taskiq worker path.to.the.module:broker --fs-discover
```

It will import all modules called `tasks.py` in the current directory and all subdirectories.

Also, we support hot reload for workers. To enable it, just pass the `--reload` flag. It will reload the worker when the code changes (To use it, install taskiq with reload extra. E.g `pip install taskiq[reload]`).


Also, we have cool integrations with popular async frameworks. For example, we have an integration with [FastAPI](https://taskiq-python.github.io/framework_integrations/taskiq-with-fastapi.html) or [AioHTTP](https://taskiq-python.github.io/framework_integrations/taskiq-with-aiohttp.html). You can use it to reuse dependencies from your web app in your tasks.

Read about all features in our documentation: https://taskiq-python.github.io/

# Local development

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
6 changes: 3 additions & 3 deletions docs/guide/dynamic-brokers.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Taskiq allows you to set up broker instances throughout your application and reg

To define tasks and assign them to a broker, use `register_task` method.

@[code python](../examples/dynamics/broker.py)
@[code python](../examples/dynamics/dyn_broker.py)

In this example, the task is defined using a lambda within the `main` function. As the lambda is not visible outside of the `main` function scope, the task is not executable by `taskiq worker` command.

Expand All @@ -24,10 +24,10 @@ To overcome this issue, you can:

Here's an example of a dynamic worker task creation:

@[code python](../examples/dynamics/receiver.py)
@[code python](../examples/dynamics/dyn_receiver.py)

In this example, a named dynamic lambda task is created and registered in a broker, similar to the previous example. The difference is the creation of a new receiver coroutine for the worker task. It will listen to the new messages and execute them. The worker task will be executed in the current event loop. After exiting the scope, the worker task will get cancelled. For illustration purposes it is cancelled explicitly.

It's possible to run a scheduler in the current event loop as well:

@[code python](../examples/dynamics/scheduler.py)
@[code python](../examples/dynamics/dyn_scheduler.py)
102 changes: 51 additions & 51 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "taskiq"
version = "0.9.2"
version = "0.9.3"
description = "Distributed task queue with full async support"
authors = ["Pavel Kirilin <[email protected]>"]
maintainers = ["Pavel Kirilin <[email protected]>"]
Expand Down Expand Up @@ -61,6 +61,7 @@ freezegun = "^1.2.2"
pytest-mock = "^3.11.1"
tzlocal = "^5.0.1"
types-tzlocal = "^5.0.1.1"
types-pytz = "^2023.3.1.1"

[tool.poetry.extras]
zmq = ["pyzmq"]
Expand Down
Loading

0 comments on commit 3e9ff9d

Please sign in to comment.