Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: non-blocking tasks #128

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

Sobes76rus
Copy link
Contributor

@Sobes76rus Sobes76rus commented May 11, 2023

non-blocking sleep through a temporary increase in the number of running tasks on demand of the user.
here is some use cases

  1. some task A is waiting for result of another task B. User can request a temporary increase in the number of tasks, so this task A does not block the execution of task B because of max_async_tasks. Example in tests with max_async_tasks = 1
  2. i have a real case scenario where very long running tasks sleeps for a very long time and there is a need to keep their scope, so i increase the max number of running tasks while they asleep

@codecov-commenter
Copy link

codecov-commenter commented May 11, 2023

Codecov Report

Merging #128 (a460da4) into develop (e66f3aa) will increase coverage by 4.98%.
The diff coverage is 82.40%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@             Coverage Diff             @@
##           develop     #128      +/-   ##
===========================================
+ Coverage    67.62%   72.60%   +4.98%     
===========================================
  Files           37       43       +6     
  Lines          942     1387     +445     
===========================================
+ Hits           637     1007     +370     
- Misses         305      380      +75     
Impacted Files Coverage Δ
taskiq/abc/middleware.py 100.00% <ø> (ø)
taskiq/abc/result_backend.py 100.00% <ø> (ø)
taskiq/cli/scheduler/run.py 0.00% <ø> (ø)
taskiq/cli/watcher.py 0.00% <ø> (ø)
taskiq/cli/worker/log_collector.py 100.00% <ø> (ø)
taskiq/cli/worker/process_manager.py 0.00% <0.00%> (ø)
taskiq/cli/worker/run.py 0.00% <0.00%> (ø)
taskiq/events.py 100.00% <ø> (ø)
taskiq/receiver/params_parser.py 96.87% <ø> (ø)
taskiq/result_backends/dummy.py 100.00% <ø> (ø)
... and 24 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Sobes76rus Sobes76rus changed the title feat: task's idle execution feat: increase number of running tasks while current task asleep May 11, 2023
@Sobes76rus Sobes76rus changed the title feat: increase number of running tasks while current task asleep feat: increase number of running tasks by user demands while current task asleep May 11, 2023
@Sobes76rus Sobes76rus changed the title feat: increase number of running tasks by user demands while current task asleep feat: increase number of running tasks by user's demands while current task asleep May 11, 2023
@Sobes76rus Sobes76rus changed the title feat: increase number of running tasks by user's demands while current task asleep feat: increasing number of running tasks on user's demands while current task is asleep May 12, 2023
@Sobes76rus Sobes76rus changed the title feat: increasing number of running tasks on user's demands while current task is asleep feat: non-blocking sleep May 22, 2023
@lembregtse
Copy link

@s3rius are there plans to have this merged / or implemented differently? We have a use-case as well that triggers this that we noticed since we have now implemented the max_async_tasks option.

We will "workarround" by separating them into two different workers for now.

@s3rius
Copy link
Member

s3rius commented Dec 6, 2023

There are plans to implement it differently. There were some attempts.

The main idea of another approach is to not give just a sleep function, but an ability to decrease counter of running tasks by the time of executing a function.

I've created an issue for that problem here: #258.

The problem with this approach is that in most cases we don't use sleep directly, but awaiting something.

@Sobes76rus
Copy link
Contributor Author

Sobes76rus commented Jul 25, 2024

Hello there @s3rius ! I changed approach to dealing with idle tasks
Now u can do smth like this

async def task(idle: TaskIdler = Depends()):
    async with idle():
        await smth()

Pls take a look

ps: there are some endless action going on. pls stop it

@Sobes76rus Sobes76rus changed the title feat: non-blocking sleep feat: non-blocking tasks Jul 25, 2024
@Sobes76rus
Copy link
Contributor Author

Hello, @s3rius. Pls take a look at the new implementation

@Sobes76rus
Copy link
Contributor Author

@s3rius A friendly reminder to check this PR :D

@Sobes76rus
Copy link
Contributor Author

Hello there @s3rius! Pls give it a try =)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants