Skip to content

Commit

Permalink
utils Async critical fix for variant behavior on pypy3, vbump
Browse files Browse the repository at this point in the history
to make a very long debug story much shorter: it turns out that pypy3
and cpython have remarkably and dangerously different behavior for
closures in comprehensions

fixing the bad closure behavior also seems to have resolved the
wild and crazy hitching and slowdowns seen toward the end of long
async runs, probably because i was also changing

these have obvious behavior that is consistent for both
pypy3 and cpython

help = (lambda : OH_NO for OH_NO in range(10))
me = list(help)
sigh = [m() for m in me]

help = (lambda : OH_NO for OH_NO in range(10))
sigh = [h() for h in help]

however consider

from time import sleep
import asyncio
from concurrent.futures import ThreadPoolExecutor
def OH_NO_NO_NO():
    ex = ThreadPoolExecutor()
    help = (lambda : (OH_NO, sleep((OH_NO + 1) / 10)) for OH_NO in range(10))
    #help = (lambda : OH_NO for OH_NO in range(10))
    async def ohno(future_):
        loop = asyncio.get_event_loop()
        futures = []
        for f in help:
            futures.append(loop.run_in_executor(ex, f))
        out = []
        for f in futures:
            out.append(await f)
            sleep(0.1)
        future_.set_result(out)

    future = asyncio.Future()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(ohno(future))
    return future.result()

cpython also fails in this case, but if you look at pypy3 the behavior
is significantly more variable, in the context of Async there is some
additional factor which has somehow ensured that cpython chunk didn't
get rebound ... or something like that ... I haven't completed the
investigation with a full explaination, but this is at least part of
the story
  • Loading branch information
tgbugs committed Jan 11, 2023
1 parent 40fdbbb commit 5a10dbb
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 5 deletions.
2 changes: 1 addition & 1 deletion pyontutils/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.1.31'
__version__ = '0.1.32'
10 changes: 6 additions & 4 deletions pyontutils/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -461,10 +461,12 @@ def inner(generator):
time_est = len(funclist) / rate
chunks = chunk_list(funclist, size)
lc = len(chunks)
print(f'Time estimate: {time_est} rate: {rate}Hz '
f'func: {funclist[0]} args: {len(funclist)} '
f'chunks: {lc} size: {size}')
generator = (lambda:list(limited_gen(chunk, smooth_offset=(i % lc)/lc, time_est=time_est, debug=debug, thread=i)) # this was the slowdown culpret
log.info(f'Time estimate: {time_est} rate: {rate}Hz '
f'func: {funclist[0]} args: {len(funclist)} '
f'chunks: {lc} size: {size}')
generator = (lambda _c=chunk, _i=i: list(limited_gen(
_c, smooth_offset=(_i % lc)/lc, time_est=time_est,
debug=debug, thread=_i))
for i, chunk in enumerate(sorted(chunks, key=len, reverse=True)))

async def future_loop(future_):
Expand Down

0 comments on commit 5a10dbb

Please sign in to comment.