A quick script to check if a job took longer than x time to run and update python in precommit

[EmanElsaban]

This script outputs the jobs that takes longer than x time to run. We can specify the time using --time. It's an easy way to get the jobs than to look at the Tron UI

[nemacysts]

I think we might want this to still use [--target-version, py36] so that it doesn't try to format anything in an incompatible way

[EmanElsaban]

its failed the test below cause:

ERROR: InvocationError for command '/home/runner/work/Tron/Tron/.tox/py36/bin/pre-commit run --all-files' (exited with code 1)

but when I try to run this locally I don't have py36 on this branch, its .tox/py38/...

[EmanElsaban]

then when I run make test after make clean:

(py38) emanelsabban@dev208-uswest1adevc:~/pg/Tron$ make test
tox -e py36
py36 create: /nail/home/emanelsabban/pg/Tron/.tox/py36
ERROR: InterpreterNotFound: python3.6
________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________
ERROR:  py36: InterpreterNotFound: python3.6
make: *** [Makefile:64: test] Error 1

[nemacysts]

oh, that's unfortunate :/

I'm just a tad worried about breaking the py36 build and/or somehow allowing py36+ code to sneak in and break at runtime (on py36)

[nemacysts]

the yelpy naming convention for this is usually parse_args :)

probably a good idea to have new code by typed as well, so we probably want this to be something like:

Suggested change

	def parse_cli():
	def parse_args() -> argparse.Namespace:

[nemacysts]

and depending on how easy it is to trace the tron code to get the type of some of these things, it might be worth using reveal_type (see https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#when-you-re-puzzled-or-when-things-are-complicated) to have mypy tell you what it thinks the types are

[nemacysts]

Suggested change

	def main():
	def main() -> Optional[int]:

would match what sys.exit expects :)

[nemacysts]

non-blocking suggestion: could we make this a pure function by having it return a list of affected runs?

(and then do something like results.extend(check_job_time(...))?

[nemacysts]

super-nit: should this be something like get_jobs_exceeding_runtime? usually check_* functions are intended to be used as monitoring checks and will return non-zero if things are failing checks (versus the current behavior here, which is just printing)

[nemacysts]

nit: we repeat this - might be worth extracting this to a named constant

[nemacysts]

do we need to do this check in both places? looks like this check is also being done by the function that calls this

[EmanElsaban]

I don't think so, Idk why I initially thought the state might change so I added it in both places

[nemacysts]

nit:

Suggested change

	if duration_seconds and duration_seconds > job_expected_runtime:
	return True
	return duration_seconds and duration_seconds > job_expected_runtime:

[nemacysts]

just curious: are we sorting so that the output is ordered later on? should we sort at printing time so that we only need to sort once if so?

[nemacysts]

should this be if not result? we default result to [] - which is not None

[nemacysts]

i've got some minor comments re: the types - but if i'm blocking you feel free to ignore them as they're not particularly serious and we can always come back and improve them :)

[nemacysts]

Suggested change

	def check_if_time_exceeded(job_runs, job_expected_runtime) -> list:
	def check_if_time_exceeded(job_runs: List[Dict[str, Any]], job_expected_runtime: int) -> List[str]:

i'm sort of guessing about the job_runs typing here - that said, if we're just using this script for the current project it's probably fine to be a little lax with the types and just do the easy ones (e.g.,

Suggested change

	def check_if_time_exceeded(job_runs, job_expected_runtime) -> list:
	def check_if_time_exceeded(job_runs: List, job_expected_runtime: int) -> List[str]:

[nemacysts]

Suggested change

	def is_job_run_exceeding_expected_runtime(job_run, job_expected_runtime) -> bool:
	def is_job_run_exceeding_expected_runtime(job_run: Dict[str, Any], job_expected_runtime: int) -> bool:

[nemacysts]

Suggested change

	def check_job_time(job, time_limit) -> list:
	def check_job_time(job: Dict[str, Any], time_limit: int) -> List[str]:

[nemacysts]

@EmanElsaban is this still something we're interested in merging?