Skip to content
This repository has been archived by the owner on Jan 29, 2022. It is now read-only.

Latest commit

 

History

History
30 lines (26 loc) · 1.22 KB

TODO.md

File metadata and controls

30 lines (26 loc) · 1.22 KB

Roadmap items

  • UI page answering "Why isn't this task instance running?"
  • Attempt removing DagBag caching for the web server
  • Distributed scheduler (supervisors)
    • Get the supervisors to run sensors (as opposed to each sensor taking a slot)
    • Improve DagBag differential refresh
    • Pickle all the THINGS! supervisors maintains fresh, versionned pickles in the database as they monitor for change
  • Pre-prod running off of master
  • Containment / YarnExecutor / Docker?
  • Get s3 logs
  • Test and migrate to use beeline instead of the Hive CLI
  • Run Hive / Hadoop / HDFS tests in Travis-CI

UI

  • Backfill form
  • Better task filtering int duration and landing time charts (operator toggle, task regex, uncheck all button)
  • Add templating to adhoc queries

Backend

  • Add a run_only_latest flag to BaseOperator, runs only most recent task instance where deps are met
  • Raise errors when setting dependencies on task in foreign DAGs
  • Add an is_test flag to the run context

Wishlist

  • Pause flag at the task level
  • Increase unit test coverage
  • Stats logging interface with support for stats and sqlalchemy to collect detailed information from the scheduler and dag processing times

Other

  • deprecate TimeSensor