Skip to content

Commit

Permalink
v0.199.0 - Enabled persistency for Tasks & Flows domains (#807)
Browse files Browse the repository at this point in the history
Tasks:
 - fully transactional task event store, scheduler
 - got rid of in-memory task queue, replaced on task event store queries
 - split out part that runs particular plans from `TaskExecutor ` into `TaskLogicalPlanRunner` component
 - support tasks metadata
 - startup recovery of running tasks (back to queued)
 - task service dependencies registration simplified
 - extended test coverage

Flows:
 - Postgres/SQlite implementations for flow event store
 - split out `FlowQueryService` component from `FlowExecutor`
 - extracted enqueue and abort helpers, and offloaded a lot of code from `FlowExecutor`
 - querying active flow configs directly instead of storing cached state in flow service
 - storing task->flow association as "task metadata"
 - pending flows information replaced with flow event store queries
 - isolated flow time wheel into a separate singleton service
 - recovering flow time wheel properly after server restart
 - extended test coverage

Migrations cleanup and table simplifications (Breaking change of database schema!)

Introducer pre_run() phase in flow executor, task executor & outbox processor to avoid startup races.
All repository stream-returning operations should not be async.
Unplugged sync service from transactions.

Minor dependency updates.
Security exception for "proc-macro-error"
  • Loading branch information
zaychenko-sergei authored Sep 6, 2024
1 parent c829296 commit 43de6c2
Show file tree
Hide file tree
Showing 268 changed files with 11,828 additions and 5,491 deletions.
14 changes: 12 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
<!--- - Changed -->
<!--- - Fixed -->

## Unreleased
### Changed
## [0.199.0] - 2024-09-06
### Added
- Persistency has been enabled for Task and Flow domains.
Both `TaskExecutor` and `FlowExecutor` now fully support transactional processing mode,
and save state in Postgres or Sqlite database.
- Tasks now support attaching metadata properties. Storing task->flow association as this type of metadata.
- Flows and Tasks now properly recover the unfinished requests after server restart
### Changed
- Simplified database schema for flow configurations and minimized number of migrations
(breaking change of the database schema)
- Introduced `pre_run()` phase in flow executor, task executor & outbox processor to avoid startup races
- Explicit in-memory task queue has been eliminated and replaced with event store queries
- Get Data Panel: use SmTP for pull & push links
- GQL api method `setConfigCompaction` allows to set `metadataOnly` configuration for both root and derived datasets
- GQL api `triggerFlow` allows to trigger `HARD_COMPACTION` flow in `metadataOnly` mode for both root and derived datasets
Expand Down
Loading

0 comments on commit 43de6c2

Please sign in to comment.