Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify internal communication / flows, starting with baseoutput #17

Open
jiping-s opened this issue Sep 30, 2021 · 0 comments
Open

Simplify internal communication / flows, starting with baseoutput #17

jiping-s opened this issue Sep 30, 2021 · 0 comments
Labels
enhancement New feature or request research Research for bug or feature

Comments

@jiping-s
Copy link
Collaborator

jiping-s commented Sep 30, 2021

Currently all internal communications and code flows are built around primitive go channels and signals, as a result the complexity has exploded especially in baseoutput related to shutdown mechanism:

  • There are async sender and receiver to enable pipelining / reduce latency
  • There are soft stop, hard stop (abort) to reconnect, and hard stop to shut down.
  • Network reading and writing can fail independently and the other side may continue to work (unintended)
  • All pending chunks whether in queue or being processed in functions need to be collected for recovery
  • The code is difficult to test. Currently the only tests involving real output are in run package and they can't cover any exceptional situations.

Instead of a free form that each piece of code decides how to use channel and signals, there could be some unified framework or library that help making:

  • The top-level flow should be visualized, in a declarative form. It needs to cover all exceptional situations because dealing with them is the main cause of complexity.
  • The components (e.g. acknowledger or sub unit) should have clear input, output, responsibilities and no implicit side-effects

The internal communication in baseoutput is not performance critical as it works on compressed log chunks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request research Research for bug or feature
Projects
None yet
Development

No branches or pull requests

1 participant