Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance PollingDao to use a thread-per-channel #128

Open
cer opened this issue Aug 3, 2022 · 3 comments
Open

Enhance PollingDao to use a thread-per-channel #128

cer opened this issue Aug 3, 2022 · 3 comments

Comments

@cer
Copy link
Contributor

cer commented Aug 3, 2022

PollingDao currently use a single thread to poll the database.

Enhance PollingDao to use a thread-per-channel, which improves throughput while preserving ordering within a channel.
cer added a commit that referenced this issue Aug 3, 2022
@cer
#128 Enhance PollingDao to use a thread-per-channel
edf8e47 
Also, removed unused bean dependencies in order to simplify testing
@cer
Copy link
Contributor Author

cer commented Aug 3, 2022

Two ways to configure:

Simple, Single pipeline configuration:

eventuatelocal.cdc.polling.parallel.channels=channelA,channelB

Multi-pipeline configuration:

eventuate.cdc.reader.mysqlreader.type: polling
...
eventuate.cdc.reader.mysqlreader.pollingparallelchannelnames: channelA,channelB

@cer cer mentioned this issue Aug 3, 2022
Open
cer added a commit that referenced this issue Aug 3, 2022
@cer
#128 Enhance PollingDao to use a thread-per-channel
ff16fab 
Fixed NPE when no polling channels configured
cer added a commit that referenced this issue Aug 4, 2022
@cer
#128 Enhance PollingDao to use a thread-per-channel - extracted prope…
8fbccc4 
…rty mapping from UnifiedCdcConfigurator into PipelineConfigPropertiesProvider in order to improve testability
@cer
Copy link
Contributor Author

cer commented Aug 4, 2022 •
edited
Loading

If you specify pollingparallelchannels: channelA,channelB the PollingDao (see this line) creates the following three threads:

  1. Polls for messages sent to channelA
  2. Polls for messages sent to channelB
  3. Polls for messages sent to other channels

@cer
Copy link
Contributor Author

cer commented Aug 4, 2022

Note: since the SQL SELECTs now contain either AND destination = 'channelA' or AND destination NOT ('channelA', 'channelB'), it might worthwhile investigating query performance and determining whether a composite index (processed, channel, ID is worthwhile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cer