Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ver 0.1 : improve performance with buffer dispatching #2

Open
wants to merge 32 commits into
base: ver-0.1
Choose a base branch
from
Open

Ver 0.1 : improve performance with buffer dispatching #2

wants to merge 32 commits into from

Conversation

smarchint
Copy link

  • add buffer for batch dispatching into bigQuery
  • add thread pool for dispatching
  • add timer based and buffersize based dispatching
  • handle shutdown events for clearing the buffer before the termination
  • rabbitmq settings changes : manual ack with some Qos value.
  • fix : prevent rabbitmq java client buffer flooding due to slow worker.

@smarchint
Buffer multiple raw insertion requests [DRAFT]
079ab97
@smarchint
resolve compilation errors
1d9b60c
@smarchint
remove unnecessary code
0751e94
@smarchint
handle SIGTINT/TERM and refactor
e0063a4
@smarchint
fix timer dispatch
a9bf88d
@smarchint
minor changes [remove comments]
4b35f61
@smarchint
minor changes
8ac7fb2
@smarchint
fix nullpointer exception at ctrl+c
03242a9
@smarchint
refactor
ec2811d
@smarchint
comment debug code
e69597e
@smarchint
remove test code
d0f82c1
@smarchint
fix : close rabbitmq channel on shutdown
a350d69
@smarchint
remove unwanted code
55985aa
@smarchint
remove comments
5ff0680
@smarchint
fix : reminent messages at client termination
39df85b
@smarchint
decouplr msg recv and processor
f35d456
@smarchint
add bounded queue
2ccffe6
@smarchint
remove comments
22df065
@smarchint
minor change
6a9dc26
@smarchint
add try catch in dispatch event
6091fc1
@smarchint
catch exceptions
9535152
@smarchint
minor change
dca185e
@smarchint
fix batch insertions with partial failures and refactor
d0f4046
@smarchint
add info logging on shutdown
5d64ab3
@smarchint
minor changes
2411773
@smarchint
add eventbus for scaling bigquery dispatcher and make rabbit consumer…
3359436 
… lightweight
@smarchint
refactor and remove unwanted code
48e845d
@smarchint
add sub-optimal prefetch count with manual ack [multiple] [DRAFT]
6816ead
@smarchint
remove comments and add debug mode for th app
bd615cf
@smarchint
remove comment
6e1b036
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@smarchint