Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deployment issue #818

Open
wants to merge 29 commits into
base: client-403-issue
Choose a base branch
from
Open

Deployment issue #818

wants to merge 29 commits into from

Conversation

ReddyAnand-BV
Copy link
Contributor

Github Issue

1234

What Are We Doing Here?

Here is where you should describe the problem you are solving as it relates to the Github Issue referred above, adding any fine
details on the solution that might otherwise not be recognizable for someone
unfamiliar with the changes. Add some pictures if it helps.

How to Test and Verify

  1. Check out this PR
  2. Run Command X, Click Button Y
  3. Profit

Risk

Level

Low, Medium, or High. Give an indication of what you think is the level of change introduced by this PR. High means a massive change to a core functionality.
Low means a really minor change that shouldn't have any regression effect.

Required Testing

Smoke, Regression, or Manual. (All changes except documentation need smoke
testing at a minimum).

Risk Summary

Add one or a few complete sentences about the possible risks or concerns for
this change.

Code Review Checklist

  • Tests are included. If not, make sure you leave us a line or two for the reason.

  • Pulled down the PR and performed verification of at least being able to
    build and run.

  • Well documented, including updates to any necessary markdown files. When
    we inevitably come back to this code it will only take hours to figure out, not
    days.

  • Consistent/Clear/Thoughtful? We are better with this code. We also aren't
    a victim of rampaging consistency, and should be using this course of action.
    We don't have coding standards out yet for this project, so please make sure to address any feedback regarding STYLE so the codebase remains consistent.

  • PR has a valid summary, and a good description.

@ReddyAnand-BV ReddyAnand-BV requested a review from a team as a code owner February 29, 2024 09:36
* updatedcltwo_final

* updatedcltwo_final_consistency
* updatedcltwo_final

* updatedcltwo_final_consistency

* snapshot_update
* Added document_id as env variable

* Test changes

* branch admin -prepare release emodb-6.5.134

* branch admin -prepare for next development iteration

---------

Co-authored-by: abhyudaya-mohanty_bveng <[email protected]>
Co-authored-by: jenkins <[email protected]>
…quisite (#829)

* feat: integrate sqs to blob service

* Create an sqs service to send messages to sqs

* Send all the incoming PUT and DELETE request to sqs

* feat: inlcude functionality to store encoded blob in sqs

* Convert the incoming blob to bytearray, encode it and send it to sqs
* Handle the incoming api request of PUT blob object
* Send the attributes and object to sqs

* feat: restructure the queue intialization following better architecture

* Create interface for queue, so can include multiple implementations

* Follow lose coupling

* Use a factory method for sqs initialization

* PD-239427 fix: fix logging and simplified factory

* feat: include params required for iss, send to sqs and update queue name

* Include queue url param to be sent to sqs

* Update the queue name

* Update the factory as per the required changes

* update: update tags for deployment

* branch admin -prepare release emodb-6.5.135

* branch admin -prepare for next development iteration

* update: add fifo queue for blobstore migration and tag changes

* branch admin -prepare release emodb-6.5.144

* branch admin -prepare for next development iteration

* fix: remove sending of bytecode to sqs

* For large files , bytecode is too huge to be sent to sqs

* Send request url instead of bytecode

* feat: add custom exceptions

* handle exceptions in a robust way

* add digestbytes to sqs

* chore: add tags skip sonatype for deployment

* branch admin -prepare release emodb-6.5.159

* branch admin -prepare for next development iteration

* fix: add back sonatype dependency

* branch admin -prepare release emodb-6.5.160

* branch admin -prepare for next development iteration

* Removed bytearray in sqs put

* Update MessagingService.java for removing byte

* Update SQSService.java

* branch admin -prepare release emodb-6.5.161

* branch admin -prepare for next development iteration

* skipping nexus plugin for testing --parent.pom

* branch admin -prepare release emodb-6.5.162

* branch admin -prepare for next development iteration

* add metrics to Queue poll

* branch admin -prepare release emodb-6.5.163

* branch admin -prepare for next development iteration

* update pom version

* branch admin -prepare release emodb-6.5.165

* branch admin -prepare for next development iteration

* branch admin -prepare release emodb-6.5.166

* branch admin -prepare for next development iteration

* feat: add poll metrics for dedupq

* feat: add send count metrics

* fix: add metrics for sendBatch

* udate: update tags from 167 to 168

* fix: fix the compilation issue remove metricregistry from constructor

* branch admin -prepare release emodb-6.5.168

* branch admin -prepare for next development iteration

* chore: update tags from 168 to 169

* branch admin -prepare release emodb-6.5.169

* branch admin -prepare for next development iteration

* fix: fix the metric tag name issue

* branch admin -prepare release emodb-6.5.170

* branch admin -prepare for next development iteration

---------

Co-authored-by: jenkins <[email protected]>
Co-authored-by: anandujayan <[email protected]>
Co-authored-by: ReddyAnand-BV <[email protected]>
Co-authored-by: Nabajyoti Dash <[email protected]>
…function firing (#844)

* PD-257019 Stabilization emodb queue service changes (#831)

* PD-249429: integrated emodb with datastorage-media-service for getting the metadata from s3.

* PD-249429: refactored code.

* PD-249428: DELETE blob with blob id.

* PD-249428: Implemented upload blob from byte array.

* PD-249428: Implemented upload blob from byte array.

* PD-249428: Fixed failing ITs.

* PD-249428: Fixed failing ITs.

* PD-249428: Changed the snapshot version.

* PD-249428: Refactored code.

* PD-249428: changed the version to 172.

* PD-249428: commented tests.

* branch admin -prepare release emodb-6.5.172

* PD-249428: changed snapshot versions.

* PD-249428: changed snapshot versions.

* branch admin -prepare release emodb-6.5.173

* branch admin -prepare for next development iteration

* changed the BASE_URL

* PD-256742: fixed all bugs.

* PD-256742: changed snapshot version.

* PD-256742: changed snapshot version.

* branch admin -prepare release emodb-6.5.176

* branch admin -prepare for next development iteration

* for testing

* feat: integrate kafka to emodb

* Create kafka producer and admin service

* Overload the sendAll method to reroute messages to kafka

* Make the required changes in Clients like jersey2

* feat: include logic for separation  of queue and dedupq

* feat: include guava dependency injection for emodb

* fix: seperation of producer and admin service and proper injection

* fix: fix DI wiring issue

* feat: integrate triggering step function execution on creation of new topic

* feat: merge changes for stepfunction

* branch admin -prepare release emodb-6.5.175

* branch admin -prepare for next development iteration

* feat: add parameter store

* fix:add aws core dependency

* fix:remove credentialProvider

* fix: changes for ci deployment

* fix: pom changes for blob

* chore: update tags to 177 for deployment

* branch admin -prepare release emodb-6.5.177

* branch admin -prepare for next development iteration

* chore: update msk servers

* fix: include msk server url

* chore: snapshot update to 181

* branch admin -prepare release emodb-6.5.181

* branch admin -prepare for next development iteration

* feat: include working parameter store and stepfn changes

* add proper exception handling in parameter store and stepfn

* make abstractqueue service code modular and organized

* branch admin -prepare release emodb-6.5.182

* branch admin -prepare for next development iteration

* feat: fetch kafka configs from ssm and dependency injection for stepfn

* fetch aws configs from ssm parameter store

* introduce dependency injection for stepfunction service

* branch admin -prepare release emodb-6.5.183

* branch admin -prepare for next development iteration

* feat: feature flag implementation for queue migrations

* include the proper naming for each stepfn execution

* include all required contracts for stepfn

* include the feature flag logic, fetch params from parameter store and based on the allowed queues migrate

* fix: fix param for allowedQueues

* branch admin -prepare release emodb-6.5.184

* branch admin -prepare for next development iteration

* fix: remove blob changes

* chore: add comments and fix  start.sh

* fix: changed logic for feature flag
* add changed logic for feature flag implementation

* wrap the payload for stepfn around a executionInput parameter

* fix: fixed exception handling and code refactoring

* throw back exceptions after logging

* include create topic and checktopic in one function

* add comments for better description

---------

Co-authored-by: vikram-vikram_bveng <[email protected]>
Co-authored-by: jenkins <[email protected]>
Co-authored-by: nabajyotiDash-hub <[email protected]>

* update pom version

* branch admin -prepare release emodb-6.5.192

* branch admin -prepare for next development iteration

* modify producer properties (#834)

* PD-257019 Add universe fetching and bug fixes (#835)

* feat: add universe fetching and bug fixes

* fix: remove debug logs

* branch admin -prepare release emodb-6.5.193

* branch admin -prepare for next development iteration

* uncached_size for dedupq (#837)

* PD-257019 Producer send and logging changes (#838)

* fix: fix send logic and exception handling changes

* added exception handling

* changed producer send logic to be synchronous

* removed redundant logs

* uncached_size for dedupq (#837)

* chore: rename queuetype as per convention

* chore: fix naming issue

---------

Co-authored-by: nabajyotiDash-hub <[email protected]>

* branch admin -prepare release emodb-6.5.194

* branch admin -prepare for next development iteration

* branch admin -prepare release emodb-6.5.195

* branch admin -prepare for next development iteration

* branch admin -prepare release emodb-6.5.196

* branch admin -prepare for next development iteration

* chore: snapshot updates

* branch admin -prepare release emodb-6.5.194

* branch admin -prepare for next development iteration

* [PD-258160] added dd metrics for queue name and queue size (#836)

* branch admin -prepare release emodb-6.5.195

* alter pom version for bvengineering deployment

* branch admin -prepare release emodb-6.5.196

* update pom to 6.5.197-SNAPSHOT

* branch admin -prepare release emodb-6.5.197

* branch admin -prepare for next development iteration

* branch admin -prepare release emodb-6.5.198

* branch admin -prepare for next development iteration

* add cache

* add allowedQueues cache

* feat: producer future logic and caching changes

* chore: clear out unused code

* PD-257019 Producer send and logging changes (#838)

* fix: fix send logic and exception handling changes

* added exception handling

* changed producer send logic to be synchronous

* removed redundant logs

* uncached_size for dedupq (#837)

* chore: rename queuetype as per convention

* chore: fix naming issue

---------

Co-authored-by: nabajyotiDash-hub <[email protected]>

* fix: fix the config changes

* Revert "Merge branch 'master' into PD-257019-producer-async-fixes"

This reverts commit 93f5a76, reversing
changes made to 15ce863.

* chore: snapshot updates for deployement

* chore: clear up testing logs

* fix: remove default servers

* branch admin -prepare release emodb-6.5.202

* branch admin -prepare for next development iteration

* chore: caching logs refactoring

* branch admin -prepare release emodb-6.5.203

* branch admin -prepare for next development iteration

---------

Co-authored-by: vikram-vikram_bveng <[email protected]>
Co-authored-by: jenkins <[email protected]>
Co-authored-by: nabajyotiDash-hub <[email protected]>
Co-authored-by: nabajyotiDash-hub <[email protected]>
Co-authored-by: Anurag Dubey <[email protected]>
* feat: execution name sanitization and unit test for stepfn

* master_size api changes

* Admin API changes (#845)

1. /UpdateParameterStore
2. /QueueExecutionAttributes/<queue_type>/<queue_name>

* PD-257044-changes-for-databus (#846)

* added datadog metrics changes

* [PD-257044] databus changes rebase (#848)

* PD-257044-changes-for-databus

* duplicated kafka and ssm util packages in emo sor module

* removed redundant dependencies

* admin api and queue name split

* Pd 257044 databus changes rebase (#849)

* PD-257044-changes-for-databus

* duplicated kafka and ssm util packages in emo sor module

* removed redundant dependencies

* additional changes

* additional changes

* additional changes

* removed unneccessary dependencies

* Revert "added datadog metrics changes"

This reverts commit 62dc31e.

* build fix

* endpoint change and databus changes

* updated updateRefInDatabus method

* address review commnets

* branch admin -prepare release emodb-6.5.205

* branch admin -prepare for next development iteration

---------

Co-authored-by: Rishu Yadav <[email protected]>
Co-authored-by: shanmohammed-bv <[email protected]>
Co-authored-by: Anurag Dubey <[email protected]>
Co-authored-by: Anandu Jayan <[email protected]>
Co-authored-by: jenkins <[email protected]>
* fix serialization issue in databus

* remove unused code

* update pom versions

* Update sor/src/main/java/com/bazaarvoice/emodb/sor/core/DefaultDataStore.java

Co-authored-by: Igor Kruiper <[email protected]>

* fix build failure

---------

Co-authored-by: Igor Kruiper <[email protected]>
* add logs

* branch admin -prepare release emodb-6.5.212

* branch admin -prepare for next development iteration

* add default to Datstore

* branch admin -prepare release emodb-6.5.213

* branch admin -prepare for next development iteration

* format logs for release

---------

Co-authored-by: jenkins <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants