ci: update workflow for deploy based on tag. #485
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5u6r054
changed the title
nolanjacobson
added a commit
that referenced
this pull request
Aug 8, 2024
* ci: update workflow for deploy based on tag. (#485) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * ci: remove unneeded / untested workflows (#487) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * this needs work, not functional now. * handy cleanup script to clear out old workflows * give the workflow for running Go tests a more descriptive name * remove duplicate workflow. * update cleanup script to handle multiple workflows with same name * fix auth pattern to GCP * fix: repair deploy portion of the build and deploy to gcp bootnodes workflow (#488) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * this needs work, not functional now. * handy cleanup script to clear out old workflows * give the workflow for running Go tests a more descriptive name * remove duplicate workflow. * update cleanup script to handle multiple workflows with same name * fix auth pattern to GCP * update workflow to deploy to test env. in GCP * chore: 2fa initial fixes * stash * chore: updates to docs.go for swagger --------- Co-authored-by: J2D3 <[email protected]>
teslashibe
added a commit
that referenced
this pull request
Aug 16, 2024
* feat: refactor sendWork function in workers.go to make modular - Separate out localWorker to send local work requests - Separate out remoteWorker to enable selection and handling of remote work requests - Lay foundation for remote worker selection - even though queue is limited all remoteWorkers still get the same request which is inefficient - Log number of responses in the queue - Make logs verbose for easy debugging of localWork and remoteWork * feat: worker-sub selection refactor * feat: Implement round-robin worker selection for distributed task processing - Add round-robin iterator for fair worker distribution - Prioritize local worker when eligible - Cycle through remote workers in subsequent calls - Improve error handling and logging for worker responses - Enhance code readability and maintainability This update ensures a balanced workload across all available workers over time, while still prioritizing local processing when possible. * Improve error handling and resilience in send_work.go - Add maxSpawnAttempts constant to limit remote worker spawn attempts - Implement retry mechanism for spawning remote workers with exponential backoff - Introduce spawnRemoteWorker function for better organization and error handling - Enhance logging for better visibility into worker spawning and processing - Improve handling of dead letters and timeouts in remote worker operations - Refactor handleRemoteWorker to be more robust against transient failures - Update tryWorker function to handle both local and remote worker scenarios - Implement round-robin worker selection with retries in SendWork function These changes aim to increase the reliability of the worker system, particularly when dealing with remote workers, and provide better insights into error scenarios for easier debugging and monitoring. * feat: add config.go to worker package to simplify settings * refactor(workers): move worker selection logic to separate file - Created new file worker_selection.go to house worker selection logic - Moved GetEligibleWorkers, isEligibleRemoteWorker, and RoundRobinIterator to worker_selection.go - Updated send_work.go to use new exported functions from worker_selection.go - Renamed newRoundRobinIterator to NewRoundRobinIterator for proper exporting - Updated imports and function calls in send_work.go to reflect new structure - Improved code organization and modularity * fix: duplication of getEligibleWorkers func * feat: Enhance worker selection process with configurable remote worker limit - Add MaxRemoteWorkers to WorkerConfig in config.go - Update tryWorkersRoundRobin function in send_work.go to respect MaxRemoteWorkers limit - Implement fallback mechanism to local worker when remote workers fail or are unavailable - Add detailed logging throughout the worker selection process for better debugging - Ensure a last resort local worker is always available if no other workers are found * fix: fine tune timeouts with testing * Refactor worker selection and add eligibility check Refactored worker selection to use NodeTracker's new GetEligibleWorkerNodes method and introduced an eligibility check for staked workers. Added a new utility function for converting strings to WorkerTypes and a method in NodeEventTracker to retrieve eligible nodes based on work categories. * fixed case where err was reassigned and hiding the original * fix: local work is not timing out for long queries >20s * added more error handling and bubbled up to the tryWorker level * chore: swagger updates (#494) * ci: update workflow for deploy based on tag. (#485) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * ci: remove unneeded / untested workflows (#487) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * this needs work, not functional now. * handy cleanup script to clear out old workflows * give the workflow for running Go tests a more descriptive name * remove duplicate workflow. * update cleanup script to handle multiple workflows with same name * fix auth pattern to GCP * fix: repair deploy portion of the build and deploy to gcp bootnodes workflow (#488) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * this needs work, not functional now. * handy cleanup script to clear out old workflows * give the workflow for running Go tests a more descriptive name * remove duplicate workflow. * update cleanup script to handle multiple workflows with same name * fix auth pattern to GCP * update workflow to deploy to test env. in GCP * chore: 2fa initial fixes * stash * chore: updates to docs.go for swagger --------- Co-authored-by: J2D3 <[email protected]> * Enable randomized node selection and improve config settings Shuffle eligible nodes before selecting workers. Added context-based connection attempts to ensure we can connect to selected worker before returning. Adjusted AppConfig initialization order and added version flag for CLI configuration. * x: add error handling for peer info creation and fine tune timeouts - Check and log errors from NewMultiaddr and AddrInfoFromP2pAddr - Resolve "value of err is never used" linter warning - Change timeouts on workers to 20s and 15s respectively * chore: add configurable connection timeout to config.go - Add ConnectionTimeout to WorkerConfig in config.go - Update GetEligibleWorkers to use the configurable timeout * chore: revert makefile version tagger * Adjust time intervals and remove unnecessary peer removal Updated sleep durations to one minute for efficiency and commented out peer removal logic to retain node activity tracking. Also removed the callback for peer removal in DHT initialization. * resolving comments in PR * fix: cleanup logs and tune timeouts * Add worker management for distributed task handling Implemented `WorkHandlerManager` to manage different work handlers, and integrated it for handling remote and local worker tasks. Introduced new functions in the workers package to execute and manage work streams. Updated API layer to utilize the new worker management system for request distribution and response handling. * fix multiaddress and local worker assignment * fix incorrect node reference * Add length-prefix to stream communication This update adds a length-prefix to both request and response messages in the stream communication to ensure proper message boundary handling. It includes encoding the length of the messages using big-endian encoding before writing to the stream and reading the length before processing the actual messages. This change improves the robustness and reliability of streaming operations. * update logging and pass back original error it happened on remote work. * Remove actor framework and work distribution logic Deleted all files and scripts related to the actor-based framework and work distribution. This includes the removal of `workers.go`, `handlers.go`, and corresponding protobuf messages and build scripts. Restructured the package and broke up the handlers into smaller more manageable files * Refactor error handling and JSON tags in worker types Updated the error field in WorkResponse to a string as it should have been. Also, added JSON tags to struct fields for better JSON serialization. updated assignment of Error in handler functions to use formatted strings to conform to the data type change. * Fixed introduced bug with error in remote worker also fixed logging to correctly display peerId when errors occur. * fixed incorrectly scoped assignment to response it needed to be the method declared not a new variable * fixing error messages * set request to nil so that it is not in the resulting JSON * fix: empty tweet case causing panic * fix: linting error and tidy * chore: Version 0.6.0 --------- Co-authored-by: Bob Stevens <[email protected]> Co-authored-by: Nolan Jacobson <[email protected]> Co-authored-by: J2D3 <[email protected]>
teslashibe
added a commit
that referenced
this pull request
Aug 16, 2024
* chore: 2fa initial fixes * chore: support for Telegram 2FA * chore: update swagger * chore: proper error checking * docs: update to docs to support 2fa * fix where app was using ApplicationVersion instead of ProtocolVersion * Feat/libp2p worker (#504) * feat: refactor sendWork function in workers.go to make modular - Separate out localWorker to send local work requests - Separate out remoteWorker to enable selection and handling of remote work requests - Lay foundation for remote worker selection - even though queue is limited all remoteWorkers still get the same request which is inefficient - Log number of responses in the queue - Make logs verbose for easy debugging of localWork and remoteWork * feat: worker-sub selection refactor * feat: Implement round-robin worker selection for distributed task processing - Add round-robin iterator for fair worker distribution - Prioritize local worker when eligible - Cycle through remote workers in subsequent calls - Improve error handling and logging for worker responses - Enhance code readability and maintainability This update ensures a balanced workload across all available workers over time, while still prioritizing local processing when possible. * Improve error handling and resilience in send_work.go - Add maxSpawnAttempts constant to limit remote worker spawn attempts - Implement retry mechanism for spawning remote workers with exponential backoff - Introduce spawnRemoteWorker function for better organization and error handling - Enhance logging for better visibility into worker spawning and processing - Improve handling of dead letters and timeouts in remote worker operations - Refactor handleRemoteWorker to be more robust against transient failures - Update tryWorker function to handle both local and remote worker scenarios - Implement round-robin worker selection with retries in SendWork function These changes aim to increase the reliability of the worker system, particularly when dealing with remote workers, and provide better insights into error scenarios for easier debugging and monitoring. * feat: add config.go to worker package to simplify settings * refactor(workers): move worker selection logic to separate file - Created new file worker_selection.go to house worker selection logic - Moved GetEligibleWorkers, isEligibleRemoteWorker, and RoundRobinIterator to worker_selection.go - Updated send_work.go to use new exported functions from worker_selection.go - Renamed newRoundRobinIterator to NewRoundRobinIterator for proper exporting - Updated imports and function calls in send_work.go to reflect new structure - Improved code organization and modularity * fix: duplication of getEligibleWorkers func * feat: Enhance worker selection process with configurable remote worker limit - Add MaxRemoteWorkers to WorkerConfig in config.go - Update tryWorkersRoundRobin function in send_work.go to respect MaxRemoteWorkers limit - Implement fallback mechanism to local worker when remote workers fail or are unavailable - Add detailed logging throughout the worker selection process for better debugging - Ensure a last resort local worker is always available if no other workers are found * fix: fine tune timeouts with testing * Refactor worker selection and add eligibility check Refactored worker selection to use NodeTracker's new GetEligibleWorkerNodes method and introduced an eligibility check for staked workers. Added a new utility function for converting strings to WorkerTypes and a method in NodeEventTracker to retrieve eligible nodes based on work categories. * fixed case where err was reassigned and hiding the original * fix: local work is not timing out for long queries >20s * added more error handling and bubbled up to the tryWorker level * chore: swagger updates (#494) * ci: update workflow for deploy based on tag. (#485) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * ci: remove unneeded / untested workflows (#487) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * this needs work, not functional now. * handy cleanup script to clear out old workflows * give the workflow for running Go tests a more descriptive name * remove duplicate workflow. * update cleanup script to handle multiple workflows with same name * fix auth pattern to GCP * fix: repair deploy portion of the build and deploy to gcp bootnodes workflow (#488) * update workflow for deploy based on tag. * give workflow better name, fix env. refs. * this needs work, not functional now. * handy cleanup script to clear out old workflows * give the workflow for running Go tests a more descriptive name * remove duplicate workflow. * update cleanup script to handle multiple workflows with same name * fix auth pattern to GCP * update workflow to deploy to test env. in GCP * chore: 2fa initial fixes * stash * chore: updates to docs.go for swagger --------- Co-authored-by: J2D3 <[email protected]> * Enable randomized node selection and improve config settings Shuffle eligible nodes before selecting workers. Added context-based connection attempts to ensure we can connect to selected worker before returning. Adjusted AppConfig initialization order and added version flag for CLI configuration. * x: add error handling for peer info creation and fine tune timeouts - Check and log errors from NewMultiaddr and AddrInfoFromP2pAddr - Resolve "value of err is never used" linter warning - Change timeouts on workers to 20s and 15s respectively * chore: add configurable connection timeout to config.go - Add ConnectionTimeout to WorkerConfig in config.go - Update GetEligibleWorkers to use the configurable timeout * chore: revert makefile version tagger * Adjust time intervals and remove unnecessary peer removal Updated sleep durations to one minute for efficiency and commented out peer removal logic to retain node activity tracking. Also removed the callback for peer removal in DHT initialization. * resolving comments in PR * fix: cleanup logs and tune timeouts * Add worker management for distributed task handling Implemented `WorkHandlerManager` to manage different work handlers, and integrated it for handling remote and local worker tasks. Introduced new functions in the workers package to execute and manage work streams. Updated API layer to utilize the new worker management system for request distribution and response handling. * fix multiaddress and local worker assignment * fix incorrect node reference * Add length-prefix to stream communication This update adds a length-prefix to both request and response messages in the stream communication to ensure proper message boundary handling. It includes encoding the length of the messages using big-endian encoding before writing to the stream and reading the length before processing the actual messages. This change improves the robustness and reliability of streaming operations. * update logging and pass back original error it happened on remote work. * Remove actor framework and work distribution logic Deleted all files and scripts related to the actor-based framework and work distribution. This includes the removal of `workers.go`, `handlers.go`, and corresponding protobuf messages and build scripts. Restructured the package and broke up the handlers into smaller more manageable files * Refactor error handling and JSON tags in worker types Updated the error field in WorkResponse to a string as it should have been. Also, added JSON tags to struct fields for better JSON serialization. updated assignment of Error in handler functions to use formatted strings to conform to the data type change. * Fixed introduced bug with error in remote worker also fixed logging to correctly display peerId when errors occur. * fixed incorrectly scoped assignment to response it needed to be the method declared not a new variable * fixing error messages * set request to nil so that it is not in the resulting JSON * fix: empty tweet case causing panic * fix: linting error and tidy * chore: Version 0.6.0 --------- Co-authored-by: Bob Stevens <[email protected]> Co-authored-by: Nolan Jacobson <[email protected]> Co-authored-by: J2D3 <[email protected]> * chore: updates to docs (#495) * chore: updates to docs * chore: updates to docs * chore: updates to docs --------- Co-authored-by: Brendan Playford <[email protected]> * chore: 2fa initial fixes * chore: support for Telegram 2FA * chore: update swagger * chore: proper error checking * docs: update to docs to support 2fa --------- Co-authored-by: J2D3 <[email protected]> Co-authored-by: Brendan Playford <[email protected]> Co-authored-by: Bob Stevens <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes deploy so that it is triggered only by tags, and will release the lastest tag to the GCP bootnodes automatically.
Notes for Reviewers
Signed commits