fix(contracts): load config from embedded #602
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was still some code reading from the filesystem instead of the embedded files in the binary. Regression introduced in #523. Fixes: #578 See also: #579 Signed-off-by: mudler <[email protected]>
These are going to be taken care of as part of #573 Signed-off-by: mudler <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #602 +/- ##
========================================
- Coverage 9.96% 9.80% -0.16%
========================================
Files 95 99 +4
Lines 5943 5892 -51
========================================
- Hits 592 578 -14
+ Misses 5276 5241 -35
+ Partials 75 73 -2 ☔ View full report in Codecov by Sentry. |
6 tasks
restevens402
added a commit
that referenced
this pull request
Oct 22, 2024
commit d654c56 Author: smeb y <[email protected]> Date: Mon Oct 21 15:55:05 2024 +0800 fix(Dockerfile): add ca-certificate (#604) Update Dockerfile Signed-off-by: smeb y <[email protected]> commit c1e624a Author: Ettore Di Giacinto <[email protected]> Date: Thu Oct 17 15:23:03 2024 +0200 chore(docs): drop unnecessary duplicated content Signed-off-by: Ettore Di Giacinto <[email protected]> commit 9529e4c Author: Ettore Di Giacinto <[email protected]> Date: Thu Oct 17 15:21:07 2024 +0200 chore(docs): update .env.example (#603) Signed-off-by: Ettore Di Giacinto <[email protected]> commit 038fad6 Author: Ettore Di Giacinto <[email protected]> Date: Thu Oct 17 14:55:25 2024 +0200 fix(contracts): load config from embedded (#602) * fix(contracts): load config from embedded There was still some code reading from the filesystem instead of the embedded files in the binary. Regression introduced in #523. Fixes: #578 See also: #579 Signed-off-by: mudler <[email protected]> * chore(tests): temporarly disable Twitter tests These are going to be taken care of as part of #573 Signed-off-by: mudler <[email protected]> --------- Signed-off-by: mudler <[email protected]> commit a8a77a6 Author: Brendan Playford <[email protected]> Date: Tue Oct 15 14:32:29 2024 -0700 feat(twitter): Implement random sleep and improve login process (#601) - Add RandomSleep function to introduce variability in request timing - Update NewScraper to use RandomSleep before and after login attempts - Adjust sleep duration range to 500ms - 2s for more natural behavior - Improve error handling and logging in the login process commit 01ec8c4 Author: Brendan Playford <[email protected]> Date: Tue Oct 15 08:59:08 2024 -0700 chore(version): update protocol version and update twitter_cookies.example.json commit 6f594e1 Author: Brendan Playford <[email protected]> Date: Tue Oct 15 08:48:01 2024 -0700 feat(twitter): Enhanced Twitter Worker Selection Algorithm (#591) * Add detailed error logging and track worker update time Enhanced the worker manager to append specific error messages to a list for better debugging. Additionally, updated node data to track the last update time, improving data consistency and traceability. * Update version.go * refactor(twitter): remove retry functionality from scraper - Remove Retry function and MaxRetries constant from config.go - Update ScrapeFollowersForProfile, ScrapeTweetsProfile, and ScrapeTweetsByQuery to remove Retry wrapper - Adjust error handling in each function to directly return errors - Simplify code structure and reduce complexity - Maintain rate limit handling functionality * chore(workers): update max workers to 50 * chore(workers): upate to 25 * feat(pubsub): improve node sorting algorithm for Twitter reliability - Prioritize nodes with more recent last returned tweets - Maintain high importance for total returned tweet count - Consider time since last timeout to allow recovery from temporary issues - Deprioritize nodes with recent "not found" occurrences - Remove NotFoundCount from sorting criteria This change aims to better balance node performance and recent activity, while allowing nodes to recover quickly from temporary issues like rate limiting. * feat(workers): improve Twitter worker selection algorithm - Modify GetEligibleWorkers to use a specialized selection for Twitter workers - Introduce controlled randomness in Twitter worker selection - Balance between prioritizing high-performing Twitter workers and fair distribution - Maintain existing behavior for non-Twitter worker selection - Preserve handling of local worker and respect original worker limit This change enhances the worker selection algorithm for Twitter tasks to provide a better balance between utilizing top-performing nodes and ensuring fair work distribution. It introduces a dynamic pool size calculation and controlled randomness for Twitter workers, while maintaining the existing round-robin approach for other worker types. --------- Co-authored-by: Bob Stevens <[email protected]> commit f09fb20 Author: Brendan Playford <[email protected]> Date: Tue Oct 8 15:38:45 2024 -0700 Feat(workers) implement adaptive worker selection for improved task distribution (#589) * feat(worker-selection): Implement performance-based worker sorting - Add performance metrics fields to NodeData struct - Implement NodeSorter for flexible sorting of worker nodes - Create SortNodesByTwitterReliability function for Twitter workers - Update GetEligibleWorkerNodes to use category-specific sorting - Modify GetEligibleWorkers to use sorted workers and add worker limit This commit enhances the worker selection process by prioritizing workers based on their performance metrics. It introduces a flexible sorting mechanism that can be easily extended to other worker categories in the future. The changes improve reliability and efficiency in task allocation across the Masa Oracle network. * feat(worker-selection): Implement priority-based selection for Twitter work - Update DistributeWork to use priority selection for Twitter category - Maintain round-robin selection for other work categories by shuffling workers - Integrate new GetEligibleWorkers function with work type-specific behavior - Respect MaxRemoteWorkers limit for all work types - Add distinct logging for Twitter and non-Twitter worker selection This commit enhances the work distribution process by implementing priority-based worker selection for Twitter-related tasks while preserving the existing round-robin behavior for other work types. It leverages the newly added performance metrics to choose the most reliable workers for Twitter tasks, and ensures consistent behavior for other categories by shuffling the worker list. This hybrid approach improves efficiency for Twitter tasks while maintaining the expected behavior for all other work types. * Update .gitignore * feat(worker-selection): Implement priority-based sorting for Twitter workers - Add LastNotFoundTime and NotFoundCount fields to NodeData struct - Enhance SortNodesByTwitterReliability function with multi-criteria sorting: 1. Prioritize nodes found more often (lower NotFoundCount) 2. Consider recency of last not-found occurrence 3. Sort by higher number of returned tweets 4. Consider recency of last returned tweet 5. Prioritize nodes with fewer timeouts 6. Consider recency of last timeout 7. Use PeerId for stable sorting when no performance data is available - Remove random shuffling from GetEligibleWorkers function This commit improves worker selection for Twitter tasks by implementing a more sophisticated sorting algorithm that takes into account node reliability and performance metrics. It aims to enhance the efficiency and reliability of task distribution in the Masa Oracle network. * feat(worker-selection): Update Twitter fields in NodeData and Worker Manager Add functions to update Twitter-related metrics in NodeData and integrate updates into Worker Manager processes. This ensures accurate tracking of tweet-related events and peer activity in the system. * feat(worker-selection): Add unit tests for NodeData and NodeDataTracker Introduce unit tests for the NodeData and NodeDataTracker functionalities, covering scenarios involving updates to Twitter-related fields. These tests ensure the correctness of the UpdateTwitterFields method in NodeData and the UpdateNodeDataTwitter method in NodeDataTracker. * chore(workers): update timeouts and bump version --------- Co-authored-by: Bob Stevens <[email protected]> commit 0ef0df4 Author: Brendan Playford <[email protected]> Date: Tue Oct 8 14:37:01 2024 -0700 feat(api): Add configurable API server enablement (#586) * feat(api): Add configurable API server enablement This commit introduces a new feature that allows the API server to be conditionally enabled or disabled based on configuration. The changes include: 1. In cmd/masa-node/main.go: - Refactored signal handling into a separate function `handleSignals` - Added conditional logic to start the API server only if enabled - Improved logging to indicate API server status 2. In pkg/config/app.go: - Added `APIEnabled` field to the `AppConfig` struct - Set default value for `APIEnabled` to false in `setDefaultConfig` - Added command-line flag for `apiEnabled` in `setCommandLineConfig` 3. In pkg/config/constants.go: - Added `APIEnabled` constant for environment variable configuration These changes provide more flexibility in node configuration, allowing users to run the node with or without the API server. This can be useful for security purposes or in scenarios where the API is not needed. The API can now be enabled via: - Environment variable: API_ENABLED=true - Command-line flag: --apiEnabled - Configuration file: apiEnabled: true By default, the API server will be disabled for enhanced security. * chore(config): update to take api-enabled=true and update Makefile with run-api case * Update Makefile
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
There was still some code reading from the filesystem instead of the embedded files in the binary.
Regression introduced in #523.
Fixes: #578
Notes for Reviewers
It also adds a simple unit test to check whether we can read from the embedded contracts. Since I was at it I also disabled the Twitter tests until properly fixed with #573
See also: #579
Signed commits