feat: event tracking for work distribution #525
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implement EventTracker for in-memory event management - Create EventClient for sending events to external API - Add configuration options with sensible defaults - Provide convenience methods for common event types - Integrate logrus for comprehensive error handling and logging - Ensure thread-safety with mutex locks
- Add error check for nil EventClient - Log SendEvent errors - Return SendEvent errors to caller - Clarify nil EventTracker error message
|
PR description is too short and seems to not fulfill PR template, please fill in |
⚠ golint failed (.)Found 212 lint suggestions; failing. Show Detail |
⚠ goimports failed (.)
|
⚠ shadow failed (.) |
⚠ gosec failed (.)Show Detail |
⚠ errcheck failed (.) |
teslashibe
changed the title
⚠ staticcheck failed (.) |
|
PR description is too short and seems to not fulfill PR template, please fill in |
teslashibe
changed the title
|
PR description is too short and seems to not fulfill PR template, please fill in |
|
PR description is too short and seems to not fulfill PR template, please fill in |
⚠ golint failed (.)Found 212 lint suggestions; failing. Show Detail |
⚠ goimports failed (.)
|
⚠ gosec failed (.)Show Detail |
⚠ errcheck failed (.) |
⚠ staticcheck failed (.) |
⚠ shadow failed (.) |
3 tasks
|
PR description is too short and seems to not fulfill PR template, please fill in |
|
PR description is too short and seems to not fulfill PR template, please fill in |
… for work requests - Add TrackWorkRequest in event_library.go to use host node's peer ID - Modify SearchTweetsRecent in handlers_data.go to pass Host peer ID - Add null checks for EventTracker and Node in API handlers - Ensure proper initialization of EventTracker in NewAPI function - Update API struct to include EventTracker field - Add debug logging for EventTracker creation and API initialization This commit enhances the accuracy of event tracking by using the host node's peer ID instead of the client IP when logging work requests. It also improves robustness with additional null checks and error handling.
|
PR description is too short and seems to not fulfill PR template, please fill in |
- Introduce DataSource field to Event struct for specifying data origins - Add Payload field to Event struct for storing request details - Update TrackWorkRequest function to include DataSource and Payload params - Modify SearchTweetsRecent handler to use new TrackWorkRequest signature - Define constants for common data sources (Twitter, Discord, Web, Telegram) - Ensure server-side timestamp generation for consistency and reliability This change enhances event tracking granularity and provides more context for work requests across different data sources.
|
PR description is too short and seems to not fulfill PR template, please fill in |
- Remove Timestamp field from Event struct in event.go - Eliminate timestamp-related operations in event_library.go - Remove timestamp setting and logging in TrackAndSendEvent function - Update validateEvent function to exclude timestamp validation This change centralizes timestamp management to the server side, ensuring consistency and eliminating potential issues with client-side time discrepancies. Timestamp handling is now delegated to the server's logging system or database layer for more accurate and reliable event timing.
|
PR description is too short and seems to not fulfill PR template, please fill in |
This was referenced Aug 26, 2024
|
PR description is too short and seems to not fulfill PR template, please fill in |
- Update TrackWorkRequest call in SearchTweetsRecent to include full request payload - Add logging of work request input in EventTracker This change improves the tracking and logging of work requests, providing more comprehensive information for debugging and analysis purposes.
|
PR description is too short and seems to not fulfill PR template, please fill in |
- Delete SearchTweetsTrends function from handlers_data.go - Remove /data/twitter/tweets/trends route from SetupRoutes in routes.go This change simplifies the API by removing the Twitter trends functionality, which may have been deprecated or no longer needed.
- Delete sentiment analysis handlers for tweets, Discord, Telegram, and web content - Remove GetLLMModelsHandler and its associated route - Remove corresponding route definitions from SetupRoutes function This change simplifies the API by removing sentiment analysis functionality and LLM model listing, which may no longer be needed or are being moved to a different service.
- Implement work request tracking for SearchTweetsProfile - Add event tracking for SearchTweetsRecent - Integrate event tracking in SearchTwitterFollowers - Ensure proper error handling and logging for event tracking - Reorganize handler functions for better readability - Remove commented-out ExchangeDiscordTokenHandler function
|
PR description is too short and seems to not fulfill PR template, please fill in |
Added defer statements to ensure streams are properly closed after use in OracleNode methods. This change helps prevent resource leaks and improves code maintainability by consolidating stream closure logic.
Integrate `EventTracker` to monitor worker failures and track task completion outcomes. Adjust log message level related to stream closure errors for consistency.
mudler
pushed a commit
that referenced
this pull request
Sep 17, 2024
* feat: add event tracking and analytics package - Implement EventTracker for in-memory event management - Create EventClient for sending events to external API - Add configuration options with sensible defaults - Provide convenience methods for common event types - Integrate logrus for comprehensive error handling and logging - Ensure thread-safety with mutex locks * chore: upadte event_library and add todos * feat: event tracking package with library * chore: update readme for event tracking package ready for implementation * fix: timestamps in UTC * feat: Improve error handling in TrackAndSendEvent function - Add error check for nil EventClient - Log SendEvent errors - Return SendEvent errors to caller - Clarify nil EventTracker error message * adding the event tracker handling to the work distribution. * chore: add API local base URL and tidy * update go.mod * fix go.mod unused import * Squashed commit of the following: commit e8e1822 Author: Ettore Di Giacinto <[email protected]> Date: Fri Aug 23 17:54:21 2024 +0200 ci: Rename static.yml to static.yml Signed-off-by: Ettore Di Giacinto <[email protected]> commit efb84f9 Author: Ettore Di Giacinto <[email protected]> Date: Fri Aug 23 17:53:15 2024 +0200 feat(tests): add baseline for an integration test (#527) * feat(tests): add baseline for an integration test Signed-off-by: mudler <[email protected]> * chore: drop obsoleted code Signed-off-by: mudler <[email protected]> * fix: HasBootnodes otherwise panics if no bootnodes are specified Signed-off-by: mudler <[email protected]> * chore(integration tests): make it possible to run the tests without generating identity keys - Introduce GetP2PMultiAddrs() to retrieve the node id multiaddress which is having p2p encapsulation, in order to be used as bootnode - Wire up some changes into the constructor of the oracle to allow to specify bootnodes and enable random identity generation Signed-off-by: mudler <[email protected]> * chore: drop unneeded code Signed-off-by: mudler <[email protected]> --------- Signed-off-by: mudler <[email protected]> commit 6dae3fe Author: Brendan Playford <[email protected]> Date: Mon Aug 19 14:01:08 2024 -0700 hotfix: .env formatting and description plus runpod and Docker image commit 5e5853e Author: Ettore Di Giacinto <[email protected]> Date: Mon Aug 19 11:03:16 2024 +0200 fix(ci): point correctly to main in `goreleaser.yml` (#516) * fix(ci): point correctly to main Otherwise goreleaser will fail building. Fixes: #514 Signed-off-by: mudler <[email protected]> * detect goreleaser Signed-off-by: mudler <[email protected]> --------- Signed-off-by: mudler <[email protected]> * Add AddrInfo to localWorker in worker_selection.go This commit adds the AddrInfo struct to the localWorker object in worker_selection.go. This change ensures that localWorker now contains information about its ID and addresses, which may be necessary for future network operations or worker identification. * Refactor event tracking to include peer ID and other details Updated event tracking functions to accept additional parameters such as peer ID. Simplified the event structure and ensured that all necessary information is captured and validated before sending events. * feat : Improve event tracking with TrackWorkRequest with Host peer ID for work requests - Add TrackWorkRequest in event_library.go to use host node's peer ID - Modify SearchTweetsRecent in handlers_data.go to pass Host peer ID - Add null checks for EventTracker and Node in API handlers - Ensure proper initialization of EventTracker in NewAPI function - Update API struct to include EventTracker field - Add debug logging for EventTracker creation and API initialization This commit enhances the accuracy of event tracking by using the host node's peer ID instead of the client IP when logging work requests. It also improves robustness with additional null checks and error handling. * feat (event): Add DataSource and Payload to event tracking - Introduce DataSource field to Event struct for specifying data origins - Add Payload field to Event struct for storing request details - Update TrackWorkRequest function to include DataSource and Payload params - Modify SearchTweetsRecent handler to use new TrackWorkRequest signature - Define constants for common data sources (Twitter, Discord, Web, Telegram) - Ensure server-side timestamp generation for consistency and reliability This change enhances event tracking granularity and provides more context for work requests across different data sources. * refactor: Move event timestamp handling to server side - Remove Timestamp field from Event struct in event.go - Eliminate timestamp-related operations in event_library.go - Remove timestamp setting and logging in TrackAndSendEvent function - Update validateEvent function to exclude timestamp validation This change centralizes timestamp management to the server side, ensuring consistency and eliminating potential issues with client-side time discrepancies. Timestamp handling is now delegated to the server's logging system or database layer for more accurate and reliable event timing. * feat(api): enhance work request tracking for SearchTweetsRecent - Update TrackWorkRequest call in SearchTweetsRecent to include full request payload - Add logging of work request input in EventTracker This change improves the tracking and logging of work requests, providing more comprehensive information for debugging and analysis purposes. * feat(api): remove Twitter trends endpoint and handler - Delete SearchTweetsTrends function from handlers_data.go - Remove /data/twitter/tweets/trends route from SetupRoutes in routes.go This change simplifies the API by removing the Twitter trends functionality, which may have been deprecated or no longer needed. * refactor(api): remove sentiment analysis and LLM model endpoints - Delete sentiment analysis handlers for tweets, Discord, Telegram, and web content - Remove GetLLMModelsHandler and its associated route - Remove corresponding route definitions from SetupRoutes function This change simplifies the API by removing sentiment analysis functionality and LLM model listing, which may no longer be needed or are being moved to a different service. * feat(api): add event tracking for Twitter-related endpoints - Implement work request tracking for SearchTweetsProfile - Add event tracking for SearchTweetsRecent - Integrate event tracking in SearchTwitterFollowers - Ensure proper error handling and logging for event tracking - Reorganize handler functions for better readability - Remove commented-out ExchangeDiscordTokenHandler function * Streamlining stream closures in OracleNode methods Added defer statements to ensure streams are properly closed after use in OracleNode methods. This change helps prevent resource leaks and improves code maintainability by consolidating stream closure logic. * Update event_library.go * Restore event tracking for worker failures and task completion Integrate `EventTracker` to monitor worker failures and track task completion outcomes. Adjust log message level related to stream closure errors for consistency. --------- Co-authored-by: Bob Stevens <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Event Tracking System and Refactor API Handlers
Overview
This PR introduces a new event tracking system and refactors several API handlers to incorporate this system. It also includes some minor adjustments to improve logging and code organization.
Key Changes
1. New Event Tracking System
eventpackage with the following components:EventTracker: Manages event tracking and sendingEventClient: Handles sending events to an external APIEvent: Struct representing individual events2. API Refactoring
NewAPIfunction to create and include anEventTrackerSearchTweetsProfile,SearchTweetsRecent, andSearchTwitterFollowershandlers to use the new event tracking system3. Logging Improvements
main.goto set log level to Debug4. Code Cleanup
Detailed Changes
cmd/masa-node/main.go
pkg/api/api.go
EventTrackerto theAPIstructNewAPIto create and include anEventTrackerpkg/api/handlers_data.go
SearchTweetsProfile,SearchTweetsRecent, andSearchTwitterFollowersto use event trackingpkg/api/routes.go
pkg/event/* (new files)
Testing
Documentation
Next Steps
Potential Impacts
This change may affect the performance of API calls due to the added event tracking. Monitor response times after deployment.
The removal of certain endpoints (e.g., sentiment analysis) may break existing integrations. Ensure all stakeholders are informed of these changes.
Yes, I signed my commits.