feat: event tracking for work distribution (#525)

* feat: add event tracking and analytics package

- Implement EventTracker for in-memory event management
- Create EventClient for sending events to external API
- Add configuration options with sensible defaults
- Provide convenience methods for common event types
- Integrate logrus for comprehensive error handling and logging
- Ensure thread-safety with mutex locks

* chore: upadte event_library and add todos

* feat: event tracking package with library

* chore: update readme for event tracking package ready for implementation

* fix: timestamps in UTC

* feat: Improve error handling in TrackAndSendEvent function

- Add error check for nil EventClient
- Log SendEvent errors
- Return SendEvent errors to caller
- Clarify nil EventTracker error message

* adding the event tracker handling to the work distribution.

* chore: add API local base URL and tidy

* update go.mod

* fix go.mod unused import

* Squashed commit of the following:

commit e8e1822
Author: Ettore Di Giacinto <[email protected]>
Date:   Fri Aug 23 17:54:21 2024 +0200

    ci: Rename static.yml to static.yml

    Signed-off-by: Ettore Di Giacinto <[email protected]>

commit efb84f9
Author: Ettore Di Giacinto <[email protected]>
Date:   Fri Aug 23 17:53:15 2024 +0200

    feat(tests): add baseline for an integration test (#527)

    * feat(tests): add baseline for an integration test

    Signed-off-by: mudler <[email protected]>

    * chore: drop obsoleted code

    Signed-off-by: mudler <[email protected]>

    * fix: HasBootnodes otherwise panics if no bootnodes are specified

    Signed-off-by: mudler <[email protected]>

    * chore(integration tests): make it possible to run the tests without generating identity keys

    - Introduce GetP2PMultiAddrs() to retrieve the node id multiaddress which
    is having p2p encapsulation, in order to be used as bootnode
    - Wire up some changes into the constructor of the oracle to allow to
      specify bootnodes and enable random identity generation

    Signed-off-by: mudler <[email protected]>

    * chore: drop unneeded code

    Signed-off-by: mudler <[email protected]>

    ---------

    Signed-off-by: mudler <[email protected]>

commit 6dae3fe
Author: Brendan Playford <[email protected]>
Date:   Mon Aug 19 14:01:08 2024 -0700

    hotfix: .env formatting and description plus runpod and Docker image

commit 5e5853e
Author: Ettore Di Giacinto <[email protected]>
Date:   Mon Aug 19 11:03:16 2024 +0200

    fix(ci): point correctly to main in `goreleaser.yml` (#516)

    * fix(ci): point correctly to main

    Otherwise goreleaser will fail building.

    Fixes: #514

    Signed-off-by: mudler <[email protected]>

    * detect goreleaser

    Signed-off-by: mudler <[email protected]>

    ---------

    Signed-off-by: mudler <[email protected]>

* Add AddrInfo to localWorker in worker_selection.go

This commit adds the AddrInfo struct to the localWorker object in worker_selection.go. This change ensures that localWorker now contains information about its ID and addresses, which may be necessary for future network operations or worker identification.

* Refactor event tracking to include peer ID and other details

Updated event tracking functions to accept additional parameters such as peer ID. Simplified the event structure and ensured that all necessary information is captured and validated before sending events.

* feat : Improve event tracking with TrackWorkRequest with Host peer ID for work requests

- Add TrackWorkRequest in event_library.go to use host node's peer ID
- Modify SearchTweetsRecent in handlers_data.go to pass Host peer ID
- Add null checks for EventTracker and Node in API handlers
- Ensure proper initialization of EventTracker in NewAPI function
- Update API struct to include EventTracker field
- Add debug logging for EventTracker creation and API initialization

This commit enhances the accuracy of event tracking by using the host node's
peer ID instead of the client IP when logging work requests. It also improves
robustness with additional null checks and error handling.

* feat (event): Add DataSource and Payload to event tracking

- Introduce DataSource field to Event struct for specifying data origins
- Add Payload field to Event struct for storing request details
- Update TrackWorkRequest function to include DataSource and Payload params
- Modify SearchTweetsRecent handler to use new TrackWorkRequest signature
- Define constants for common data sources (Twitter, Discord, Web, Telegram)
- Ensure server-side timestamp generation for consistency and reliability

This change enhances event tracking granularity and provides more
context for work requests across different data sources.

* refactor: Move event timestamp handling to server side

- Remove Timestamp field from Event struct in event.go
- Eliminate timestamp-related operations in event_library.go
- Remove timestamp setting and logging in TrackAndSendEvent function
- Update validateEvent function to exclude timestamp validation

This change centralizes timestamp management to the server side,
ensuring consistency and eliminating potential issues with client-side
time discrepancies. Timestamp handling is now delegated to the server's
logging system or database layer for more accurate and reliable
event timing.

* feat(api): enhance work request tracking for SearchTweetsRecent

- Update TrackWorkRequest call in SearchTweetsRecent to include full request payload
- Add logging of work request input in EventTracker

This change improves the tracking and logging of work requests, providing more
comprehensive information for debugging and analysis purposes.

* feat(api): remove Twitter trends endpoint and handler

- Delete SearchTweetsTrends function from handlers_data.go
- Remove /data/twitter/tweets/trends route from SetupRoutes in routes.go

This change simplifies the API by removing the Twitter trends functionality,
which may have been deprecated or no longer needed.

* refactor(api): remove sentiment analysis and LLM model endpoints

- Delete sentiment analysis handlers for tweets, Discord, Telegram, and web content
- Remove GetLLMModelsHandler and its associated route
- Remove corresponding route definitions from SetupRoutes function

This change simplifies the API by removing sentiment analysis functionality
and LLM model listing, which may no longer be needed or are being moved to a
different service.

* feat(api): add event tracking for Twitter-related endpoints

- Implement work request tracking for SearchTweetsProfile
- Add event tracking for SearchTweetsRecent
- Integrate event tracking in SearchTwitterFollowers
- Ensure proper error handling and logging for event tracking
- Reorganize handler functions for better readability
- Remove commented-out ExchangeDiscordTokenHandler function

* Streamlining stream closures in OracleNode methods

Added defer statements to ensure streams are properly closed after use in OracleNode methods. This change helps prevent resource leaks and improves code maintainability by consolidating stream closure logic.

* Update event_library.go

* Restore event tracking for worker failures and task completion

Integrate `EventTracker` to monitor worker failures and track task completion outcomes. Adjust log message level related to stream closure errors for consistency.

---------

Co-authored-by: Bob Stevens <[email protected]>

cmd/masa-node/main.go

@@ -23,6 +23,9 @@ import (
 
 func main() {
 
+	logrus.SetLevel(logrus.DebugLevel)
+	logrus.Debug("Log level set to Debug")
+
 	if len(os.Args) > 1 && os.Args[1] == "--version" {
 		logrus.Infof("Masa Oracle Node Version: %s\nMasa Oracle Protocol verison: %s", versioning.ApplicationVersion, versioning.ProtocolVersion)
 		os.Exit(0)

pkg/api/api.go

@@ -5,17 +5,33 @@ import (
 	"strconv"
 
 	"github.com/gin-gonic/gin"
+	"github.com/sirupsen/logrus"
 
 	masa "github.com/masa-finance/masa-oracle/pkg"
+	"github.com/masa-finance/masa-oracle/pkg/event"
 )
 
 type API struct {
-	Node *masa.OracleNode
+	Node         *masa.OracleNode
+	EventTracker *event.EventTracker
 }
 
 // NewAPI creates a new API instance with the given OracleNode.
 func NewAPI(node *masa.OracleNode) *API {
-	return &API{Node: node}
+	eventTracker := event.NewEventTracker(nil)
+	if eventTracker == nil {
+		logrus.Error("Failed to create EventTracker")
+	} else {
+		logrus.Debug("EventTracker created successfully")
+	}
+
+	api := &API{
+		Node:         node,
+		EventTracker: eventTracker,
+	}
+
+	logrus.Debugf("Created API instance with EventTracker: %v", api.EventTracker)
+	return api
 }
 
 // GetPathInt converts the path parameter with name to an int.