feat: Add solver metrics

[bgins]

Summary

This pull request makes the following changes:

• Add solver metrics options
• Add metrics config and provider
• Configure solver with metrics
• Add meter to solver controller
• Add solver system metrics
• Add solver deal state metrics
• Add solver matcher metrics

This pull request also includes a new solver store method and two minor refactors to support the work above:

• Add GetDealsAll solver store method
• Initialize tracer provider with noop provider
• Lift resource instantiation to SetupOTelSDK

We would like improved metrics observability on the solver to monitor system and process performance, deal matching stats, and deal status stats.

Task/Issue reference

Closes: #434

Test plan

Start the observability server on the bgins/feat-add-solver-dashboards branch. Open localhost:3000 in a browser and select "Dashboards" from the hamburger menu on the left.

Two dashboards have been added:

• Solver System Metrics (system and process metrics)
• Solver Metrics (deal state metrics, matcher metrics)

Start the stack. Run the solver with explicit metrics configuration:

export ENABLE_METRICS=true
export METRICS_URL="http://localhost:8500"
export METRICS_TOKEN="eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJhdXRob3JpemVkIjp0cnVlLCJ1c2VyIjoicmVzb3VyY2UtcHJvdmlkZXIifQ.n36M_ngwC4XPQ_pEkkWAnPiOinnx6-0VO1v_WgCTUEERD7b_p9KHCU6SY5bUdFh5UXRZHAhc1gfyc7rjAnmeDQ"
./stack solver

Run a job. Metrics are collected and sent in batches once per minute, so it may take a moment before metrics are reported.

Once reported, the Solver System Metrics dashboard should display standard metrics for system and process performance.

The Solver Metrics dashboard displays deal state metrics. For example, in a local run we can see deal states over a couple jobs:

The Solver Metrics dashboard also displays job offer, resource offer, and deal counts.

Details

This pull request adds metrics to the solver. We may want to add the system metrics for the resource provider in the future, but we start with the solver to test the implementation before making it broadly available. It should be easy to add to resource providers in the future, though we may want to revisit some of the configuration decisions made here.

Configuration

This pull request configures metrics with MetricsOptions and MetricsConfig. We have kept configuration separate from TelemetryOptions and TelemetryConfig for a couple of reasons:

• We only want metrics config for the solver
• We may send metrics and logs to a local collector like FluentBit, while still sending traces directly to the observability instance

We may want to revisit configuration in the future, but this approach is a first iteration while we consider the bigger picture of where and how metrics will be used.

Metrics must be configured explicitly with environment variables or command line options. My thinking here is that these metrics are mostly interesting in production, and it's easy enough to configure them in our deployments.

System Metrics

This pull request includes an initial set of system metrics including:

• System uptime
• System load average (1, 5, 15)
• System CPU used percent and logical CPU count
• System memory used percent, used, available, and total
• System network connections count, bytes transmitted, bytes received
• System filesystem used percent, used, free, total, inodes used percent
• Process uptime, memory used percent, CPU used percent, connections count, num threads used

Solver matching metrics

This pull request records the following matcher metrics:

• Resource offers available to match
• Job offers available to match
• Deals made during matching

These metrics are recorded once per solver control loop iteration.

Solver deal state metrics

This pull request records deal state metrics once per solver control loop iteration. This should be considered a form of sampling because deal states are updated outside of control loop iteration.

Eventually, we should implement UpDownCounters for each deal state when we have a better handle on the job lifecycle and its state transitions.

Computing deal states once per control loop iteration may have performance implications. On my local runs with a single job, the traces report ~12ms of time spent computing deal state metrics. Likely to be more in production, we can keep an eye on it.

Related issues or PRs

Epic: https://github.com/Lilypad-Tech/internal/issues/345
Solver metrics dashboards: https://github.com/Lilypad-Tech/observability/pull/22
Future work: #439

[narbs91]

[nit] Is there anything else we might want to do if options.Enable is true? If not maybe we can combine the two if statements together

[bgins]

Yeah good call! Simplified this check. 👌

[narbs91]

More of a question here but in the case of the err block e=being entered on line 309, would otel be smart enough to still add the report_deal_metrics.done span or is there another event upstream that will highlight the error state?

[bgins]

Ooh good catch! It would not record the report_deal_metrics.done span event. Also realized I wasn't handling the reportDealMetrics error.

Made some updates and now it looks like:

lilypad/pkg/solver/controller.go

Lines 307 to 319 in 5e522c5

	span.AddEvent("report_deal_metrics.start")
	storedDeals, err := controller.store.GetDealsAll()
	if err != nil {
	span.SetStatus(codes.Error, "get all deals failed")
	span.RecordError(err)
	return err
	}
	err = reportDealMetrics(ctx, controller.meter, storedDeals)
	if err != nil {
	span.SetStatus(codes.Error, "report deal metrics failed")
	span.RecordError(err)
	}
	span.AddEvent("report_deal_metrics.done")

Note that we don't return an error when reportDealMetrics fails, we just record it in the trace. It's also logged in the reportDealMetrics function body.

[walkah]

LGTM, tested locally and ... got metrics! 🙌 🚀