Enabling tracing by usage of opentelemetry

[rem1-dev]

I've added tracing to the project by introducing mainly telemetry, opentelemetry and Tokyo crates. Having tracing allows analysis of bottleneck functions (functions that take majority of execution time).

[RCasatta]

Isn't prometheus trying to achieve the same thing? Should we remove that if we decide to use opentelemetry?

[philippem]

Isn't prometheus trying to achieve the same thing? Should we remove that if we decide to use opentelemetry?

• prometheus gives us metrics ( example there is a histogram of times spent making RPC calls to bitcoind). Many alerts can be directly built on prometheus metrics.
• opentelemtry (OTLP) gives us tracing within and between processes (for example we can look at a mempool update operation and see where the time is being spent)

They both enhance observability but are complementary.

[shesek]

Isn't prometheus trying to achieve the same thing? Should we remove that if we decide to use opentelemetry?

...
They both enhance observability but are complementary.

I'm still not quite sure I understand the differences. Prometheus can and is being used to track function execution times, for example in these functions (via _timer and its Drop) where opentelemetry was added too:

+    #[instrument(skip_all, name="Mempool::utxo")]
     pub fn utxo(&self, scripthash: &[u8]) -> Vec<Utxo> {
         let _timer = self.latency.with_label_values(&["utxo"]).start_timer();
         let entries = match self.history.get(scripthash) {
@@ -209,6 +216,7 @@ impl Mempool {
             .collect()
     }
 
+    #[instrument(skip_all, name="Mempool::stats")]
     // @XXX avoid code duplication with ChainQuery::stats()?
     pub fn stats(&self, scripthash: &[u8]) -> ScriptStats {
         let _timer = self.latency.with_label_values(&["stats"]).start_timer();
@@ -258,12 +266,14 @@ impl Mempool {
         stats
     }
 
+    #[instrument(skip_all, name="Mempool::txids")]
     // Get all txids in the mempool
     pub fn txids(&self) -> Vec<&Txid> {
         let _timer = self.latency.with_label_values(&["txids"]).start_timer();
         self.txstore.keys().collect()
     }

[shesek]

Missing newline (should probably cargo fmt the other changes too)

[shesek]

Was this intentionally disabled?

[shesek]

Why?

[shesek]

6 new (direct) dependencies, oh my 😅

Should we perhaps put this behind a feature flag so that the extra dependencies could be avoided by users that don't require tracing?

[shesek]

It's not critical, but it could aid review if the addition of #[instrument()] attributes was separated to its own commit

[philippem]

the feature name should maybe be "otlp-tracing" or "tracing". "enabled" is what we do with it.

[philippem]

let's call this "tracing"

[shesek]

How do you feel about defining a no-op #[instrument] macro when the tracing feature is disabled, so that we don't have to wrap it in #[cfg_attr(feature = "tracing",... )) every time?