Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inbound fees: collect and display stats #113

Merged
merged 3 commits into from
Nov 20, 2024
Merged

inbound fees: collect and display stats #113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
363cab6
mod: bump lnd and lndclient
bitromortac Oct 17, 2024
c766291
collectors: collect inbound fee stats
bitromortac Oct 28, 2024
f7938f7
dashboards: add inbound fee stats
bitromortac Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions collectors/graph_collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,32 @@ package collectors
import (
"context"
"fmt"
"math"

"github.com/lightninglabs/lndclient"
"github.com/prometheus/client_golang/prometheus"
)

const (
// inboundFeeFeatureBits is the feature id used to advertise inbound
// fees in gossip TLV extensions.
inboundFeeFeatureBits = 55555

// Define some collector names and help texts for inbound fees.
inboundFeeRateName = "lnd_graph_inbound_fee_rate_msat_histogram"
inboundFeeBaseName = "lnd_graph_inbound_fee_base_msat_histogram"
inboundFeeRateHelp = "histogram of inbound fee rates for channel " +
"routing policies in msat"
inboundFeeBaseHelp = "histogram of inbound base fees for channel " +
"routing policies in msat"

// Define labels to categorize inbound fees into negative and positive
// buckets.
inboundFeeSignLabel = "sign"
inboundFeeSignLabelPositive = "positive"
inboundFeeSignLabelNegative = "negative"
)

// GraphCollector is a collector that keeps track of graph information.
type GraphCollector struct {
numEdgesDesc *prometheus.Desc
Expand Down Expand Up @@ -46,6 +67,9 @@ type GraphCollector struct {
minFeeRateMsatDesc *prometheus.Desc
avgFeeRateMsatDesc *prometheus.Desc

inboundFeeBaseMsatDesc *prometheus.Desc
inboundFeeRateMsatDesc *prometheus.Desc

medianMaxHtlcMsatDesc *prometheus.Desc
maxMaxHtlcMsatDesc *prometheus.Desc
minMaxHtlcMsatDesc *prometheus.Desc
Expand Down Expand Up @@ -207,6 +231,15 @@ func NewGraphCollector(lnd lndclient.LightningClient,
nil, nil,
),

inboundFeeBaseMsatDesc: prometheus.NewDesc(
inboundFeeBaseName, inboundFeeBaseHelp,
[]string{inboundFeeSignLabel}, nil,
),
inboundFeeRateMsatDesc: prometheus.NewDesc(
inboundFeeRateName, inboundFeeRateHelp,
[]string{inboundFeeSignLabel}, nil,
),

medianMaxHtlcMsatDesc: prometheus.NewDesc(
"lnd_graph_max_htlc_msat_median",
"median max htlc for a channel routing policy in msat",
Expand Down Expand Up @@ -274,6 +307,9 @@ func (g *GraphCollector) Describe(ch chan<- *prometheus.Desc) {
ch <- g.avgFeeRateMsatDesc
ch <- g.medianFeeRateMsatDesc

ch <- g.inboundFeeBaseMsatDesc
ch <- g.inboundFeeRateMsatDesc

ch <- g.minMaxHtlcMsatDesc
ch <- g.maxMaxHtlcMsatDesc
ch <- g.avgMaxHtlcMsatDesc
Expand Down Expand Up @@ -366,6 +402,27 @@ func (g *GraphCollector) collectRoutingPolicyMetrics(

feeBaseStats = newStatsCompiler(numEdges)
feeRateStats = newStatsCompiler(numEdges)

inboundFeeBaseStats = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: inboundFeeBaseName,
Help: inboundFeeBaseHelp,
Comment on lines +408 to +409
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe there's a better way to avoid the constants and extract the name and help text from the GraphCollector instance?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah as attributes? I think the constants fit better since they bind the name, label, etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was more looking at the pattern that we use for other values

	ch <- prometheus.MustNewConstMetric(
		g.minTimelockDeltaDesc, prometheus.GaugeValue,
		timelockReport.min,
	)

where we take the description which is defined in a single place in NewGraphCollector, but in this PR the constants are referenced in two places. We could use prometheus.MustNewConstHistogram to follow that pattern, but that seems to be harder to use since we would need to keep track of buckets ourselves.

Buckets: prometheus.ExponentialBuckets(
1, 2, 20, // 1 to 1_048_576 msat
),
},
[]string{inboundFeeSignLabel},
)
inboundFeeRateStats = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: inboundFeeRateName,
Help: inboundFeeRateHelp,
Buckets: prometheus.ExponentialBuckets(
1, 2, 15, // 1 to 32768 PPM ~ 3 %
),
},
[]string{inboundFeeSignLabel},
)
)

for _, edge := range edges {
Expand All @@ -385,6 +442,36 @@ func (g *GraphCollector) collectRoutingPolicyMetrics(

feeBaseStats.Observe(float64(policy.FeeBaseMsat))
feeRateStats.Observe(float64(policy.FeeRateMilliMsat))

// Collect optional non-zero inbound fee statistics.
_, ok := policy.CustomRecords[inboundFeeFeatureBits]
if ok {
absBase := math.Abs(float64(
policy.InboundBaseFeeMsat,
))
if policy.InboundBaseFeeMsat < 0 {
inboundFeeBaseStats.WithLabelValues(
inboundFeeSignLabelNegative,
).Observe(absBase)
} else if policy.InboundBaseFeeMsat > 0 {
inboundFeeBaseStats.WithLabelValues(
inboundFeeSignLabelPositive,
).Observe(absBase)
}

absRate := math.Abs(
float64(policy.InboundFeeRatePPM),
)
if policy.InboundFeeRatePPM < 0 {
inboundFeeRateStats.WithLabelValues(
inboundFeeSignLabelNegative,
).Observe(absRate)
} else if policy.InboundFeeRatePPM > 0 {
inboundFeeRateStats.WithLabelValues(
inboundFeeSignLabelPositive,
).Observe(absRate)
}
}
}
}

Expand Down Expand Up @@ -478,4 +565,7 @@ func (g *GraphCollector) collectRoutingPolicyMetrics(
g.medianFeeRateMsatDesc, prometheus.GaugeValue,
feeRateReport.median,
)

inboundFeeBaseStats.Collect(ch)
inboundFeeRateStats.Collect(ch)
}
137 changes: 77 additions & 60 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,107 +5,127 @@ require (
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/jessevdk/go-flags v1.5.0
github.com/jrick/logrotate v1.0.0
github.com/lightninglabs/lndclient v0.17.4-4
github.com/lightningnetwork/lnd v0.17.4-beta
github.com/lightninglabs/lndclient v0.18.0-6
github.com/lightningnetwork/lnd v0.18.3-beta
github.com/prometheus/client_golang v1.18.0
github.com/stretchr/testify v1.8.4
github.com/stretchr/testify v1.9.0
google.golang.org/grpc v1.59.0
)

require (
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
github.com/aead/siphash v1.0.1 // indirect
github.com/andybalholm/brotli v1.0.3 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/btcsuite/btcd v0.24.1-0.20240123000108-62e6af035ec5 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.3.2 // indirect
github.com/btcsuite/btcd v0.24.2-beta.rc1.0.20240625142744-cc26860b4026 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.3.3 // indirect
github.com/btcsuite/btcd/btcutil/psbt v1.1.8 // indirect
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0 // indirect
github.com/btcsuite/btcwallet v0.16.10-0.20240127010340-16b422a2e8bf // indirect
github.com/btcsuite/btcwallet/wallet/txauthor v1.3.2 // indirect
github.com/btcsuite/btcwallet/wallet/txrules v1.2.0 // indirect
github.com/btcsuite/btcwallet/wallet/txsizes v1.2.3 // indirect
github.com/btcsuite/btcwallet/walletdb v1.4.0 // indirect
github.com/btcsuite/btcwallet/wtxmgr v1.5.0 // indirect
github.com/btcsuite/btcwallet v0.16.10-0.20240718224643-db3a4a2543bd // indirect
github.com/btcsuite/btcwallet/wallet/txauthor v1.3.4 // indirect
github.com/btcsuite/btcwallet/wallet/txrules v1.2.1 // indirect
github.com/btcsuite/btcwallet/wallet/txsizes v1.2.4 // indirect
github.com/btcsuite/btcwallet/walletdb v1.4.2 // indirect
github.com/btcsuite/btcwallet/wtxmgr v1.5.3 // indirect
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd // indirect
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792 // indirect
github.com/btcsuite/winsvc v1.0.0 // indirect
github.com/cenkalti/backoff/v4 v4.1.3 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/containerd/continuity v0.3.0 // indirect
github.com/coreos/go-semver v0.3.0 // indirect
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f // indirect
github.com/coreos/go-systemd/v22 v22.3.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/crypto/blake256 v1.0.0 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect
github.com/decred/dcrd/lru v1.0.0 // indirect
github.com/dsnet/compress v0.0.1 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/fergusstrange/embedded-postgres v1.10.0 // indirect
github.com/decred/dcrd/crypto/blake256 v1.0.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 // indirect
github.com/decred/dcrd/lru v1.1.2 // indirect
github.com/docker/cli v20.10.17+incompatible // indirect
github.com/docker/docker v24.0.7+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/fergusstrange/embedded-postgres v1.25.0 // indirect
github.com/go-errors/errors v1.0.1 // indirect
github.com/go-logr/logr v1.3.0 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.4.2 // indirect
github.com/golang-migrate/migrate/v4 v4.17.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/btree v1.0.1 // indirect
github.com/google/uuid v1.3.1 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.5.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/imdario/mergo v0.3.12 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgconn v1.14.3 // indirect
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa // indirect
github.com/jackc/pgerrcode v0.0.0-20240316143900-6e2875d9b438 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.3 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/pgtype v1.14.0 // indirect
github.com/jackc/pgx/v4 v4.18.2 // indirect
github.com/jackc/pgx/v5 v5.3.1 // indirect
github.com/jonboulle/clockwork v0.2.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/juju/loggo v0.0.0-20210728185423-eebad3a902c4 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kkdai/bstream v1.0.0 // indirect
github.com/klauspost/compress v1.13.6 // indirect
github.com/klauspost/pgzip v1.2.5 // indirect
github.com/lib/pq v1.10.3 // indirect
github.com/lib/pq v1.10.9 // indirect
github.com/lightninglabs/gozmq v0.0.0-20191113021534-d20a764486bf // indirect
github.com/lightninglabs/neutrino v0.16.0 // indirect
github.com/lightninglabs/neutrino/cache v1.1.1 // indirect
github.com/lightningnetwork/lightning-onion v1.2.1-0.20230823005744-06182b1d7d2f // indirect
github.com/lightninglabs/neutrino v0.16.1-0.20240425105051-602843d34ffd // indirect
github.com/lightninglabs/neutrino/cache v1.1.2 // indirect
github.com/lightningnetwork/lightning-onion v1.2.1-0.20240712235311-98bd56499dfb // indirect
github.com/lightningnetwork/lnd/clock v1.1.1 // indirect
github.com/lightningnetwork/lnd/healthcheck v1.2.3 // indirect
github.com/lightningnetwork/lnd/kvdb v1.4.4 // indirect
github.com/lightningnetwork/lnd/fn v1.2.0 // indirect
github.com/lightningnetwork/lnd/healthcheck v1.2.5 // indirect
github.com/lightningnetwork/lnd/kvdb v1.4.10 // indirect
github.com/lightningnetwork/lnd/queue v1.1.1 // indirect
github.com/lightningnetwork/lnd/sqldb v1.0.4 // indirect
github.com/lightningnetwork/lnd/ticker v1.1.1 // indirect
github.com/lightningnetwork/lnd/tlv v1.1.1 // indirect
github.com/lightningnetwork/lnd/tlv v1.2.3 // indirect
github.com/lightningnetwork/lnd/tor v1.1.2 // indirect
github.com/ltcsuite/ltcd v0.0.0-20190101042124-f37f8bf35796 // indirect
github.com/mattn/go-isatty v0.0.16 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
github.com/mholt/archiver/v3 v3.5.0 // indirect
github.com/miekg/dns v1.1.43 // indirect
github.com/mitchellh/mapstructure v1.4.1 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/nwaples/rardecode v1.1.2 // indirect
github.com/pierrec/lz4/v4 v4.1.8 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/opencontainers/runc v1.1.12 // indirect
github.com/ory/dockertest/v3 v3.10.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.5.0 // indirect
github.com/prometheus/common v0.45.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rogpeppe/fastuuid v1.2.0 // indirect
github.com/sirupsen/logrus v1.9.2 // indirect
github.com/soheilhy/cmux v0.1.5 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stretchr/objx v0.5.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect
github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 // indirect
github.com/ulikunitz/xz v0.5.11 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
go.etcd.io/bbolt v1.3.7 // indirect
Expand All @@ -127,41 +147,38 @@ require (
go.uber.org/atomic v1.7.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
go.uber.org/zap v1.17.0 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/exp v0.0.0-20230315142452-642cacee5cc0 // indirect
golang.org/x/mod v0.10.0 // indirect
golang.org/x/net v0.23.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/term v0.18.0 // indirect
golang.org/x/crypto v0.22.0 // indirect
golang.org/x/exp v0.0.0-20240325151524-a685a6edb6d8 // indirect
golang.org/x/mod v0.16.0 // indirect
golang.org/x/net v0.24.0 // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/sys v0.19.0 // indirect
golang.org/x/term v0.19.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/time v0.0.0-20220224211638-0e9765cccd65 // indirect
golang.org/x/tools v0.9.1 // indirect
google.golang.org/genproto v0.0.0-20230822172742-b8732ec3820d // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230822172742-b8732ec3820d // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d // indirect
google.golang.org/grpc v1.59.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.19.0 // indirect
google.golang.org/genproto v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231030173426-d783a09b4405 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/errgo.v1 v1.0.1 // indirect
gopkg.in/macaroon-bakery.v2 v2.0.1 // indirect
gopkg.in/macaroon.v2 v2.1.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/uint128 v1.2.0 // indirect
modernc.org/cc/v3 v3.40.0 // indirect
modernc.org/ccgo/v3 v3.16.13 // indirect
modernc.org/libc v1.22.2 // indirect
modernc.org/mathutil v1.5.0 // indirect
modernc.org/memory v1.4.0 // indirect
modernc.org/opt v0.1.3 // indirect
modernc.org/sqlite v1.20.3 // indirect
modernc.org/strutil v1.1.3 // indirect
modernc.org/token v1.0.1 // indirect
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
modernc.org/libc v1.49.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
modernc.org/sqlite v1.29.10 // indirect
modernc.org/strutil v1.2.0 // indirect
modernc.org/token v1.1.0 // indirect
sigs.k8s.io/yaml v1.2.0 // indirect
)

// We want to format raw bytes as hex instead of base64. The forked version
// allows us to specify that as an option.
replace google.golang.org/protobuf => github.com/lightninglabs/protobuf-go-hex-display v1.30.0-hex-display

go 1.21
go 1.21.4
Loading
Loading