Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gnmic] gRPC dialout receive error #519

Open
mnorman111 opened this issue Sep 13, 2024 · 2 comments
Open

[gnmic] gRPC dialout receive error #519

mnorman111 opened this issue Sep 13, 2024 · 2 comments

Comments

@mnorman111
Copy link

Hi!

I'm geting errors
[gnmic] gRPC dialout receive error: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: proto: cannot parse invalid wire-format data
this seems to cause gnmic to eat all available memory and become unresponsive. Happens about once a day.

@karimra
Copy link
Collaborator

karimra commented Sep 13, 2024

Can you share some details on your config, setup and what you are trying to achieve ?

@mnorman111
Copy link
Author

gnmic is receiving telemetry data from about 500 Nokia routers and inserts it to influxdb2
typical memory usage
Top 3 processes by memory consumption:
PID COMMAND %MEM
187366 gnmic 23.0
936 influxd 22.4

occasionally
Top 3 processes by memory consumption:
PID COMMAND %MEM
187366 gnmic 92.3
936 influxd 2.0

and then it stops receiving any data. when it happens there is log
2024/09/16 12:21:43.872233 [gnmic] gRPC dialout receive error: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: proto: cannot parse invalid wire-format data

gnmic config:

global:
address: 0.0.0.0:57400
log-file: /var/log/gnmic.log
gnmi-server:

the address the gNMI server will listen to

address: 0.0.0.0:57400
insecure: true
max-subscriptions: 512
max-unary-rpc: 512
min-sample-interval: 1ms
default-sample-interval: 60s
min-heartbeat-interval: 1s
enable-metrics: true
debug: false

cache:

type: oc

address:

username:

password:

expiration: 60s

debug: false

max-bytes:

max-msgs-per-subscription:

fetch-batch-size:

fetch-wait-time:

outputs:
influx-live:
type: influxdb
url: http://localhost:8086
org: Telemetry
bucket: MPLS-telem
token: xxxxxx
batch-size: 1000
flush-timer: 10s
use-gzip: false
enable-tls: false
tls:
ca-file:
cert-file:
key-file:
skip-verify: false
override-timestamps: false
timestamp-precision: ns
health-check-period: 30s
debug: false
add-target: "overwrite"
target-template: '{{ index . "subscription-name" }}'
enable-metrics: false
event-processors:
- delete-subscription-name
- add-measurement-as-subscription-name

node config:
destination-group "telem" create
allow-unsecure-connection
tcp-keepalive
shutdown
exit
destination 10.255.10.11 port 57400 create
exit
exit
sensor-groups
sensor-group "stats" create
path "/state/lag/port-scheduler-policy/statistics" create
exit
path "/state/lag/statistics" create
exit
path "/state/port[port-id=]/egress/voq" create
exit
path "/state/port[port-id=
]/statistics" create
exit
path "/state/port[port-id=*]/transceiver" create
exit
path "/state/router/interface" create
exit
path "/state/service/epipe" create
exit
path "/state/service/vpls" create
exit
path "/state/service/vprn/interface" create
exit
path "/state/system/ptp/instance/current-ds/mean-path-delay" create
exit
path "/state/system/ptp/instance/current-ds/offset-from-master" create
exit
exit
exit
persistent-subscriptions
subscription "aasa-ixreb1" create
destination-group "telem"
encoding proto
mode sample
sample-interval 60000
sensor-group "stats"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants