How to fix so-steno missing?

[lonelyadmin]

Version

2.4.111

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

2

RAM

4

Storage for /

65G 54G available

Storage for /nsm

425G 45G available

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Status is showing so-steno is missing. I can't seem to pinpoint what is causing it. Other forward nodes configured and deployed identically (AFAIK) are not having any issues. What should I look at next?

Looking at /opt/so/log/stenographer/stenographer.log shows a panic:

2025/01/06 17:20:39 Blockfile opening: "/tmp/stenographer2599814940/PKT0/1730483233099119"
2025/01/06 17:20:39 opening index "/tmp/stenographer2599814940/IDX0/1730483233099119"
2025/01/06 17:20:39 new blockfile "/tmp/stenographer2599814940/PKT0/1730483233099119"
2025/01/06 17:20:39 Blockfile opening: "/tmp/stenographer2599814940/PKT0/1730483294100141"
2025/01/06 17:20:39 opening index "/tmp/stenographer2599814940/IDX0/1730483294100141"
panic: runtime error: slice bounds out of range [:4111] with capacity 2337

goroutine 21 [running]:
github.com/golang/leveldb/table.(*blockIter).Next(0xc00034d920)
        /home/mreeves/go/pkg/mod/github.com/golang/[email protected]/table/reader.go:125 +0x48a
github.com/golang/leveldb/table.block.seek({0xc000390000, 0x13f8?, 0x13f8?}, {0xbd4f80, 0x15a4620}, {0xc0000140a0, 0x1, 0x1})
        /home/mreeves/go/pkg/mod/github.com/golang/[email protected]/table/reader.go:85 +0x2d9
github.com/golang/leveldb/table.(*Reader).find(0xc0002c3ae0, {0xc0000140a0, 0x1, 0x1}, 0xc0000140a0?, 0x0?)
        /home/mreeves/go/pkg/mod/github.com/golang/[email protected]/table/reader.go:369 +0xb3
github.com/golang/leveldb/table.(*Reader).Get(0xbd90b8?, {0xc0000140a0, 0x1, 0xc0000428c0?}, 0x31?)
        /home/mreeves/go/pkg/mod/github.com/golang/[email protected]/table/reader.go:337 +0x4b
github.com/google/stenographer/indexfile.NewIndexFile({0xc0000a1840, 0x31}, 0xc00025d170)
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/indexfile/indexfile.go:67 +0x1df
github.com/google/stenographer/blockfile.NewBlockFile({0xc0000a17c0, 0x31}, 0xc00025d170)
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/blockfile/blockfile.go:63 +0xec
github.com/google/stenographer/thread.(*Thread).trackNewFile(0xc0001cae60, {0xc0000a0821, 0x10})
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/thread/thread.go:172 +0x8a
github.com/google/stenographer/thread.(*Thread).syncFilesWithDisk(0xc0001cae60)
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/thread/thread.go:139 +0x176
github.com/google/stenographer/thread.(*Thread).SyncFiles(0xc0001cae60)
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/thread/thread.go:363 +0x30
github.com/google/stenographer/env.(*Env).syncFiles(...)
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/env/env.go:267
github.com/google/stenographer/env.(*Env).callEvery(0xc000188540, 0xc000103b30, 0x0?)
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/env/env.go:180 +0x73
created by github.com/google/stenographer/env.New in goroutine 1
        /home/mreeves/go/pkg/mod/github.com/google/[email protected]/env/env.go:131 +0x3f8
2025-01-06T17:20:39.350447Z T:2ee496 [stenotype.cc:437] Got signal 15, stopping threads

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[rh-ops]

Try restarting steno with the command so-pcap-restart

[lonelyadmin]

Here is the response I get:

=========================================================================
Restarting steno...

This could take a while if another Salt job is running. 
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
Error response from daemon: No such container: so-steno

[rh-ops]

Okay so what happened is you likely have a corrupt pcap index. So try running the following commands in the given order:

sudo mv /nsm/pcapindex /nsm/pcapindex.old

rm /nsm/pcapindex.old (if pcapindex.old already exists)

sudo so-pcap-restart

sudo salt-call state.apply pcap

Should note that the pcapindex.old folder will eventually need to be deleted to regain storage space.