Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nfdump with geodb, nfprofile and nfsen - profile issue #557

Closed
Takalele opened this issue Aug 19, 2024 · 7 comments
Closed

nfdump with geodb, nfprofile and nfsen - profile issue #557

Takalele opened this issue Aug 19, 2024 · 7 comments
Assignees
@phaag
Labels
Feature request Feature request

Comments

@Takalele
Copy link
Contributor

Hi,

I'm having trouble getting the GeoDB feature working with profiles in NfSen. I have added the path in the /usr/local/etc/nfdump.conf file:

geodb.path = "/data/nfsen/maxmind/mmc.nf"

The directory looks like this:

ll /data/nfsen/maxmind/
total 9228
drwxr-xr-x 2 netflow www-data    4096 Aug 18 17:25 ./
drwxr-xr-x 1 root    root        4096 Aug 19 22:38 ../
-rw-r--r-- 1 netflow www-data 9441170 Aug 18 17:24 mmc.nf

In the live profile, I can see that enrichment is working:

** nfdump -M /data/nfsen/profiles-data/live/mjolnir  -T  -r 2024/08/19/nfcapd.202408192230 -o 'fmt:fmt -o predefined %ts %td %pr %sap -> %dap %flg %tos %pkt %byt %pps %bps %bpp %fl %sas %sasn %das %dasn' -c 20
nfdump filter:
dst as 3598 or dst as 5761 or dst as 6182 or dst
as 6194 or dst as 6291 or dst as 6584 or dst as
8068 or dst as 12076 or dst as 13399 or dst as
13811 or dst as 14719 or dst as 17345 or
dst as 20046 or dst as 22692 or dst as 23468 or
dst as 25796 or dst as 26222 or dst as 30135 or
dst as 30575 or dst as 31792 or dst as 32476 or
dst as 36006 or dst as 40066 or dst as 63245 or
dst as 63314 or dst as 95496 or dst as 395524 or
dst as 395851 or dst as 396463 or dst as 397096 or
dst as 397466 or dst as 397996 or dst as 398575 or
dst as 398656 or dst as 398961 or dst as 400572 or
dst as 400884
                  Date first seen         Duration         Proto      Src IP Addr:Port          Dst IP Addr:Port     Flags Tos   Packets    Bytes      pps      bps    Bpp Flows Src AS Src AS organisation Dst AS Dst AS organisation
fmt -o predefined 2024-09-08 20:18:49.992     00:00:00.005 TCP          10.8.2.82:49320 ->      13.107.5.93:443   ........    0       13     3205     2600    5.1 M    246     1      0    8068 MICROSOFT-CORP-MSN-AS-BLOCK
fmt -o predefined 2024-09-08 20:19:19.962     00:00:00.005 TCP          10.8.2.82:49320 ->      13.107.5.93:443   ........    0        0        0        0        0      0     1      0    8068 MICROSOFT-CORP-MSN-AS-BLOCK
fmt -o predefined 2024-09-08 20:19:34.947     00:00:00.005 TCP          10.8.2.82:49320 ->      13.107.5.93:443   ........    0        0        0        0        0      0     1      0    8068 MICROSOFT-CORP-MSN-AS-BLOCK
fmt -o predefined 2024-09-08 20:19:49.932     00:00:00.005 TCP          10.8.2.82:49320 ->      13.107.5.93:443   ........    0        0        0        0        0      0     1      0    8068 MICROSOFT-CORP-MSN-AS-BLOCK
Summary: total flows: 4, total bytes: 3205, total packets: 13, avg bps: 427, avg pps: 0, avg bpp: 246
Time window: 2024-09-08 20:14:20 - 2024-09-08 20:20:50
Total flows processed: 18405, passed: 4, Blocks skipped: 0, Bytes read: 1546212
Sys: 0.0468s User: 0.0907s Wall: 0.0081s flows/second: 2267740.8 Runtime: 0.0081s

However, my custom profiles with AS filters are not working. The nfcapd files are all empty:

** nfdump -M /data/nfsen/profiles-data/CDNs/Microsoft:Amazon:Netflix:Akamai:Google  -T  -R 2024/08/19/nfcapd.202408192120:2024/08/19/nfcapd.202408192340 -o 'fmt:fmt -o predefined %ts %td %pr %sap -> %dap %flg %tos %pkt %byt %pps %bps %bpp %fl %sas %sasn %das %dasn' -c 20
nfdump filter:
any
                  Date first seen         Duration         Proto      Src IP Addr:Port          Dst IP Addr:Port     Flags Tos   Packets    Bytes      pps      bps    Bpp Flows Src AS Src AS organisation Dst AS Dst AS organisation
No matching flows
Summary: total flows: 0, total bytes: 0, total packets: 0, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: 
Total flows processed: 0, passed: 0, Blocks skipped: 0, Bytes read: 4640
Sys: 0.0431s User: 0.0833s Wall: 0.0086s flows/second: 0.0 Runtime: 0.0086s

And here's the directory structure for the custom profiles-data:

ll /data/nfsen/profiles-data/CDNs/*/2024/08/19/
/data/nfsen/profiles-data/CDNs/Akamai/2024/08/19/:
total 152
drwxr-xr-x 2 netflow www-data 4096 Aug 19 23:50 ./
drwxr-xr-x 3 netflow www-data 4096 Aug 19 20:55 ../
-rw-r--r-- 1 netflow www-data  123 Aug 19 20:55 nfcapd.202408192050
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:00 nfcapd.202408192055
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:05 nfcapd.202408192100
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:10 nfcapd.202408192105
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:15 nfcapd.202408192110
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:20 nfcapd.202408192115
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:25 nfcapd.202408192120
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:30 nfcapd.202408192125
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:35 nfcapd.202408192130
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:40 nfcapd.202408192135
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:45 nfcapd.202408192140
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:50 nfcapd.202408192145
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:55 nfcapd.202408192150
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:00 nfcapd.202408192155
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:05 nfcapd.202408192200
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:10 nfcapd.202408192205
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:15 nfcapd.202408192210
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:20 nfcapd.202408192215
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:25 nfcapd.202408192220
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:30 nfcapd.202408192225
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:36 nfcapd.202408192230
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:40 nfcapd.202408192235
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:45 nfcapd.202408192240
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:50 nfcapd.202408192245
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:55 nfcapd.202408192250
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:00 nfcapd.202408192255
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:05 nfcapd.202408192300
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:10 nfcapd.202408192305
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:15 nfcapd.202408192310
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:20 nfcapd.202408192315
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:25 nfcapd.202408192320
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:30 nfcapd.202408192325
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:35 nfcapd.202408192330
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:43 nfcapd.202408192335
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:45 nfcapd.202408192340
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:50 nfcapd.202408192345

/data/nfsen/profiles-data/CDNs/Amazon/2024/08/19/:
total 152
drwxr-xr-x 2 netflow www-data 4096 Aug 19 23:50 ./
drwxr-xr-x 3 netflow www-data 4096 Aug 19 20:55 ../
-rw-r--r-- 1 netflow www-data  123 Aug 19 20:55 nfcapd.202408192050
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:00 nfcapd.202408192055
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:05 nfcapd.202408192100
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:10 nfcapd.202408192105
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:15 nfcapd.202408192110
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:20 nfcapd.202408192115
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:25 nfcapd.202408192120
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:30 nfcapd.202408192125
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:35 nfcapd.202408192130
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:40 nfcapd.202408192135
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:45 nfcapd.202408192140
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:50 nfcapd.202408192145
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:55 nfcapd.202408192150
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:00 nfcapd.202408192155
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:05 nfcapd.202408192200
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:10 nfcapd.202408192205
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:15 nfcapd.202408192210
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:20 nfcapd.202408192215
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:25 nfcapd.202408192220
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:30 nfcapd.202408192225
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:36 nfcapd.202408192230
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:40 nfcapd.202408192235
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:45 nfcapd.202408192240
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:50 nfcapd.202408192245
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:55 nfcapd.202408192250
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:00 nfcapd.202408192255
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:05 nfcapd.202408192300
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:10 nfcapd.202408192305
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:15 nfcapd.202408192310
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:20 nfcapd.202408192315
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:25 nfcapd.202408192320
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:30 nfcapd.202408192325
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:35 nfcapd.202408192330
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:43 nfcapd.202408192335
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:45 nfcapd.202408192340
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:50 nfcapd.202408192345

/data/nfsen/profiles-data/CDNs/Google/2024/08/19/:
total 152
drwxr-xr-x 2 netflow www-data 4096 Aug 19 23:50 ./
drwxr-xr-x 3 netflow www-data 4096 Aug 19 20:55 ../
-rw-r--r-- 1 netflow www-data  123 Aug 19 20:55 nfcapd.202408192050
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:00 nfcapd.202408192055
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:05 nfcapd.202408192100
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:10 nfcapd.202408192105
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:15 nfcapd.202408192110
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:20 nfcapd.202408192115
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:25 nfcapd.202408192120
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:30 nfcapd.202408192125
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:35 nfcapd.202408192130
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:40 nfcapd.202408192135
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:45 nfcapd.202408192140
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:50 nfcapd.202408192145
-rw-r--r-- 1 netflow www-data  123 Aug 19 21:55 nfcapd.202408192150
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:00 nfcapd.202408192155
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:05 nfcapd.202408192200
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:10 nfcapd.202408192205
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:15 nfcapd.202408192210
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:20 nfcapd.202408192215
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:25 nfcapd.202408192220
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:30 nfcapd.202408192225
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:36 nfcapd.202408192230
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:40 nfcapd.202408192235
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:45 nfcapd.202408192240
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:50 nfcapd.202408192245
-rw-r--r-- 1 netflow www-data  123 Aug 19 22:55 nfcapd.202408192250
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:00 nfcapd.202408192255
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:05 nfcapd.202408192300
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:10 nfcapd.202408192305
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:15 nfcapd.202408192310
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:20 nfcapd.202408192315
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:25 nfcapd.202408192320
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:30 nfcapd.202408192325
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:35 nfcapd.202408192330
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:43 nfcapd.202408192335
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:45 nfcapd.202408192340
-rw-r--r-- 1 netflow www-data  123 Aug 19 23:50 nfcapd.202408192345

/data/nfsen/profiles-data/CDNs/Microsoft/2024/08/19/:
total 132
drwxr-xr-x 2 netflow www-data 4096 Aug 19 23:50 ./
drwxr-xr-x 3 netflow www-data 4096 Aug 19 21:20 ../
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:20 nfcapd.202408192115
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:25 nfcapd.202408192120
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:30 nfcapd.202408192125
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:35 nfcapd.202408192130
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:40 nfcapd.202408192135
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:45 nfcapd.202408192140
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:50 nfcapd.202408192145
-rw-r--r-- 1 netflow www-data  126 Aug 19 21:55 nfcapd.202408192150
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:00 nfcapd.202408192155
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:05 nfcapd.202408192200
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:10 nfcapd.202408192205
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:15 nfcapd.202408192210
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:20 nfcapd.202408192215
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:25 nfcapd.202408192220
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:30 nfcapd.202408192225
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:36 nfcapd.202408192230
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:40 nfcapd.202408192235
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:45 nfcapd.202408192240
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:50 nfcapd.202408192245
-rw-r--r-- 1 netflow www-data  126 Aug 19 22:55 nfcapd.202408192250
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:00 nfcapd.202408192255
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:05 nfcapd.202408192300
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:10 nfcapd.202408192305
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:15 nfcapd.202408192310
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:20 nfcapd.202408192315
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:25 nfcapd.202408192320
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:30 nfcapd.202408192325
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:35 nfcapd.202408192330
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:43 nfcapd.202408192335
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:45 nfcapd.202408192340
-rw-r--r-- 1 netflow www-data  126 Aug 19 23:50 nfcapd.202408192345

/data/nfsen/profiles-data/CDNs/Netflix/2024/08/19/:
total 152
drwxr-xr-x 2 netflow www-data 4096 Aug 19 23:50 ./
drwxr-xr-x 3 netflow www-data 4096 Aug 19 20:55 ../
-rw-r--r-- 1 netflow www-data  124 Aug 19 20:55 nfcapd.202408192050
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:00 nfcapd.202408192055
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:05 nfcapd.202408192100
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:10 nfcapd.202408192105
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:15 nfcapd.202408192110
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:20 nfcapd.202408192115
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:25 nfcapd.202408192120
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:30 nfcapd.202408192125
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:35 nfcapd.202408192130
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:40 nfcapd.202408192135
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:45 nfcapd.202408192140
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:50 nfcapd.202408192145
-rw-r--r-- 1 netflow www-data  124 Aug 19 21:55 nfcapd.202408192150
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:00 nfcapd.202408192155
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:05 nfcapd.202408192200
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:10 nfcapd.202408192205
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:15 nfcapd.202408192210
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:20 nfcapd.202408192215
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:25 nfcapd.202408192220
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:30 nfcapd.202408192225
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:36 nfcapd.202408192230
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:40 nfcapd.202408192235
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:45 nfcapd.202408192240
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:50 nfcapd.202408192245
-rw-r--r-- 1 netflow www-data  124 Aug 19 22:55 nfcapd.202408192250
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:00 nfcapd.202408192255
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:05 nfcapd.202408192300
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:10 nfcapd.202408192305
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:15 nfcapd.202408192310
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:20 nfcapd.202408192315
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:25 nfcapd.202408192320
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:30 nfcapd.202408192325
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:35 nfcapd.202408192330
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:43 nfcapd.202408192335
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:45 nfcapd.202408192340
-rw-r--r-- 1 netflow www-data  124 Aug 19 23:50 nfcapd.202408192345

logs:

nfcapd[43657]: Ident: 'mjolnir' Flows: 17361, Packets: 67546, Bytes: 12444663, Sequence Errors: 2, Bad Packets: 0, Blocks: 0
nfsen[43663]: Signal 'start-periodic'
nfsen[43663]: Run periodic at Mon Aug 19 23:55:00 2024
nfsen[43663]: Prepare profiling './CDNs'
nfsen[43663]: Prepare profiling './live'
nfsen[43663]: 5 channels/alerts to profile
nfsen[43664]: comm child[45385] terminated with no exit value
nfsen[45386]: profile opts: .#CDNs#2#Google#mjolnir for profiler 0
nfsen[45386]: profile opts: .#CDNs#2#Microsoft#mjolnir for profiler 0
nfsen[45386]: profiler 0 started
nfsen[45387]: profile opts: .#CDNs#2#Amazon#mjolnir for profiler 1
nfsen[45387]: profile opts: .#CDNs#2#Netflix#mjolnir for profiler 1
nfsen[45387]: profile opts: .#CDNs#2#Akamai#mjolnir for profiler 1
nfsen[45387]: profiler 1 started
nfprofile[45389]: Process line '.#CDNs#2#Amazon#mjolnir'
nfprofile[45388]: Process line '.#CDNs#2#Google#mjolnir'
nfprofile[45389]: Process line '.#CDNs#2#Netflix#mjolnir'
nfprofile[45388]: Process line '.#CDNs#2#Microsoft#mjolnir'
nfprofile[45389]: Process line '.#CDNs#2#Akamai#mjolnir'
nfprofile[45388]: Setup channel 'Google' in profile 'CDNs' group '.', channellist 'mjolnir'
nfprofile[45389]: Setup channel 'Amazon' in profile 'CDNs' group '.', channellist 'mjolnir'
nfprofile[45389]: Setup channel 'Netflix' in profile 'CDNs' group '.', channellist 'mjolnir'
nfprofile[45388]: Setup channel 'Microsoft' in profile 'CDNs' group '.', channellist 'mjolnir'
nfprofile[45389]: Setup channel 'Akamai' in profile 'CDNs' group '.', channellist 'mjolnir'
nfsen[45386]: profiler 0 finished
nfsen[45387]: profiler 1 finished
nfsen[43663]: Update profile CDNs in group .
nfsen[43663]: Add .:CDNs:202408192350 for plugin processing
nfsen[43663]: Update profile live in group .
nfsen[43663]: Add .:live:202408192350 for plugin processing
nfsen[43663]: Expire forked
nfsen[45418]: expire child
nfsen[43663]: Run plugins for 202408192350
nfsen[45418]: Run expire at Mon Aug 19 23:55:00 2024
nfsen[43664]: connection on UNIX socket
nfsen[45418]: Expire has 228s in this slot!
nfsen[45418]: Expire profile CDNs group . low water mark: 90%
nfsen[43664]: comm server started: 45419
nfsen[45419]: Cmd Decode: run-plugins
nfsen[45419]: Plugin Cycle: ., CDNs, 202408192350
nfsen[45419]: Plugin Cycle: ., live, 202408192350
nfsen[45419]: Cmd Decode: quit
nfsen[43663]: Run plugins done.
nfsen[43663]: Check alerts for Mon Aug 19 23:50:00 2024
nfsen[43663]: Check alerts done.
nfsen[43663]: wait for expire child
nfsen[45418]: nfexpire: Expired files:      0
nfsen[45418]: nfexpire: Expired file size:  0 B
nfsen[45418]: nfexpire: Expired time range: 0 sec
nfsen[45418]: nfexpire:
nfsen[45418]: Expire has 228s in this slot!
nfsen[45418]: Expire profile live group . low water mark: 90%
nfsen[43664]: comm child[45419] terminated with no exit value
nfsen[45418]: nfexpire: Include nfcapd bookkeeping record in /data/nfsen/profiles-data/live/mjolnir
nfsen[45418]: nfexpire: Expired files:      0
nfsen[45418]: nfexpire: Expired file size:  0 B
nfsen[45418]: nfexpire: Expired time range: 0 sec
nfsen[45418]: nfexpire:
nfsen[45418]: End expire at Mon Aug 19 23:55:00 2024
nfsen[45418]: expire child done
nfsen[43663]: Expire child terminated
nfsen[43664]: connection on UNIX socket
nfsen[43664]: comm server started: 45422
nfsen[45422]: Cmd Decode: signal
nfsen[45422]: Cmd Decode: quit
nfsen[45422]: Cleanup Routine
nfsen[43663]: Signal 'end-periodic'
nfsen[43664]: comm child[45422] terminated with no exit value

Screenshots:
nfsen_cdn_overview
nfsen_cdn_detail

nfsen_cdn_stats
nfsen_cdn_stats_source_ms

Should this work with GeoDB enrichment in NfProfiles? If so, do you have any idea what might be going wrong?

Systemspecs: Ubuntu 24.04 (container)
nfdump: 1.7.4 with sflow nfpcapd maxmind nsel nfprofile nftrack ja4 readpcap ftconv
nfSen: 1.3.10, nfdump: 7
php: 8.3.6
apache: 2.4.58
Netflow exporter: freertr with dpdk dataplane

Thank you for the excellent work you're doing!

BR
Takalele
@phaag
Copy link
Owner

phaag commented Aug 20, 2024

Yes - nfprofile does not use the geoDB, as the original design was the enrichment of the nfdump output. The additional purpose to use it for AS filtering was implemented later. The same is true for the torDB. ip tor returns no flows.
I will check, if I can implement that for the next release.

@Takalele
Copy link
Contributor Author

I will check, if I can implement that for the next release.

that sounds great! If you need a tester, I’m happy to help.

thank you!

@brownej
Copy link

brownej commented Aug 20, 2024

I was wondering about profiling tor. Thanks for the confirmation that ip tor returns no flows for a profile.

I too can see some use cases around geo and tor data in profiles and maybe even augmenting data at capture. This seems like a good topic for discussion and I'm sure you, Peter, have spent some time thinking about it. I'd like to discuss it further but a PR doesn't seem like the right place.

phaag added a commit that referenced this issue Aug 20, 2024
@phaag
Implement maxmind and tor lookup filter in nfprofile. See #557
813da06
@phaag
Copy link
Owner

phaag commented Aug 20, 2024 •
edited
Loading

I implemented a first poc for geo and tor filtering in nfprofile. Please make sure you have the DBs configured properly in nfdump.conf, otherwise nfprofile does not find the DBs.
@Takalele could you please test, if it works?

@phaag
Copy link
Owner

phaag commented Aug 20, 2024

@brownej Feel free to open a new issue to discuss this topic.

@Takalele
Copy link
Contributor Author

@phaag the poc code works for both db's. Initially, I encountered an issue with torlookup, where the script was unable to download more than one month of data. I have submitted a fix in PR #558 to resolve this.

btw that was blazing fast, amazing. thank you very much.

tor:
image

geodb:
image

@phaag
Copy link
Owner

phaag commented Aug 21, 2024

Nice to hear it works!

@phaag phaag closed this as completed Aug 21, 2024
@phaag phaag self-assigned this Aug 21, 2024
@phaag phaag added the Feature request Feature request label Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phaag phaag

Labels
Feature request Feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@phaag @Takalele @brownej