v1.7.5

nfdump-1.7.5

Although it's only minor increase in the version number, nfdump, git a lot of code refresh and changes since 1.7.4.

There are too many changes to report them all here, so please consult the ChangeLog file.
The most notable changes:

• Improved parallel filtering to improve performance of nfdump.
• Parallel processing of reading, filtering and processing data.
• add ndjson output format
• Speedup output
• Add ja4 processing
• Officially integrate torlookup into nfdump.
• Add support for NOKIA enterprise
• Lot of code cleanup.
• Bug fixes.

For the detailed list, see the ChangLog file.

nfdump-1.7.4

nfdump-1.7.4 contains various bug fixes and stability fixes. Lot of old code got cleaned up. Further improvements are planned.

• 4f409e9 2024-02-17 (HEAD -> master, origin/master, origin/HEAD) Release v1.7.4
• 4a7de41 2024-02-17 Fix compiler warnings for lz4
• 4e98a35 2024-02-17 Update lz4 code
• 7997ac4 2024-02-10 Replace Changelog file with git log history. Modify gen_version.sh. See #482
• 0001529 2024-02-10 Honor printPlain Flag for String_evt. See #498
• 9db4215 2024-02-03 Fix buggy sflow code. Fixes #506
• a20249c 2024-02-03 Dissolve code for output
• 56919df 2024-02-03 Change link order as some Linux linker do not correctly find static symbols. Fix #505
• 1b6cdd5 2024-01-29 Fix some output_fmt format issues
• 627565a 2024-01-29 Remove local m4 files
• f0d2750 2024-01-29 Fix err var in nfprofile
• dc1b1b2 2024-01-29 Fix #503 - Append records in nfprofile
• e325de6 2024-01-22 Implement #501. Add -X option for sfcapd
• 4b961bd 2024-01-14 Fix #498 - event string in output_fmt
• 389acc5 2024-01-06 Remove NSEL option from nfanon. NSEL included by default
• d49a050 2024-01-06 Fix error message
• b78ffba 2024-01-06 Refrag code into userio files. Prepare for next level code update
• 05430dd 2024-01-06 Move code. Cleanup
• 272c235 2024-01-03 Fix Offset/Size values in nfxV3.h
• 5c19ff2 2023-12-31 Fix runtest with new field IPfrag flags
• 3981646 2023-12-27 Add IP fragmentflags tag #197
• 71d0afc 2023-12-27 Add cgNAT decoding to IPFIX decoder
• b6841f6 2023-12-27 Fix #496 - Add RFC8158 NAT event flags
• dec1d92 2023-12-16 Update Offset/Size for header fields
• 305f1be 2023-12-15 Merge pull request #489 from g0tar/keep_flowStartMilliseconds
• aaaee7e 2023-12-13 Don't clear flowStartMilliseconds when flowEndMilliseconds==0
• 93797b2 2023-12-05 Fixed #486. Pull request #487
• 2fd5df7 2023-12-03 Cleanup grammar.y with useless rule
• 295f514 2023-12-03 Implement request #485 in sfcapd to store nat IP/port for sflow records if available
• 042e68b 2023-12-02 Cleanup filter. String arguments in filter elements may be single or double quoted. Strings without quotes are accepted as strings as long as the string is not a reserved fil
  ter key word.#
• 71f888f 2023-11-26 Update nfdump.spec. Thanks rexy74!
• 40eb5c9 2023-11-26 Fix grammar
• eaf96ce 2023-11-26 More fixes for #484
• df50860 2023-11-26 Fix #484 - workaround not really a fix
• 7f75c90 2023-11-18 Fix #481 - expire.c
• 469dd81 2023-11-18 Merge branch 'master' of github.com:phaag/nfdump sync local
• cc1073a 2023-11-18 Cleanup flist code
• cbc617f 2023-11-01 Fix #480 Build error on CentOS
• fded574 2023-10-19 Merge pull request #477 from chadf/configure-disable-xxxx
• 622670b 2023-10-16 Fixed remaining ./configure --disable-xxxx issues. Corrected use of $enabled_xxxx vs $build_xxxx where appropriete.
• ce7ff2c 2023-10-15 Fix ./configure --disable-xxxx handling.
• e2b9f43 2023-10-15 Add multiple dir wildcard '@' for multiple dir argument -M
• 6b69002 2023-10-14 Fix all HAVE_ZSTD macros
• 3ac05de 2023-10-14 Repair version string
• ec8b602 2023-10-14 Check time window, if given, of a multi file selection
• 94743ee 2023-10-14 Fix compiler warning in kbtree.h
• 286d70d 2023-10-14 Update Readme
• 3e4a992 2023-10-14 Improve build process for external compression libraries. Make bzip library optional
• 9a50550 2023-10-08 Fix #474. Reject unspecified IP addresses and invalid prefix length in filter
• 46d2db0 2023-09-25 Fix #469. Update usage and error messages and man pages
• 02ee700 2023-09-25 Fix #467. Rework number of workers setting. Set max to 64

nfdump-1.7.3

Nfdump-1.7.3 adds zstd compression speeds compression (multi threading) and adds compression levels.
Furthermore it fixes some nfprofile issues.

Other changes:

5e2aef9: Improve error handling in privsep.c
584453e: Fix #459. change vars to uint64_t
696f563: Merge branch 'master' of github.com:phaag/nfdump
ec1f121: Allow direction status > 2, althogh not defined
c2c097f: Fix #452 - to add GeoInfo
1163722: Merge branch 'master' of github.com:phaag/nfdump
c51ce1f: Fix #448 man page type
de9811c: Cleanup unused extensions. Implemented as arrays. Collect Juniper inline-monitorig packet data. Data is not yet decoded.
3a810ad: Add types for Juniper inline-monitoring. No decoding yet
130fafe: Update test sequence for compression levels
14be8fb: Cleanup nffile struct
f9515d2: Fix potential compressed appendix corruption
4b7c42a: Update Readme for building instructions on older Ubuntu 18.04 LTS
837419b: Fix broken installation of nfdump.conf.dist
df4fc16: Cleanup all compress files into sub dir
6c47de5: Move conf subdir to nfdump library. Fix linker errors
18f7bc5: Add maxworkers to nfdump.conf
336cb05: Make count check more dynamic in v5
1a71629: Fix nfsen #15 on RH
65ea877: Bark if rename fails
8802fa0: Add zstd to QueryFile()
98385f2: Cleanup and fix minor bugs
91aeeb8: Enable multiple writer threads, depending on number of Cores online
a9a4322: Update Readme
92f8d01: Update Readme
2efd259: Add zstd compression
cb254b1: Enable lz4 compression levels 3..12
3b7b92f: Update lz4 code
8123283: Add code for compression level handling
c6fedc0: Update compression code
c8c98d0: Add compresson level parsing
7413188: Fix parsing compression option to work for all possible options
12a90b4: Adapt -J option in nfdump to new compression spec, being still compatible with old one.
a1f4f19: Print usage() if no arguments are given
d8620c0: Update man pages
e928c42: Unify compression arguments. Use now -z= for compression. Accept still old -j and -y for now
e8ee2fc: Remove static var for lzo wrkmem
f2ddf67: Cleanup nfprofile and fix issue #398
69e4d98: Add syslog by default for nfprofile
7041498: Add file verification for nfprofile
efd28ad: Cleanup nfprofile code
b54e266: Make geolookup more robust reading buggy files
d90d561: Add flush() for nfprofile
399014d: Move from c11 to c17 compile standard
7b791a1: Cleanup profile code
27d65be: Prevent loading geoDB if not needed
f3a174f: change var name
2c5678f: Add BlockReports
18a34c1: Update nfprofile
43f3f14: Fix last commit
881707e: Remove blockqueue. Reduce memory footprint
e10c2ef: Code cleanup
af0f43e: Code cleanup
2944ac9: Make consistent protocol ID for nfd raw
0742648: Code cleanup and minor bug fix
328daaf: Rework QueryFile code and cleanup
058cecb: Merge and cleanup nfpcapd and nfreplay code for sending raw records
2864358: Add nfdump protocol for nfreplay

nfdump-1.7.2

Nfdump-1.7.2 removes old code, and got some new features. It fixes a few bugs and improves stability.
As of nfdump-1.7.2, it is now the recommended version in production.

For the full list of changes, see the ChangeLog file.

Some highlights:

• Update nfcapd/sfcapd man pages for erspan data link
• Implement erspan protocol in nfpcapd packet processing
• Sync ipfix and netflow_v9 for option field processing
• Handle buggy option data(!) from exporter
• Fixes compile warnings for Linux(es) and *BSDs.
• Fix v9 option template processing
• Fix option record processing for multiple records
• Fix ipfix nbar processing
• Update pcap reader with erspan device
• Fix fmt argument parsing
• Add IPv4 in IPv6 special cases in geolookup
• Fix IP version check in geolookup
• Disable signals in threads. Different OS handle signals differently
• Recongnize old sampler with missing algorithm tag #35
• Fix memory leaks
• Update pcap_reader and add CISCO application performance monitor tags
• Add ERSPAN decoding to pcap_reader
• Check uid root for src spoofing option
• Fix valgrind error for uninitalised memory. #431
• Update lz4 code
• Update number of enabled tags in v9 and ipfix
• Add collector option -X to limit stored data.
• Implement %sasn, %dasn organisation name printing tags. See #430
• Improve geoDB handling. Needs rebuild of the geoDB.
• Wire sampling extension for individual sampling
• Update ICMP type/code handling. Issue #423
• Codespell - cleanup
• Fix #415. ICMP decoding in netflow v9
• Pimp pcap_reader
• Added source address support for nfreplay
• Fix dst tos bug in output_raw
• Add geo info in JSON output when using GeoDB. #402
• Fix 408. Interchanged time stamps
• Add icmp type/code elements 176, 177, 178, 179
• Add unique version string
• Add OpenBSD pflog format in nfpcapd
• Add Linux nflog in nfpcapd
• Replace old packet repeater code with more modern privsep code
• Replace old launcher startup code with more modern privsep code
• Fix compile issue #395 on Alpine Linux

nfdump-1.7.1

This release fixes mostly bugs from 1.7.0 and is now the recommended version for production. It works together with well advanced in years NfSen 1.3.9 https://github.com/phaag/nfsen.

Changelog:

• Fix #394. Event labeling
• Implement #393 consistent logging
• Remove extra ':' in getopt of nfcapd
• Add feature #391. Add country code aggregation
• Fix #392. Fix format options with IPv6
• Implement #390. Aggregation for GeoDB's enriched AS data
• Add OpenBSD pflog decoding in nfpcapd and nfdump
• Fix Ident change
• Sync nfcapd, sfcapd code
• Fix #389 receiving IPv4 on IPv6 socket in sfcapd
• Fix #385 bug when compiled on i386 arch - 32bit alignment
• Fix #384 bug when compile with --enable-nsel
• Implement #366 Linux NFLOG link layer protocol in nfpcapd
• Fix #381 pcap overwrite in nfpcapd fixed
• Fix #380 nbar string validation
• Implement #377. Rework sampling code in general. Switch to packet interval/space notation. Map older sampling to new notation.
• Fix #375 relative timestamps with sysUptime id 160
• Rework nbar code. Use new array records and fix nbar bug in older versions.
• Fix #370. Help shows correct option -A
• Fix #369. Legacy -M for NfSen works again
• Improve nbar handling. Add private enterprise number decoding
• Merge pull request #357

nfdump-1.6.25

nfdump-1.6.25

This release is the last release for the nfdump-1.6.x tree and created for users of the nfdump-1.6.x branch.
For new setups and new features, it's recommended to use nfdump-1.7.x and the v.1.7.x releases.

nfdump-1.7.0.1

nfdump-1.7.0.1

Fix build issues in nfdump-1.7.0

Full Changelog: v.1.7.0...v1.7.0.1

nfdump-1.7.0

NFDUMP switches to new release 1.7.0

A lot of old code has beed remove, and was rewritten. nfdump-1.7.0 replaces nfdump-1.6.x. A lot of code has been improved and new features have been added. The nfpcapd collector has been reworked completely. It allows to merge pcap and flow data.

• nfdump is now a multi-threaded program and uses parallel threads mainly for reading, writing and processing flows as well as for sorting. This may result in a 2 to 3 times faster flow processing, depending on the tasks. The speed improvement also heavily depends on the hardware (SSD/HD) and flow compression option.
• For netflow v9 and IPFIX, nfdump now supports flexible length fields. This improves compatibility with some exporters such as yaf and others. The netflow v9 decoder is more flexible in decoding.
• Support for Cisco Network Based Application Recognition (NBAR).
• Supports Maxmind geo location information to tag/geolocate IP addresses and AS numbers.
• nfpcapd automatically uses TPACKET_V3 for Linux or direct BPF sockets for *BSD. This improves packet processing. It adds new options to collect MAC and VLAN information as well as the first packet of the payload.
• Metric exports: By default, every 60s a flow summary statistics can be sent to a UNIX socket. The corresponding program may be nfinflux to insert these metrics into an influxDB or nfexporter for Prometheus monitoring.

nfdump-1.6.24

Release 1.6.24 fixes various bugs:

• Release 1.6.24
• Update m4 files
• Update Makefile.am with ACLOCAL_AMFLAGS: #336
• Update Doxigen.in file: #332
• Fix cmd line processinf in nfanon. #328
• Make configure.ac autoconf 2.69 compatible
• Cleanup automake files. Fixes #304.
• Fix link handling in nfpcapd
• Fix compile flags #304
• Fix nfdump man page #301
• Fix minor bugs
• Add NAT event record support for IPFIX. #298
• Fix issue #296 - broken json format with option -q
• Fix json msec formating
• Silence short packet logs due to small snaplen in pcaproc.c #221

nfdump-1.6.23

• Fix potential FreeNode without valid Node in nfpcapd.
• Add all non TCP/UDP IP protocols as streams in nfpcapd
• Add mpls unwrap in nfpcapd. Skip MPLS labels
• Add ESP to processed protocols in nfpcapd.
• Some Code cleanup
• Change spin lock to native C11 lock
• Cleanup code for issue #283
• Fix minor nfpcapd issues
• Add mpls unwrap in sflow code - adds mpls labels if available
• Update rbtree.
• Fix potential deadlock in nfpcapd if it terminates.
• Add packet capture buffer size to nfpcapd
• Fix sflow code extended field parsing. #262 and #273
• Fix endless loop of nfexpire, if it does not find files
• Fix processing deoding error for yaf exporter
• Zero out tcp flags for non TCP records
• Add reverse element enterprise ID 29305 for counter values
• Add biFlow direction element 239
• Add flow end reason element 136
• Make -Tall the default for nfcapd to collect extensions
• Code cleanup and boundary checks in option template processing
• Implement element 160 (SystemInitTime) in option template
• Add Element 160 (SystemInitTime) in flow record used by Huawei
• Fix path handling for -l
• Fix print plain numbers #263