From 79d251c89b4075ebea1e6939883e6fe489b7611a Mon Sep 17 00:00:00 2001 From: Peter Haag Date: Wed, 23 Oct 2024 19:57:56 +0200 Subject: [PATCH] Commit Releas 1.7.5 --- ChangeLog | 278 ++++++++++++++++++++++++++++++++++- configure.ac | 2 +- src/libnffile/gen_version.sh | 2 +- 3 files changed, 278 insertions(+), 4 deletions(-) diff --git a/ChangeLog b/ChangeLog index 209560ac..8e49d544 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,9 +1,283 @@ -- c9fe29c 2024-03-03 (HEAD -> nfdump-ja4, origin/nfdump-ja4) Prepare nfstat for var length fields such as ja4 +- 18dc06d 2024-10-23 (HEAD -> master, origin/master, origin/HEAD) Fix nfeplay template identifu ID +- 4c8b892 2024-10-19 Merge pull request #571 from TiceDB/patch-2 +- b96828f 2024-10-19 Merge pull request #570 from TiceDB/patch-1 +- b1d2a1f 2024-10-18 Point JSON field in_bytes to inBytes instead of inPackets +- 6825cb0 2024-10-18 Add missing delimiter for JSON output +- 04cc2b8 2024-09-22 Fix #567 devel compile error. +- 0d78cf7 2024-09-09 Fix broken length calculation +- b72105d 2024-09-09 Replace sprintf with snprintf in output files +- 49ed7d8 2024-09-08 Add RTT lantencu code - but disabled for now +- 0e80ef2 2024-09-07 Merge pull request #564 from paineta/master +- 76aa13a 2024-09-06 Update output_fmt.c +- dc85e39 2024-08-31 Addint64 and uint64 config routines +- 15b9bfd 2024-08-31 Change GRE cli option -G in sfcapd for -o gre option string. #559 +- 87a9e67 2024-08-31 Add new option functions in nfconf and wire nfpcapd options +- 14b2179 2024-08-31 Use sysconfdir for default nfdump.conf file +- 99db1cd 2024-08-31 Merge pull request #559 from simonschdev/parse-gre +- 5c67031 2024-08-31 ADd FIN/RST signal flag to node +- dbaa772 2024-08-29 Fix latency calculation in nfpcapd +- 6d7d179 2024-08-27 Add src/dst tcp/udp port IE 180-183 +- ae52e09 2024-08-27 Fix #560 - geo filter containing keywords +- dfb736f 2024-08-25 Add array decoding in ConfInventory() +- 8b08c33 2024-08-25 Replace toml code with better maintained version +- 71a2dd2 2024-08-25 Fix potential use after free +- 6409763 2024-08-24 Change record count from 32 to 64bits +- fb344fb 2024-08-23 minor cosmetic fixes +- 268ff3f 2024-08-23 implemented GRE tunnel introspection for sFlow +- c0ffe7d 2024-08-23 Fix duration string in summary line +- 341575a 2024-08-23 Merge pull request #558 from Takalele/master +- 5e08f54 2024-08-22 Fix #507 - gcc-14 issues with rrd version 1.9.x +- 312951f 2024-08-20 fixed updateTorDB.sh +- e1eed87 2024-08-20 Show some love to alpine linux +- 0b2796b 2024-08-20 Make gcc-14 happy +- 813da06 2024-08-20 Implement maxmind and tor lookup filter in nfprofile. See #557 +- 925dd28 2024-08-20 Cleanup nfpcapd Makefile and make Alpine Linux happy with GNU flag +- 1cce258 2024-08-18 Merge pull request #556 from longregen/patch-1 +- 84c71a6 2024-08-17 fix: missing -W option in nfpcapd argparsing +- 1fc74f0 2024-08-16 Make sure, only flow nodes get dummped - see also #555 +- 6ec9d81 2024-08-16 Test if node is fragement node in defragment code. Fix #555 +- 56c7ddf 2024-08-09 Switch tcp flags to alpha string in json. ndjson output. #553 +- 10d4666 2024-08-09 Fix #554 - code dump in new processing pipeline. +- 493c542 2024-08-07 Update nfdump(1) man page for ndjson and cleanup +- 75cca36 2024-08-07 Fix coredump #552. Replace json-log by ndjson. #553 Cleanup json output. Replace " in maxmind org and city names by '. Otherwise breaks json output. +- a29354c 2024-08-06 Add poc for post filter. See #551 +- aef51ae 2024-08-03 Align large vlan labels in ra output +- c4a6a55 2024-08-01 Add file torlookup.1 man page +- 0c815b2 2024-08-01 Add torlookup.1 man page as well as more information for torlookup in general +- 7d437e5 2024-08-01 Fix tor code and update script +- 2c461b5 2024-07-31 Add src,dst ip tor filter to match tor exit nodes and cleanup output buffer code +- 2a78afa 2024-07-29 Merge branch 'master' of ssh://github.com/phaag/nfdump +- 0c5e288 2024-07-29 Convert cvs output to itoa and local string buffer +- 266b960 2024-07-29 Fix buffer check in output_csv_fast.c +- cc0f593 2024-07-29 Add token %tsg, %teg for GMT start and end time in fmt and csv format. Update man page +- 51fded0 2024-07-29 Cleanup some output code +- 9d3cdd9 2024-07-29 Replace printf by local buffer and itoa. See #548 +- fa2785b 2024-07-29 Add fixed but fast csv output mode csv-fast. See #548 +- 87ee686 2024-07-29 Add local itoa +- bd93c60 2024-07-29 Speedup json output +- b6179a6 2024-07-22 Silence know record error message in filter thread +- 9fb15d1 2024-07-22 Strip quotes from org name in Maxmind DB +- 4804dbf 2024-07-22 Update mmhash.c +- 2fbc960 2024-07-22 Update mmhash.c +- 0577dee 2024-07-20 Fix ebuff size +- ad70a1f 2024-07-20 Fix platform issues and compiler warnings +- 961f0ab 2024-07-20 Update geolookup man page +- 1811ea3 2024-07-20 Add AS lookup to geolookup and add new -s srcasn, dstasn and asn for AS organistation name. Needs maxmind DB to work. +- 3a9908e 2024-07-20 Cleanup grammar. Fix error messages +- a253e80 2024-07-20 Cleanup grammar file +- 3f26afa 2024-07-19 Regex filter multithreading patch +- 9965ceb 2024-07-19 Fix AS mmdb lookup for flow addregation +- 62b8fb1 2024-07-11 Fix inherit compression bug. #525 +- 64759bd 2024-07-11 Implement #543 and moved fragment flags +- 53f9fa0 2024-07-07 Cleanup fixes, and fix gcc complains +- bc3c2c9 2024-07-07 Merge pull request #545 from dceara/issue544-fix-typo-domain-id +- 44c2fe5 2024-07-05 output_json: Fix observationDoaminID typo in json output. +- 0659254 2024-07-05 Wire dedup into optional packet dump -p +- 9562322 2024-06-29 Add metrohash to nflowcache +- fb660f7 2024-06-29 Fix action script +- 69f5d13 2024-06-29 Update c-cpp.yml +- 745c60b 2024-06-29 Change hash to metrohash for de-duplication +- 65ad2f9 2024-06-29 Add packet de-duplication for nfpcapd +- 3f8fcdd 2024-06-28 Cleanup Makefiles +- 483e780 2024-06-25 Wire new csv format into aggregation output. #529 +- e56f7f1 2024-06-24 Fix compiler warnings on some Linux systems +- ce52547 2024-06-24 Fix minor format issues +- 2c661f6 2024-06-24 Fix minor format issues +- a78db7f 2024-06-24 Fix minor format issues +- 23c7f4a 2024-06-23 Switch to new csv format definition +- 35bcbff 2024-06-19 Fix ja3/ja4 printing +- e44d934 2024-06-10 Fix format printing in packet-linux +- 8b1416f 2024-06-10 Add csv headers and config +- 595c3bd 2024-06-09 Fix runtest +- 2d68eca 2024-06-09 First batch of changes for flexible csv format output mode - still compatible +- 0b20860 2024-06-09 First batch of changes for flexible csv format output mode - still compatible +- e7df924 2024-06-03 update .gitignore +- 3b06350 2024-06-03 Remove vcs_track in repo. Created automatically +- 109860b 2024-06-02 Remove debug output +- 2728a91 2024-06-02 Block formats other than auto fmt for aggregation +- 7919b31 2024-06-02 Add json output for element stat. #528 +- f4fecaa 2024-06-02 Add size in verify mode -v +- b3869bc 2024-06-02 Implement nokia NAT logging. See #533 +- f0a5f6f 2024-06-02 Add NOKIA enterprise IDs. Not yet wired. #533 +- 1619a60 2024-06-01 Merge pull request #538 from piorek94/fix_srcip6_agg +- 558b513 2024-06-01 Remove local files - I hate the desktop client! +- bf36a07 2024-06-01 Remove local files - I hate the desktop client! +- c96efb9 2024-06-01 Remove local files - I hate the desktop client! +- 044b1ae 2024-06-01 Remove local files - I hate the desktop client! +- 5c12e57 2024-06-01 Fix some memory leaks, although process ends. +- 5118e35 2024-06-01 fix srcip6 aggregation +- 2b5db3b 2024-06-01 Fix convert.c from merge +- c96f913 2024-06-01 Add multithreading filter processing +- 56907f7 2024-06-01 Merge remote-tracking branch 'origin/nfdump-devel' +- 3810acf 2024-05-31 Null IPs if inet aggregation does not match flow inet version. #537 +- 1d25d32 2024-05-31 Fix aggregation bug with v4/v6 flows #536 +- 7b701c9 2024-05-25 Add DLT_LINUX_SLL2 link type to pcap reader +- af5d8d8 2024-05-24 Fix man page for 'bgp next ip' #531 +- daf20df 2024-05-24 Merge pull request #532 from piorek94/agg_zero_division_fix +- 2ac8675 2024-05-24 Merge pull request #530 from piorek94/csv_newline_fix +- 74c286f 2024-05-24 Fix #534 typo +- 825ec11 2024-05-23 fix zero division issue in aggregation mode +- 6b9d320 2024-05-23 remove etherType +- 18e174a 2024-05-21 Fix coredump on dynamic exporter mode with no packets. #527 +- da06eb0 2024-05-20 Fix help and error text for compression +- 564f3a9 2024-05-11 Cleanup minor issues and add example for ft2nfdump +- d105409 2024-05-06 Add another dataBlock test +- 43fd78d 2024-05-06 Fix #521. Cleanup unclear naming of xlate records +- e8e8c38 2024-05-04 Remove heapsort. No longer needed these days +- 5cd6645 2024-05-04 Fix ja3 pre-processing in filter +- e3f4436 2024-05-03 Improve help text +- 2450ace 2024-05-03 Remove hash back reference in nflowcache +- d222bc0 2024-05-03 Cleanup hash code for nfstat and nflowcache +- 885f7df 2024-05-02 Rework nflowcache. Replace hash code to speedup hash and have cleaner code for later multi-threading -s -A stats. +- 73a2d8a 2024-04-20 Merge branch 'nfdump-devel' of ssh://github.com/phaag/nfdump into nfdump-devel +- cfca245 2024-04-20 Test new hash in nfstat +- f402ba3 2024-04-20 Merge branch 'master' of github.com:phaag/nfdump +- 8feced6 2024-04-20 Test new hash in nfstat +- f9e9e6a 2024-04-16 Improve boundary checking +- edd489a 2024-04-16 Improve boundary checking +- 9bc3553 2024-04-14 Sync with devel +- def1e91 2024-04-14 Sync changes with master +- 686f749 2024-04-14 Remove some old 1.6 code and clean up compat reader +- ea0286a 2024-04-14 Finally remove old master_record +- 18eff82 2024-04-14 Fix queue_close with multiple producers +- 774e29d 2024-04-14 Fix IP lookup in filter +- cb88b55 2024-04-14 Fix IP lookup in filter +- d6a5d32 2024-04-14 Merge branch 'nfdump-devel' of ssh://github.com/phaag/nfdump into nfdump-devel +- c1ba7d2 2024-04-14 Mark some structs as const +- 383fa76 2024-04-14 mark some structs as const +- 95e70eb 2024-04-14 Clear filter flag after use +- b1373fb 2024-04-13 Update AS statistics fix +- 833a09d 2024-04-13 Fix element stat for AS statistics. #519 +- 122b882 2024-04-12 Sync branch +- 5abfd0d 2024-04-12 Work on filter code +- 66cd4d3 2024-04-12 More work on threadding code +- 97b619a 2024-04-12 refine const definitions in filter code +- 63309d6 2024-04-12 Fix tunnel flow fmt printing +- 37e70f3 2024-04-11 New convert block type 2 blocks code in pipelining +- ff08a2d 2024-04-09 Enable convert1.6 in pipeline +- 5071c2f 2024-04-07 Improve record processing +- a16f86f 2024-04-07 Fix debugging messages +- 020a772 2024-04-07 Test multithreadding nfdump +- 1400011 2024-04-06 Fix nfstat ptr offset +- 0c9293d 2024-04-06 Cleanup nfdump main loop +- e7ef540 2024-04-05 Remove debugging code in tor.c +- a541557 2024-04-05 Fix typos in torlookup +- 6ccca84 2024-04-05 Make sure to use new compiled geolookup binary to update mmdm +- 5cfefa9 2024-04-05 Make sure to use new compiled torlookup binary +- 5586eea 2024-04-05 Fix tor.c +- 151aeb8 2024-04-05 Fix maxmind.c +- 3adace8 2024-04-03 Fix json output +- f462e2b 2024-04-02 Address #518 and silence output for normal conditions +- b064481 2024-04-02 More readable output +- 903f0c6 2024-04-02 Fix runtests +- b42bdec 2024-04-02 Fix raw output for ip version +- d36ecc5 2024-04-02 Update space to 64bit +- 21c61f9 2024-04-02 Reserve space for vxLan element +- 8941a79 2024-04-02 Fix nflowcache aggregation (see #515) and cleanup some code +- 804d92e 2024-04-01 Cleanup code and fix ChangeIdent +- d876444 2024-04-01 Cleanup #517 and fix runtest +- fc2fae5 2024-04-01 Merge pull request #517 from jav4/ethertype +- 50126db 2024-04-01 Update output_raw.c +- f2c7375 2024-04-01 Update nffile_compat.c +- 163bf0f 2024-04-01 Update output_json.c +- 4a58dd5 2024-04-01 Update output_csv.c +- 671e205 2024-04-01 Update nfxV3.h +- a0d32aa 2024-04-01 Update output_raw.c +- d9683df 2024-04-01 Update output_json.c +- 2e13f89 2024-04-01 Update output_json.c +- 5a5f00b 2024-04-01 Update output_fmt.c +- 0615d34 2024-04-01 Update output_csv.c +- 757be79 2024-04-01 Update nfstat.c +- a532205 2024-04-01 Update nfx.h +- 0c239df 2024-04-01 Update nflowcache.c +- d145f87 2024-04-01 Update nfx.c +- c0dbeb0 2024-04-01 Update nfdump_1_6_x.h +- 0589904 2024-04-01 Update netflow_v9.c +- e12da4b 2024-04-01 Update grammar.y +- 4d69b07 2024-04-01 Merge branch 'master' into ethertype +- 77b04b1 2024-04-01 Merge dot1q and physical extension into a single layer2 extension +- 65fe66e 2024-04-01 Fix potential deadlock, if file queue empty +- 9289f6a 2024-03-31 Add working threads to nfprofile +- 819d8bc 2024-03-31 Remove exporter stuff from profiler. Not needed +- d4420f7 2024-03-31 Rename pthread_barrier to avoid collision with Linux version +- 9c9b71a 2024-03-31 Cleanup nfanon +- 7169029 2024-03-30 Add user defined workers for nfanon +- 3202fab 2024-03-30 Move barrier code into own files +- 928653c 2024-03-30 Add threadding support to nfanon +- 06475c9 2024-03-30 Fix nfreader sample code +- ea6322e 2024-03-30 Cleanup nfanon +- e7cac3e 2024-03-30 Fix AS range check +- 2553d3e 2024-03-30 Sync nfcapd, sfcapd code +- dabf79c 2024-03-30 Cleanup collector code and sync sfcapd +- 5cffb3a 2024-03-29 Cleanup collector loop +- 2d806f6 2024-03-29 Add more detaied error message +- 903a8aa 2024-03-29 Handle potential file rotate error +- 39a6946 2024-03-29 Remove unneeded check +- c6ea78a 2024-03-29 Add IP version in json output +- 98b1928 2024-03-29 Implement #515 features +- 1b6139e 2024-03-29 Add dot1q in ipfix. Fix for #515 +- c4a2575 2024-03-29 Fix #516 +- 32770d5 2024-03-29 Dissolve dataBlock in nffile_t for writing in order to prepare for multi threading processing flows +- ef928d6 2024-03-29 Add Netflow v9 ethertype field support +- fb664fc 2024-03-22 Dissolve dataBlock for reading in nffile_t in order to prepare for mutli-threading processing. +- 9198d94 2024-03-17 New FilterSetParam function +- d253340 2024-03-17 Improve performance to read and convert nfdump-1.6.x records. See #512 +- 3b388b8 2024-03-16 Cleanup configure.ac +- 70732b2 2024-03-16 Add conditional build and licensing terms for ja4+ +- 282187a 2024-03-16 Update nfdump.1 man page with ssl and ja4 filters +- 6f4f6ad 2024-03-16 Add SSL/TLS sni filter +- 69959f0 2024-03-16 Add SSL/TLS filter +- 45ade57 2024-03-16 Add ja4 tests to nftest +- 2e6d172 2024-03-16 Fix nftest +- 8c18cff 2024-03-16 Add ja4s to filter syntax +- 23fd4a2 2024-03-15 separate ja4, ja4s in nfstat +- dedb73a 2024-03-15 Streamline ja3, ja4, ssl code +- d5220a9 2024-03-14 Add ssl preproc to filter +- 9b7a54a 2024-03-14 Fix getopt parsing - #513 +- 77ae680 2024-03-14 Add ja4 to filter. First try +- 8fe6a2b 2024-03-11 Work on ja4 in nfstat +- 595952a 2024-03-10 Work on ja4 +- ab0810d 2024-03-10 Wire ja4/ja4s in output json and do more work on SSL payload +- 070cd71 2024-03-10 Wire ja4/ja4s in fmt output format +- f09d7a5 2024-03-09 Wire ja4/ja4s in raw output format +- b1bf5a1 2024-03-09 Move tor code to libnfdump +- c9969ec 2024-03-09 Cleanup Makefiles +- 8c97e7e 2024-03-09 Integrate maxmind code into libnfdump +- b733d6c 2024-03-09 Fix typo +- 84b425b 2024-03-09 Fix library order errors +- 2043de7 2024-03-09 Split libnfdump into libnffile and libnfdump +- 784e62d 2024-03-09 Merge remote-tracking branch 'origin/nfdump-ja4' +- 8a0c883 2024-03-09 Fix typo +- d5add6b 2024-03-09 Fix compile errors from pull #511 +- 9241fee 2024-03-09 Merge pull request #511 from blkmajik/json-log +- f0f9f4b 2024-03-09 Merge branch 'master' into json-log +- c5c5e47 2024-03-09 Code cleanup +- 0b59bea 2024-03-08 Update nfdump.conf.dist +- 3ff30f2 2024-03-08 Fix OS library order +- d77c056 2024-03-08 Fix configure.ac +- 93ce7fb 2024-03-08 Add torlookup for IPs +- b65a3ff 2024-03-08 Maxmind code refactoring +- c4f8169 2024-03-08 Streamline ja3/ja4 +- fb63274 2024-03-05 Move sampled to end to handle proper syntax on closing. No final , on last field +- 23ec1e5 2024-03-05 Use record separator instead of field separator between records +- 031b462 2024-03-05 Swap fs/rs for json-log format +- 3dfc829 2024-03-05 Make fprintf happy +- 432925e 2024-03-05 Const char instead +- da3cf0d 2024-03-05 Integrate json-log format into output_json +- 9c1021d 2024-03-03 Merge remote-tracking branch 'origin/nfdump-ja4' +- cd9b43e 2024-03-03 Fix Makefiles +- 626be0b 2024-03-03 Merge remote-tracking branch 'origin/nfdump-ja4' +- 17ead4c 2024-03-03 Updeate ChangeLog +- c9fe29c 2024-03-03 Prepare nfstat for var length fields such as ja4 - 2a55c3b 2024-03-03 Commit +- 109c1c9 2024-03-03 Fix ssl decoder +- f991322 2024-03-03 Remove debugging flag - 1e88803 2024-03-01 Implement ja4s - 4a60672 2024-03-01 Update Readme - c129b78 2024-03-01 Add ja4 Readme for license issue and add --enable-ja4, default no for building all ja4 modules - 5c72cc8 2024-03-01 Add ja3 filter in nfdump-ja4 +- 5cdc989 2024-03-01 Cleanup include +- 539902c 2024-03-01 Insert ja3 filter code - 8ed9f34 2024-03-01 Add ja4_c - 2ded6b5 2024-02-29 Implement ja_a and ja_b of ja - ae92ef6 2024-02-29 Add ja4 files to start implementation @@ -14,7 +288,7 @@ - 07136b3 2024-02-25 Use new ssl module for ja3 - 17d127e 2024-02-24 Debug ssl code - Client Hello - 79e9b8a 2024-02-24 Defrag ssl and ja3 - Add plain ssl code -- 00e0e45 2024-02-24 (origin/master, origin/HEAD) Fix nfreader code +- 00e0e45 2024-02-24 Fix nfreader code - eb5bef5 2024-02-22 Cleanup code to handle legacy sampler record - a988a40 2024-02-18 Rework maxmind code - 9a0dddf 2024-02-17 Fix library dependencies on some *nix diff --git a/configure.ac b/configure.ac index bf231daf..b669d3b6 100644 --- a/configure.ac +++ b/configure.ac @@ -3,7 +3,7 @@ AC_PREREQ([2.69]) AC_REVISION($Revision: 244 $)dnl -AC_INIT([nfdump],[1.7.4],[peter@people.ops-trust.net]) +AC_INIT([nfdump],[1.7.5],[peter@people.ops-trust.net]) AC_CONFIG_HEADERS([config.h]) AM_INIT_AUTOMAKE([subdir-objects]) diff --git a/src/libnffile/gen_version.sh b/src/libnffile/gen_version.sh index eb722049..6d97b62a 100755 --- a/src/libnffile/gen_version.sh +++ b/src/libnffile/gen_version.sh @@ -42,7 +42,7 @@ if [ -d ../../.git ]; then else # no git directory - most likely release - zip or tarball hash="release" - date="Sat Feb 17 15:05:20 CET 2024" + date="Wed Oct 23 19:53:03 CEST 2024" fi echo \#ifndef __VCS_TRACK_H__ >$f