From 8046e69cc48cfe287d1fa006398b8ecad6cb63ec Mon Sep 17 00:00:00 2001 From: Tomas Cejka Date: Mon, 8 Feb 2016 11:37:15 +0100 Subject: [PATCH 01/10] updated revision of Nemea-Framework (Python3) Merged python3 branch, support added by Martin Juren from Flowmon Networks, a.s. --- nemea-framework | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nemea-framework b/nemea-framework index 1635417..d1e14e6 160000 --- a/nemea-framework +++ b/nemea-framework @@ -1 +1 @@ -Subproject commit 163541716ad25a5ab0bd8f93c90e471f9e35d26e +Subproject commit d1e14e61b8132a65026805a26077e117df256842 From ae0598fdf86e265ffcd9909f3da79807f77ebbbc Mon Sep 17 00:00:00 2001 From: Tomas Cejka Date: Mon, 8 Feb 2016 13:04:26 +0100 Subject: [PATCH 02/10] README.md: fixed links --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5545781..01d6aec 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ be supplied soon. Source Codes installation ------------------------- -The Nemea system consists of the [Nemea framework](cejkato2/Nemea-Framework), Nemea modules (basic and detection) and Nemea Supervisor. The whole system +The Nemea system consists of the [Nemea framework](CESNET/Nemea-Framework), [Nemea modules](CESNET/Nemea-Modules), [Nemea detectors](CESNET/Nemea-Detectors) and [Nemea Supervisor](CESNET/Nemea-Supervisor). The whole system is based on GNU/Autotools build system that makes dependency checking and building process much more easier. From c38ef0eb19359c1b8b333280f3bc5637f47b259f Mon Sep 17 00:00:00 2001 From: Tomas Cejka Date: Mon, 8 Feb 2016 13:22:14 +0100 Subject: [PATCH 03/10] README: fixed links (replaced with correct format) Link to external repositories is not handled automatically by github. --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 01d6aec..d9ebcd1 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ be supplied soon. Source Codes installation ------------------------- -The Nemea system consists of the [Nemea framework](CESNET/Nemea-Framework), [Nemea modules](CESNET/Nemea-Modules), [Nemea detectors](CESNET/Nemea-Detectors) and [Nemea Supervisor](CESNET/Nemea-Supervisor). The whole system +The Nemea system consists of the [Nemea framework](https://github.com/CESNET/Nemea-Framework), [Nemea modules](https://github.com/CESNET/Nemea-Modules), [Nemea detectors](https://github.com/CESNET/Nemea-Detectors) and [Nemea Supervisor](https://github.com/CESNET/Nemea-Supervisor). The whole system is based on GNU/Autotools build system that makes dependency checking and building process much more easier. @@ -89,10 +89,10 @@ that can be connected with each other. Information about every module can be fou ``` Every Nemea module can have one or more communication interfaces (IFC) implemented in -[libtrap](./nemea-framework/libtrap). There are two types of IFCs: **input** and **output**. Numbers of module's IFCs +[libtrap](https://github.com/CESNET/Nemea-Framework/tree/master/libtrap). There are two types of IFCs: **input** and **output**. Numbers of module's IFCs can be found in its help. -At the beginning, let's try the `logreplay` module ([./modules/logreplay](./modules/logreplay)). +At the beginning, let's try the `logreplay` module ([modules/logreplay](https://github.com/CESNET/Nemea-Modules/tree/master/logreplay)). The help output shows that `logreplay` has one output IFC: ``` Name: LogReplay @@ -103,7 +103,7 @@ Description: of CSV file has to be data format of fields. ``` -The complement module is `logger` ([./modules/logger](./modules/logger)), help output: +The complement module is `logger` ([modules/logger](https://github.com/CESNET/Nemea-Modules/tree/master/logger)), help output: ``` Name: Logger Inputs: variable @@ -119,7 +119,7 @@ Description: Two modules can be interconnected using one input IFC and one output IFC. The [./use-cases](./use-cases) directory contains example scripts that demonstrate usage and functionality of -Nemea modules. `logreplay` and `logger` can be found in [logger-repeater.sh](./use-cases/logger-repeater.sh). +Nemea modules. `logreplay` and `logger` can be found in [./use-cases/logger-repeater.sh](./use-cases/logger-repeater.sh). Start the script to see how flow records are replayed from CSV file by `logreplay` and received by `logger`: ``` cd use-cases @@ -131,7 +131,7 @@ To get usage of scripts from `use-cases`, execute a script without parameter. Th scripts. `logreplay` is one of possible ways of getting data into the Nemea system. -There is a [nfreader](./modules/nfreader) module that is able to read and replay `nfdump` files. +There is a [nfreader](https://github.com/CESNET/Nemea-modules/tree/master/nfreader) module that is able to read and replay `nfdump` files. Last but not least, there is an [ipfixcol](https://github.com/CESNET/ipfixcol/) with [ipfixcol2unirec](https://github.com/CESNET/ipfixcol/tree/master/plugins/storage/unirec) that is capable of exporting flow data in UniRec format and sending it via libtrap IFC. @@ -142,7 +142,7 @@ The Nemea system can be managed and monitored by a special module called [Supervisor](https://github.com/CESNET/Nemea-Supervisor). Some modules that are contained in Nemea-Modules and Nemea-Detectors provide their default -configuration in [https://github.com/CESNET/Nemea-Supervisor/tree/master/configs/](nemea-supervisor/configs/). +configuration in [nemea-supervisor/configs/](https://github.com/CESNET/Nemea-Supervisor/tree/master/configs/). To use prepared configuration, run `make` in `nemea-supervisor/configs` and start: ``` nemea-supervisor/supervisor -f nemea-supervisor/configs/supervisor_config.xml From 78b31ba9603c082de5051ee7d4d8dc8ccbf4cc8e Mon Sep 17 00:00:00 2001 From: Tomas Cejka Date: Tue, 9 Feb 2016 09:50:04 +0100 Subject: [PATCH 04/10] updated revision of Nemea-Framework --- nemea-framework | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nemea-framework b/nemea-framework index d1e14e6..82f4b59 160000 --- a/nemea-framework +++ b/nemea-framework @@ -1 +1 @@ -Subproject commit d1e14e61b8132a65026805a26077e117df256842 +Subproject commit 82f4b59bb2f89edde900b71f14f262d18b30c081 From 2a7ec4d04ba057591d551a27f7f510af8c7e1612 Mon Sep 17 00:00:00 2001 From: Tomas Cejka Date: Tue, 9 Feb 2016 16:51:32 +0100 Subject: [PATCH 05/10] doc: move links to other repos to the top --- README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d9ebcd1..5c72386 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,12 @@ NEMEA System Travis CI build: [![Build Status](https://travis-ci.org/CESNET/Nemea.svg?branch=master)](https://travis-ci.org/CESNET/Nemea) +The Nemea system consists of: +* [Nemea framework](https://github.com/CESNET/Nemea-Framework): The heart of the system that provides interconnection of modules, data format (and its handling) and common functions, algorithms and data structures. +* [Nemea modules](https://github.com/CESNET/Nemea-Modules): Base modules of the system for export&storage of flow data, replay of stored flow data, filtering, merging, and others. It also contains a basic flow exporter capable of L7 information. +* [Nemea detectors](https://github.com/CESNET/Nemea-Detectors): Detection modules that can detect and report various types of malicious traffic such as DoS, DDoS, scanning, bruteforce attacks. +* [Nemea Supervisor](https://github.com/CESNET/Nemea-Supervisor): Central management and monitoring module of the Nemea system. It takes care of running modules according to configuration. + This file describes the installation and basic usage of the Nemea system. To see more general information, please have a look at https://www.liberouter.org/nemea. @@ -30,8 +36,7 @@ be supplied soon. Source Codes installation ------------------------- -The Nemea system consists of the [Nemea framework](https://github.com/CESNET/Nemea-Framework), [Nemea modules](https://github.com/CESNET/Nemea-Modules), [Nemea detectors](https://github.com/CESNET/Nemea-Detectors) and [Nemea Supervisor](https://github.com/CESNET/Nemea-Supervisor). The whole system -is based on GNU/Autotools build system that makes dependency checking and +The whole system is based on GNU/Autotools build system that makes dependency checking and building process much more easier. To clone the read-only repositories, use: From 6c17327e2ea7b9b01725ec07d189168a92afa4f1 Mon Sep 17 00:00:00 2001 From: Tomas Cejka Date: Thu, 11 Feb 2016 18:01:57 +0100 Subject: [PATCH 06/10] updated revision of Nemea-Framework&Modules: report2idea Support of IDEA reports that can be stored into MongoDB or sent into Warden - system for incident sharing (https://wardenw.cesnet.cz/). --- modules | 2 +- nemea-framework | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules b/modules index 491e0bc..addca4c 160000 --- a/modules +++ b/modules @@ -1 +1 @@ -Subproject commit 491e0bc30b2046b769347b954bfa34e8f2c15278 +Subproject commit addca4cded8f612c0b276e828bf98ce351c9fa7f diff --git a/nemea-framework b/nemea-framework index 82f4b59..aa010bb 160000 --- a/nemea-framework +++ b/nemea-framework @@ -1 +1 @@ -Subproject commit 82f4b59bb2f89edde900b71f14f262d18b30c081 +Subproject commit aa010bb07fba3fa907417c4fcd1385beee8f202f From 71cd1555b359ab52d5d1b7bdc59af494a3942658 Mon Sep 17 00:00:00 2001 From: Tomas Jansky Date: Tue, 16 Feb 2016 11:40:41 +0100 Subject: [PATCH 07/10] nemea: now possible to create all RPM packages and nemea meta package --- configure.ac | 26 +++++++++++++++++--------- nemea.spec.in | 19 +++++++++++++++++++ rpms.sh | 29 +++++++++++++++++++++-------- 3 files changed, 57 insertions(+), 17 deletions(-) create mode 100644 nemea.spec.in diff --git a/configure.ac b/configure.ac index 8511469..148b961 100644 --- a/configure.ac +++ b/configure.ac @@ -4,17 +4,25 @@ AC_PREREQ([2.63]) AC_INIT([nemea], [2.1.0], [traffic-analysis@cesnet.cz]) AM_INIT_AUTOMAKE([foreign subdir-objects]) +RELEASE=1 +AC_SUBST(RELEASE) +USERNAME=`git config --get user.name` +USERMAIL=`git config --get user.email` +AC_SUBST(USERNAME) +AC_SUBST(USERMAIL) +# Check for rpmbuild +AC_CHECK_PROG(RPMBUILD, rpmbuild, rpmbuild, [""]) -# Checks for programs. +AC_SUBST(RPMBUILD) +if test -z "$RPMBUILD"; then + AC_MSG_WARN([Due to missing rpmbuild you will not able to generate RPM package.]) +fi -# Checks for libraries. +RPM_RELEASE=1 +AC_SUBST(RPM_RELEASE) +AM_CONDITIONAL(MAKE_RPMS, test x$RPMBUILD != x) -# Checks for header files. - -# Checks for typedefs, structures, and compiler characteristics. - -# Checks for library functions. - -AC_CONFIG_FILES([Makefile]) +AC_CONFIG_FILES([Makefile + nemea.spec]) AC_CONFIG_SUBDIRS([nemea-framework nemea-supervisor modules detectors]) AC_OUTPUT diff --git a/nemea.spec.in b/nemea.spec.in new file mode 100644 index 0000000..2a9a8eb --- /dev/null +++ b/nemea.spec.in @@ -0,0 +1,19 @@ +Summary: Nemea system +Name: nemea +Version: @PACKAGE_VERSION@ +Release: @RELEASE@ +URL: http://www.liberouter.org/ +#Source: https://www.liberouter.org/ +Group: Liberouter +License: BSD +Vendor: CESNET, z.s.p.o. +Packager: @USERNAME@ <@USERMAIL@> +BuildRoot: %{_tmppath}/%{name}-%{version}-%{release} +BuildArch: noarch +Requires: libtrap nemea-common nemea-modules nemea-detectors nemea-python nemea-pycommon nemea-supervisor + +%description +Meta-package of the Nemea system, the distributed modular system for network traffic analysis and anomaly detection. + +%files + diff --git a/rpms.sh b/rpms.sh index 8ceee5e..eb27941 100755 --- a/rpms.sh +++ b/rpms.sh @@ -73,37 +73,50 @@ export chuser ( cd libtrap su $chuser -p -c "$topdir/generate-rpm.sh" - $pkginst install -y -q ./RPM*/RPMS/*/* + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) ( cd common su $chuser -p -c "$topdir/generate-rpm.sh" - $pkginst install -y -q ./RPM*/RPMS/*/* + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) ( cd unirec su $chuser -p -c "$topdir/generate-rpm.sh" - $pkginst install -y -q ./RPM*/RPMS/*/* + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) + su $chuser -p -c "./bootstrap.sh >/dev/null 2>/dev/null&& ./configure -q" + ( cd python - su $chuser -p -c "$topdir/generate-rpm.sh" - $pkginst install -y -q ./RPM*/RPMS/*/* + su $chuser -p -c "make -j4 && make rpm" + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) + ) + ( + cd pycommon + su $chuser -p -c "make -j4 && make rpm" + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) ) ( cd modules su $chuser -p -c "$topdir/generate-rpm.sh" - $pkginst install -y -q ./RPM*/RPMS/*/* + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) ( cd detectors su $chuser -p -c "$topdir/generate-rpm.sh" - $pkginst install -y -q ./RPM*/RPMS/*/* + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) ( cd nemea-supervisor su $chuser -p -c "$topdir/generate-rpm.sh" - $pkginst install -y -q ./RPM*/RPMS/*/* + $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) +su $chuser -p -c "$topdir/bootstrap.sh >/dev/null 2>/dev/null&& $topdir/configure -q" +mkdir -p "`pwd`/RPMBUILD" +rpmbuild -ba nemea.spec --define "_topdir `pwd`/RPMBUILD" +mkdir -p "`pwd`/rpms" +find -name *.rpm -not -path "./rpms/*" -exec mv {} rpms/ \; + From 329f2be2d62b72bdfe3dfafac07b367f2ff581a6 Mon Sep 17 00:00:00 2001 From: Tomas Jansky Date: Tue, 16 Feb 2016 15:22:35 +0100 Subject: [PATCH 08/10] rpms.sh: updated releasing nemea-python and nemea-pycommon packages --- rpms.sh | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/rpms.sh b/rpms.sh index eb27941..24870b2 100755 --- a/rpms.sh +++ b/rpms.sh @@ -85,16 +85,14 @@ export chuser su $chuser -p -c "$topdir/generate-rpm.sh" $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) - su $chuser -p -c "./bootstrap.sh >/dev/null 2>/dev/null&& ./configure -q" - ( cd python - su $chuser -p -c "make -j4 && make rpm" + su $chuser -p -c "python setup.py bdist_rpm" $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) ( cd pycommon - su $chuser -p -c "make -j4 && make rpm" + su $chuser -p -c "python setup.py bdist_rpm" $pkginst install -y -q $(find \( -name '*noarch.rpm' -o -name '*64.rpm' \)) ) ) From cb1b125f97bcb1ec9eb736a268e55b9bf77819c1 Mon Sep 17 00:00:00 2001 From: Tomas Jansky Date: Tue, 16 Feb 2016 15:55:52 +0100 Subject: [PATCH 09/10] increased version of nemea --- configure.ac | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure.ac b/configure.ac index 148b961..a20dc73 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ([2.63]) -AC_INIT([nemea], [2.1.0], [traffic-analysis@cesnet.cz]) +AC_INIT([nemea], [2.2.0], [traffic-analysis@cesnet.cz]) AM_INIT_AUTOMAKE([foreign subdir-objects]) RELEASE=1 AC_SUBST(RELEASE) From dce4aaf13849d56f667920915d575ffc244bf7b3 Mon Sep 17 00:00:00 2001 From: Tomas Jansky Date: Tue, 16 Feb 2016 15:56:37 +0100 Subject: [PATCH 10/10] updated revision of nemea, released RPM package --- configure.ac | 2 +- detectors | 2 +- modules | 2 +- nemea-framework | 2 +- nemea-supervisor | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/configure.ac b/configure.ac index a20dc73..ec56cc3 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ([2.63]) -AC_INIT([nemea], [2.2.0], [traffic-analysis@cesnet.cz]) +AC_INIT([nemea], [2.2.1], [traffic-analysis@cesnet.cz]) AM_INIT_AUTOMAKE([foreign subdir-objects]) RELEASE=1 AC_SUBST(RELEASE) diff --git a/detectors b/detectors index d114aa3..744d59a 160000 --- a/detectors +++ b/detectors @@ -1 +1 @@ -Subproject commit d114aa39656cbd827840816a610b1942651efefe +Subproject commit 744d59a578725837813198e496a69c81bb43c700 diff --git a/modules b/modules index addca4c..0ffaadf 160000 --- a/modules +++ b/modules @@ -1 +1 @@ -Subproject commit addca4cded8f612c0b276e828bf98ce351c9fa7f +Subproject commit 0ffaadf9e2fd6bd137b494b4a97644998de0b91a diff --git a/nemea-framework b/nemea-framework index aa010bb..fa04c56 160000 --- a/nemea-framework +++ b/nemea-framework @@ -1 +1 @@ -Subproject commit aa010bb07fba3fa907417c4fcd1385beee8f202f +Subproject commit fa04c569456584faa972d4a2c8f6342b47c339ae diff --git a/nemea-supervisor b/nemea-supervisor index 1cbb791..b5d717e 160000 --- a/nemea-supervisor +++ b/nemea-supervisor @@ -1 +1 @@ -Subproject commit 1cbb791559ac43008c2c4b541e29bd1c1598d67c +Subproject commit b5d717ec385743e7e728db653a257cf4428e9fde