Security/NiFi: added script user acc creation + env vars. Task 8694kp…

…aud.

deploy/nifi.env

@@ -63,7 +63,3 @@ USER_SCRIPT_DB_DIR="/opt/nifi/user-scripts/db/"
 #LDAP_TLS_KEYSTORE=/opt/certs/ldaps_keystore.jks
 #LDAP_TLS_KEYSTORE_TYPE=JKS
 #LDAP_TLS_KEYSTORE_PASSWORD=
-
-
-# NiFi user
-

deploy/services.yml

@@ -929,6 +929,7 @@ services:
       - ./project.env
       - ./general.env
       - ./network_settings.env
+      - ../security/nifi_users.env
     shm_size: 1024mb
     environment:
       - http_proxy=$HTTP_PROXY
@@ -958,10 +959,12 @@ services:
       - ../nifi/user-schemas:/opt/nifi/user-schemas:rw
 
       # INFO: uncomment below to map security certificates if need to secure NiFi endpoints 
-      - ./${NIFI_SECURITY_DIR:-../security/nifi_certificates/}:/opt/nifi/security:ro
+      - ./${NIFI_SECURITY_DIR:-../security/nifi_certificates/}:/opt/nifi-current/nifi_certificates:ro
       - ./${ELASTICSEARCH_SECURITY_DIR:-../security/es_certificates/}:/opt/nifi/nifi-current/es_certificates:ro
       - ../security/nifi_certificates/localhost/keystore.jks:/opt/nifi/nifi-current/conf/keystore.jks
       - ../security/nifi_certificates/localhost/truststore.jks:/opt/nifi/nifi-current/conf/truststore.jks
+      # Security credentials scripts
+      - ../security/nifi_create_single_user_auth.sh:/opt/nifi/nifi-current/security_scripts/nifi_create_single_user_auth.sh:ro
 
       # # Nifi properties file:
       - ../nifi/conf/:/opt/nifi/nifi-current/conf/:rw
@@ -981,9 +984,10 @@ services:
       - nifi-vol-database:/opt/nifi/nifi-current/database_repository
       - nifi-vol-flowfiles:/opt/nifi/nifi-current/flowfile_repository
       - nifi-vol-content:/opt/nifi/nifi-current/content_repository
+      - nifi-vol-state:/opt/nifi/nifi-current/state
+
       # errors generated during data processing
       - nifi-vol-errors:/opt/nifi/pipeline/flowfile-errors
-      - nifi-vol-state:/opt/nifi/nifi-current/state
     extra_hosts:
       - ${ELASTICSEARCH_1_HOST_NAME:-test-1:0.0.0.0}
       - ${ELASTICSEARCH_2_HOST_NAME:-test-2:0.0.0.0}

docs/deploy/services.md

@@ -255,9 +255,11 @@ Alternatively, please refer to [the official Apache NiFi documentation](https://
 #### ENV/CONF files:
 - `/deploy/nifi.env` - most notable settings are related to port mapping and proxy
 - `/security/certificates_nifi.env` - define NiFi certificate settings here
-
+- `/security/nifi_users.env` - defines the NiFi user credentials for single user auth & others
 More configuration options are covered in [nifi-doc](../nifi/main.md).
 
+Other `.env` files are mounted but those are only useful for custom scripts where you plan to use certain vars from other services, check the `services.yml` nifi `env-file` section definition.
+
 ## Tika Service
 
 `tika-service` provides document text extraction functionality of [Apache Tika](https://tika.apache.org/).

docs/nifi/main.md

@@ -213,42 +213,7 @@ This custom image will use embedeed Zookeeper within NiFi service and will use t
 
 In previous nifi versions by default there was no user assigned and authentication was anonymous. Since 1.14.0 this changed. So now we have HTTPS enabled by default via port 8443 (configurable in nifi.properties and the services.yml file).
 
-Before starting the NIFI container it's important to take note of the following things if we wish to enable HTTPS functionality:
-
-- this step is optional (as you might have done it before from configuring other certificates), run `create_root_ca_cert.sh` to create the ROOT certificates, these will be used by NiFi/OpenSearch/OCR_service/Tika/MedcatService/Jupyterhub etc.
-
-- the `nifi_toolkit_security.sh` script is used to download the nifi toolkit and generate new certificates and keys that are used by the container, take note that inside the `localhost` folder there is another nifi.properties file that is generated, we must look to the following setttings which are generated randomly and copy them to the `nifi/conf/nifi.properties` file. 
-- the trust/store keys generated for production will be in the `nifi_certificates/localhost` folder and  the `nifi-cert.pem` + `nifi-key.key` files. in the baes `nifi_certificates` folder.
-
-- as port of the security process the `nifi.sensitive.props.key` should be set to a random string or a password of minimum 12 characters. Once this is set do NOT modify it as all the other sensitive passwords will be hashed with this string. By default this is set to <strong>```cogstackNiFipass```</strong>
-
-Example:
-```
-    nifi.security.keystorePasswd=ZFD4i4UDvod8++XwWzTg+3J6WJF6DRSZO33lbb7hAgc
-    nifi.security.keyPasswd=ZFD4i4UDvod8++XwWzTg+3J6WJF6DRSZO33lbb7hAgc
-    nifi.security.truststore=./conf/truststore.jks
-    nifi.security.truststoreType=jks
-    nifi.security.truststorePasswd=lzMGadNB1JXQjgQEnFStLiNkJ6Wbbgw0bFdCTICKtKo
-```
-
-### `nifi-nginx service`
-In order to be able to properly access the nifi instance securely, you also need to start the nifi-nginx container as it is configured to provide access from any source to nifi, available at https://localhost:8443/nifi .
-
-### Access via user account
-<strong>This is entirely optional, if you have configered the security certs as described in ```security/README.md``` then you are good to go.</strong>
-<br>
-Default username : 
-<br>
-```
-username: admin     
-password: admincogstacknifi
-```
-
-- the `login-identity-providers.xml` file in `/nifi/conf/` stores the password for the user account, to generate a password one must use the following command within the container : `/opt/nifi/nifi-current/bin/nifi.sh set-single-user-credentials USERNAME PASSWORD`, once done, you would need to copy the file from `/opt/nifi/nifi-current/conf/login-identity-providers.xml` locally with docker cp and replace the one in the `nifi/conf` folder and rebuild the container.
-
-URL: https://localhost:8443/nifi/login
-
-Troubleshooting Security : if you encounter errors related to sensitive key properties not being set please clear/delete the docker volumes of the nifi container or delete all volumes of inactive containers `docker volume prune`.
+Please use the guide provided in the [SECURITY.md](../security.md#apache-nifi) section to set up accounts and certificates.
 
 ## Drivers
 The drivers are provided in [`drivers`](https://github.com/CogStack/CogStack-NiFi/tree/master/nifi/drivers) directory.

docs/security.md

@@ -31,6 +31,8 @@ The information provided in this README hence should be only considered as a hin
 ├── es_roles <-------------------------- This folder stores Elasticsearch native/Opensearch account roles and role_mappings.
 ├── nginx_users.env <------------------- Nginx users
 ├── nifi_certificates <----------------- Location of NiFi cerficiates post-generation.
+├── nifi_init_create_user_auth.sh <----- Script used to start the NiFi container for singler user account creation
+├── nifi_create_single_user_auth.sh <--- Script used create single user credentials for NiFi (executed inside the container)
 ├── nifi_toolkit_security.sh <---------- Script for generating NiFi certificates
 ├── root-ca-truststore.key <------------ all `root-ca` files  are generated by the `create_root_ca_cert.sh` script
 ├── root-ca.key <------------------------|
@@ -243,10 +245,51 @@ For securing Apache NiFi endpoint with self-signed certificates please refer to
 Regarding connecting to services that use self-signed certificates (such as Elasticsearch), it is required that these certificates use JKS keystore format.
 The certificates can be generated using `create_keystore.sh`. Usage: bash create_keystore.sh <cert_name> <jks_store> <password> | the password is optional.
 
+Before starting the NIFI container it's important to take note of the following things if we wish to enable HTTPS functionality:
+
+- this step is optional (as you might have done it before from configuring other certificates), run `create_root_ca_cert.sh` to create the ROOT certificates, these will be used by NiFi/OpenSearch/OCR_service/Tika/MedcatService/Jupyterhub etc.
+
+- the `nifi_toolkit_security.sh` script is used to download the nifi toolkit and generate new certificates and keys that are used by the container, take note that inside the `localhost` folder there is another nifi.properties file that is generated, we must look to the following setttings which are generated randomly and copy them to the `nifi/conf/nifi.properties` file. 
+- the trust/store keys generated for production will be in the `nifi_certificates/localhost` folder and  the `nifi-cert.pem` + `nifi-key.key` files. in the baes `nifi_certificates` folder.
+
+- as port of the security process the `nifi.sensitive.props.key` should be set to a random string or a password of minimum 12 characters. Once this is set do NOT modify it as all the other sensitive passwords will be hashed with this string. By default this is set to <strong>```cogstackNiFipass```</strong>
+
+Example:
+```
+    nifi.security.keystorePasswd=ZFD4i4UDvod8++XwWzTg+3J6WJF6DRSZO33lbb7hAgc
+    nifi.security.keyPasswd=ZFD4i4UDvod8++XwWzTg+3J6WJF6DRSZO33lbb7hAgc
+    nifi.security.truststore=./conf/truststore.jks
+    nifi.security.truststoreType=jks
+    nifi.security.truststorePasswd=lzMGadNB1JXQjgQEnFStLiNkJ6Wbbgw0bFdCTICKtKo
+```
+
+### Setting up access via user account (SINGLE USER CREDETIAL)
+<strong>This is entirely optional, if you have configered the security certs as described in ```security/README.md``` then you are good to go.</strong>
+<br>
+Default username : 
+<br>
+```
+username: admin     
+password: cogstackNiFi
+```
+
+- the `login-identity-providers.xml` file in `/nifi/conf/` stores the password for the user account, to generate a password one must use the following command within the container : `/opt/nifi/nifi-current/bin/nifi.sh set-single-user-credentials USERNAME PASSWORD`, once done, you would need to copy the file from `/opt/nifi/nifi-current/conf/login-identity-providers.xml` locally with docker cp and replace the one in the `nifi/conf` folder and rebuild the container.
+
+- alternative to the above step: go into the `/security` folder, set the desired nifi username & password in the `/security/nifi_users.env` file. Make sure to STOP any running NiFi containers `docker stop cogstack-nifi` and execute the following script: `bash /security/nifi_init_create_user_auth.sh`, this script will start a NiFi container for the time of the account creation and then remove itself, after it finishes, go back to the `/deploy` folder and start your NiFi container, all should be working! 
+
+URL: https://localhost:8443/nifi/login
+
+Troubleshooting Security : if you encounter errors related to sensitive key properties not being set please clear/delete the docker volumes of the nifi container or delete all volumes of inactive containers `docker volume prune`.
+
+### Disabling the login screen
+If for some reason you do not wish to authenticate every time you connect to NiFi, you can enable the client certificates in the [nginx.conf](../services/nginx/config/nginx.conf) line 86-87 and delete the commented lines.
+
+## `nifi-nginx`
 
-## NGINX
 Alternatively, one can secure the access to selected services by using NGINX reverse proxy.
 This may be essential in case some of the web services that need to be exposed to end-users do not offer SSL encryption. 
 See [the official documentation](https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/) for more details on using NGINX for that.
 
-Nginx only requires the root-CA certificate by default, so use the above [generate cert](#generating-the-base-certificates-for-nifinginxjupyterhubocr-servicetikamedcat-service-certificates) section to create it.
+Nginx only requires the root-CA certificate by default, so use the above [generate cert](#generating-the-base-certificates-for-nifinginxjupyterhubocr-servicetikamedcat-service-certificates) section to create it.
+
+In order to be able to properly access the nifi instance securely, you also need to start the nifi-nginx container as it is configured to provide access from any source to nifi, available at https://localhost:8443/nifi .

nifi/conf/login-identity-providers.xml

@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="UTF-8"?><!--
+<?xml version="1.0" encoding="UTF-8" standalone="no"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
@@ -29,8 +29,8 @@
     <provider>
         <identifier>single-user-provider</identifier>
         <class>org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider</class>
-        <property name="Username">a6510d3e-25e1-4f70-8a09-be894c9af8f7</property>
-        <property name="Password">$2b$12$J15HtovNfS7hez6ao0e15.hbzlYzt.EVq4rw6DJGE5uJv5Z/b80za</property>
+        <property name="Username">admin</property>
+        <property name="Password">$2b$12$xPMAjNXTFFeppRPM2rUiiuYaH4vOimlBGUbejJWF9UNkv1afGhiSC</property>
     </provider>
     <!--
         Identity Provider for users logging in with username/password against an LDAP server.

nifi/conf/nifi.properties

@@ -136,7 +136,7 @@ nifi.status.repository.questdb.persist.component.days=3
 nifi.status.repository.questdb.persist.location=./status_repository
 
 # Site to Site properties
-nifi.remote.input.host=3d0f2d1a9d62
+nifi.remote.input.host=35becaf53b3d
 nifi.remote.input.secure=true
 nifi.remote.input.socket.port=10000
 nifi.remote.input.http.enabled=true
@@ -159,7 +159,7 @@ nifi.web.http.network.interface.default=
 
 #############################################
 
-nifi.web.https.host=3d0f2d1a9d62
+nifi.web.https.host=35becaf53b3d
 nifi.web.https.port=8443
 nifi.web.https.network.interface.default=
 nifi.web.jetty.working.directory=./work/jetty
@@ -263,7 +263,7 @@ nifi.cluster.protocol.is.secure=true
 
 # cluster node properties (only configure for cluster nodes) #
 nifi.cluster.is.node=false
-nifi.cluster.node.address=3d0f2d1a9d62
+nifi.cluster.node.address=35becaf53b3d
 nifi.cluster.node.protocol.port=
 nifi.cluster.node.protocol.max.threads=50
 nifi.cluster.node.event.history.size=25

security/nifi_create_single_user_auth.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+set -e
+
+if [ -n "${NIFI_SINGLE_USER_CREDENTIALS_USERNAME}" ] && [ -n "
+    ${NIFI_SINGLE_USER_CREDENTIALS_PASSWORD}" ]; then
+    ${NIFI_HOME}/bin/nifi.sh set-single-user-credentials "${NIFI_SINGLE_USER_CREDENTIALS_USERNAME}" "${NIFI_SINGLE_USER_CREDENTIALS_PASSWORD}"
+fi

security/nifi_init_create_user_auth.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+set -e
+
+echo "Deleting the current nifi container"
+docker container rm -f $(docker ps -a -q --filter name="cogstack-nifi*") || true
+
+echo "starting up the nifi container for script execution ..." 
+docker-compose -f ../deploy/services.yml up -d nifi
+
+echo "Creating NiFi single-user-credentials from nifi_users.env ..."
+docker exec -it cogstack-nifi /bin/bash /opt/nifi/nifi-current/security_scripts/nifi_create_single_user_auth.sh
+
+echo "Deleting the nifi container"
+docker container rm -f $(docker ps -a -q --filter name="cogstack-nifi*") || true

security/nifi_users.env

@@ -0,0 +1,4 @@
+
+# used by single-user-authentication
+NIFI_SINGLE_USER_CREDENTIALS_USERNAME=admin
+NIFI_SINGLE_USER_CREDENTIALS_PASSWORD=cogstackNiFi

services/nginx/config/nginx.conf

@@ -82,8 +82,9 @@ http {
         ssl_certificate /etc/nginx/root-ca.pem;
         ssl_certificate_key /etc/nginx/root-ca.key;
 
-        proxy_ssl_certificate /etc/nginx/nifi-cert.pem;
-        proxy_ssl_certificate_key /etc/nginx/nifi-key.key;
+        # if you comment this out then you will automatically enable authentication, because nginx won't certify your session...
+        # proxy_ssl_certificate /etc/nginx/nifi-cert.pem;
+        # proxy_ssl_certificate_key /etc/nginx/nifi-key.key;
 
         proxy_ssl_trusted_certificate /etc/nginx/nifi-cert.pem;
         proxy_ssl_server_name on;