Version 1.1.0 (#13)

## v1.1.0
|||
|---|---|
|Date|2021-08-18
|Kind|Bugfix / Feature
|Author|[email protected]
- Features
  - Added **DNS** and **PROXY** feeds to ETP Input (<3 Sara)
- Minor improvements
  - Version number fix (Stated 0.9.0 instead of 1.x.x)
  - debug "message" fix ( changed HTTP to HTTP(S) to avoid misunderstanding)
  - documented workaround for discovered proxy issue
  - enabled json highlighting in [Log_overview](./LOG_OVERVIEW.md)
  - added better error guidance when basic stuff is unset (input / output)
  - moved docker-compose from root dir to /docs
  - added `read_only: true` to the docker-compose.yml files (security enhancement)

README.md

@@ -38,16 +38,18 @@ It can be run directly as Python code, as a provided Docker container or through
     - [Enterprise Threat Protectors (ETP)](https://www.akamai.com/us/en/products/security/enterprise-threat-protector.jsp)
       - [THREAT](docs/LOG_OVERVIEW.md#threat-log-threat)
       - [AUP](docs/LOG_OVERVIEW.md#accceptable-use-policy-logs-aup)
+      - [DNS](docs/LOG_OVERVIEW.md#dns)
+      - [PROXY](docs/LOG_OVERVIEW.md#proxy)
     - [Akamai Phish-proof Multi Factor Authenticator (AKAMAI-MFA)](https://www.akamai.com/us/en/products/security/akamai-mfa.jsp)
       - [AUTH](docs/LOG_OVERVIEW.md#authentication-logs-auth)
       - [POLICY](docs/LOG_OVERVIEW.md#policy-logs-policy)
 
 
 - Supported data outputs
-    - TCP Socket (tcp://host:port)
-    - UDP Socket (udp://host:port)
-    - HTTP(S) URL (http(s)://host:port/path) (supporting Authentication)
-    - RAW (>STDOUT)
+    - TCP Socket (tcp://host:port) `--output tcp`
+    - UDP Socket (udp://host:port) `--output udp`
+    - HTTP(S) URL (http(s)://host:port/path) (supporting Authentication) `--output http`
+    - RAW (>STDOUT) `--output raw`
 
 
 - Operation types

bin/config/global_config.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
 # Common global variables / constants
-__version__ = "0.9.0"
+__version__ = "1.1.0"
 __tool_name_long__ = "Akamai Unified Log Streamer"
 __tool_name_short__ = "ULS"
 
@@ -17,9 +17,13 @@
 bin_eaa_cli = "ext/cli-eaa/bin/akamai-eaa"
         # Available EAA CLI feeds
 eaa_cli_feeds = ['ACCESS', 'ADMIN', 'CONHEALTH']
+
     # ETP
-bin_etp_cli = "ext/cli-etp/bin/akamai-etp"      # Path to the ETP CLI Executable
-etp_cli_feeds = ['THREAT', 'AUP']               # Available ETP CLI feeds
+        # Path to the ETP CLI Executable
+bin_etp_cli = "ext/cli-etp/bin/akamai-etp"
+        # Available ETP CLI feeds
+etp_cli_feeds = ['THREAT', 'AUP', 'DNS', 'PROXY']
+
     # MFA
 bin_mfa_cli = "ext/cli-mfa/bin/akamai-mfa"      # Path to the MFA CLI Executable
 mfa_cli_feeds = ['POLICY', 'AUTH']              # Available MFA CLI feeds

bin/modules/UlsArgsParser.py

@@ -70,7 +70,9 @@ def init():
                              dest='inproxy',
                              type=str,
                              default=(os.environ.get('ULS_INPUT_PROXY') or None),
-                             help="Use a proxy Server for the INPUT requests (fetching data from AKAMAI API'S)")
+                             help=argparse.SUPPRESS)
+                            # We're surpressing this for now, as the param does not seem to work (mschiess-20210818 - see EME-498)
+                             #help="Use a proxy Server for the INPUT requests (fetching data from AKAMAI API'S)")
     # RAWCMD
     input_group.add_argument('--rawcmd',
                              action='store',

bin/modules/UlsInputCli.py

@@ -43,10 +43,10 @@ def _feed_selector(self, feed, product_feeds):
         if feed in product_feeds:
             # feed matches the given list
             aka_log.log.debug(f'{self.name} - selected feed: {feed}')
-        elif not feed:
+        elif not feed or feed == "DEFAULT":
             # Set default (first of feeds)
             feed = product_feeds[0]
-            aka_log.log.debug(f'{self.name} - using default feed: {feed}')
+            aka_log.log.warning(f'{self.name} - using default feed: {feed}')
         else:
             aka_log.log.critical(
                 f"{self.name} - Feed ({feed}) not available - Available: {product_feeds}")

bin/modules/UlsOutput.py

@@ -135,7 +135,7 @@ def connect(self, output_type: str, host: str, port: int,
                                              f'Use --httpurl instead of --host / --port')
                         sys.exit(1)
                     else:
-                        aka_log.log.debug(f"{self.name} attempting to connect via HTTP to {http_url} ")
+                        aka_log.log.debug(f"{self.name} attempting to connect via HTTP(S) to {http_url} ")
 
                     # Let'S do an options request
                     self.http_url = http_url

bin/modules/UlsTools.py

@@ -112,3 +112,17 @@ def uls_check_edgerc(configfile, configsection, configvalues):
         else:
             aka_log.log.debug(f"Required configuration value '{configvalue}' found.")
     return 0
+
+
+def uls_check_args(input, output):
+    missing = None
+    if not input:
+        missing = "INPUT"
+    elif not output:
+        missing = "OUTPUT"
+    if missing:
+        aka_log.log.critical(f"Required argument / ENV var not set: {missing}")
+        aka_log.log.critical(f"Please run `bin/uls.py --help` for additional information")
+        sys.exit(1)
+    else:
+        return 0

bin/uls.py

@@ -62,6 +62,9 @@ def main():
     # Load the LOG system
     aka_log.init(uls_args.loglevel, uls_config.__tool_name_short__)
 
+    # Verify the given core params (at least input and output should be set)
+    UlsTools.uls_check_args(uls_args.input, uls_args.output)
+
     # Check CLI Environment
     UlsTools.uls_check_sys()
 

docs/AKAMAI_API_CREDENTIALS.md

@@ -27,6 +27,8 @@ This document describes how to create Akamai API credentials and configure them
 |Enterprise Application Access|EAA|HEALTH|[{OPEN} API / Enterprise Application Access](#eaa-open-api-for-connector-health-feed)|
 |Enterprise Threat Protector|ETP|THREAT|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
 |Enterprise Threat Protector|ETP|AUP|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
+|Enterprise Threat Protector|ETP|DNS|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
+|Enterprise Threat Protector|ETP|PROXY|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
 |Akamai MFA|MFA|AUTH|[MFA Integration](#mfa-integration-for-logging)|
 |Akamai MFA|MFA|POLICY|[MFA Integration](#mfa-integration-for-logging)|
 

docs/ARGUMENTS_ENV_VARS.md

@@ -14,9 +14,9 @@ The following tables list all available command line parameters and their corres
 |Parameter|Env - Var|Options|Default|Description|
 |---|---|---|---|---|
 |-i <br> --input | ULS_INPUT | 'EAA', 'ETP', 'MFA' | None | Specify the desired INPUT source |
-|--feed | ULS_FEED | EAA: 'ACCESS', 'ADMIN', 'CONHEALTH'<br> ETP: 'THREAT', 'AUP'<br> MFA: 'AUTH','POLICY' | None | Specify the desired INPUT feed |
+|--feed | ULS_FEED | EAA: 'ACCESS', 'ADMIN', 'CONHEALTH'<br> ETP: 'THREAT', 'AUP', 'DNS', 'PROXY'<br> MFA: 'AUTH','POLICY' | None | Specify the desired INPUT feed |
 |--format | ULS_FORMAT | 'JSON', 'TEXT' | JSON | Specify the desired INPUT (=OUTPUT) format |
-|--inproxy<br>--inputproxy | ULS_INPUT_PROXY | HOST:PORT| None | Adjust proxy usage for INPUT data collection (cli) |
+|--inproxy<br>--inputproxy | ULS_INPUT_PROXY | HOST:PORT| None | Adjust proxy usage for INPUT data collection (cli) <br>There is a known issue in the usage, [please read more about it here](./FAQ.md#--inputproxy-proxy-does-not-work-as-expected)|
 |--rawcmd | ULS_RAWCMD | \<cli command\> | None | USE with caution /!\ <br> This is meant only to be used when told by AKAMAI [Click here for more information](ADDITIONAL_FEATURES.md#rawcmd---rawcmd-feature)|
 |--edgerc | ULS_EDGERC | /path/to/your/.edgerc | '~/.edgerc' | Specify the location of the .edgerc EDGE GRID AUTH file |
 |--section | ULS_SECTION | edgerc_config_section | 'default' | Specify the desired section within the .edgerc file |

docs/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Version History
 
+## v1.1.0
+|||
+|---|---|
+|Date|2021-08-18
+|Kind|Bugfix / Feature
+|Author|[email protected]
+- Features
+  - Added **DNS** and **PROXY** feeds to ETP Input (<3 Sara)
+- Minor improvements
+  - Version number fix (Stated 0.9.0 instead of 1.x.x)
+  - debug "message" fix ( changed HTTP to HTTP(S) to avoid misunderstanding)
+  - documented workaround for discovered proxy issue
+  - enabled json highlighting in [Log_overview](./LOG_OVERVIEW.md)
+  - added better error guidance when basic stuff is unset (input / output)
+  - moved docker-compose from root dir to /docs
+  - added `read_only: true` to the docker-compose.yml files (security enhancement)
+
 ## v1.0.0
 |||
 |---|---|

docs/DOCKER-COMPOSE_USAGE.md

@@ -52,8 +52,8 @@ docker compose up -d
   This will run the "simple" use case in foreground. 
   The `docker-compose.yml` file will reference the `etp-threat.env` and provide the configuration from that file.  
   **Files:**  
-    - [docker-compose.yml](../docker-compose/simple/docker-compose.yml)
-    - [etp-threat.env](../docker-compose/simple/etp-threat.env)
+    - [docker-compose.yml](docker-compose/simple/docker-compose.yml)
+    - [etp-threat.env](docker-compose/simple/etp-threat.env)
 
 
 - Complex docker-compose setup delivering different streams to different endpoints
@@ -63,7 +63,7 @@ docker compose up -d
   ```
   This triggers a more complex setup consisting out of 3 different data feeds.  
   **Files:**  
-    - [docker-compose.yml](../docker-compose/complex/docker-compose.yml)
-    - [etp-threat.env](../docker-compose/complex/etp-threat.env)
-    - [eaa-admin.env](../docker-compose/complex/eaa-access.env)
-    - [eaa-access.env](../docker-compose/complex/eaa-access.env)
+    - [docker-compose.yml](docker-compose/complex/docker-compose.yml)
+    - [etp-threat.env](docker-compose/complex/etp-threat.env)
+    - [eaa-admin.env](docker-compose/complex/eaa-access.env)
+    - [eaa-access.env](docker-compose/complex/eaa-access.env)

docs/FAQ.md

@@ -7,6 +7,7 @@
 - [What command line Options are available ? ](#what-command-line-options-are-available-)
 - [What environmental variables (ENV VARS) are available](#what-environmental-variables-env-vars-are-available-#)
 - [--version does not show all versions](#ulspy---version-does-not-show-all-versions)
+- [--inputproxy <proxy> does not work as expected](#--inputproxy-proxy-does-not-work-as-expected)
 
 ----
 ## FAQ
@@ -38,11 +39,22 @@ There is a dedicated document explaining the [command line parameters and enviro
 There is a dedicated document explaining the [command line parameters and environment variables.](ARGUMENTS_ENV_VARS.md)
 
 ---
-<<<<<<< HEAD
 ### `uls.py --version` does not show all versions
 This is (sadly) a known issue. It is a problem within some of the CLI's if no ".edgerc" file is provided. If you provide a `.edgerc`, the show is correct.
 
 ---
 
-=======
->>>>>>> 2d20b502da2fcc131088bf0498ddcf56a12d531d
+### `--inputproxy <proxy>` does not work as expected
+This is (sadly) a known issue.  
+The good news is we do have a proper workaround for this.
+Instead of setting the Option `--inputproxy <proxy>` or the ENV var `ULS_INPUT_PROXY` do the following:
+
+Set the ENV following ENV vars to your environment / container.
+```text
+HTTP_PROXY=http://your.proxy.internal:3128"
+HTTPS_PROXY=http://your.proxy.internal:3128"
+NO_PROXY="localhost,127.0.0.1,::1"
+```
+Those can also be added to the .evn file when using docker / docker-compose.  
+**Please ensure, you are ADDING YOUR SIEM HOST IP to the NO_PROXY line when the SIEM is internal to avoid issues**  
+`NO_PROXY="localhost,127.0.0.1,::1,my_siem.host"`