[New Hunt] Add Initial Linux Hunting Files (elastic#3847)

* added 'Uncommon Process Execution from Suspicious Directory' hunt * adds all linux hunting files * moves linux hunting files to queries folder * adds generated docs * fixing windows hunts * fixing windows hunts * updated README * Removed 2, updated a few, changed some names/descriptions and added list of str * updated windows for language schema changes, regenerated docs; updated README and index * changed UUIDs to hex only with standard hyphen format * removing unecessary docs * Fixed queries based on Samir feedback * ++ * regenerating linux docs * Update hunting/linux/queries/command_and_control_via_network_connections_with_low_occurrence_frequency_for_unique_agents.toml Co-authored-by: Samirbous <[email protected]> * Update hunting/linux/queries/defense_evasion_via_hidden_process_execution.toml Co-authored-by: Samirbous <[email protected]> * Update hunting/linux/queries/command_and_control_via_unusual_file_downloads_from_source_addresses.toml Co-authored-by: Samirbous <[email protected]> * Update hunting/linux/queries/defense_evasion_via_capitalized_process_execution.toml Co-authored-by: Samirbous <[email protected]> * Update hunting/linux/queries/defense_evasion_via_hidden_process_execution.toml Co-authored-by: Samirbous <[email protected]> * Updates * Update * Update hunting/linux/queries/command_and_control_via_network_connections_with_low_occurrence_frequency_for_unique_agents.toml Co-authored-by: Samirbous <[email protected]> * Updates * regenerating linux docs --------- Co-authored-by: Ruben Groenewoud <[email protected]> Co-authored-by: Samirbous <[email protected]>
shashank-elastic · Jul 5, 2024 · f0b2cb7 · f0b2cb7
1 parent 215d5a0
commit f0b2cb7
Show file tree

Hide file tree

Showing 137 changed files with 3,930 additions and 178 deletions.
diff --git a/hunting/README.md b/hunting/README.md
@@ -2,41 +2,44 @@
 
 ---
 
-Welcome to the `hunting` folder within the `detection-rules` repository! This directory houses a curated collection of threat hunting queries designed to enhance security monitoring and threat detection capabilities using the Elastic Stack. Each file in this directory provides a query tailored for identifying specific security threats or suspicious activities.
+Welcome to the `hunting` folder within the `detection-rules` repository! This directory houses a curated collection of threat hunting queries designed to enhance security monitoring and threat detection capabilities using the Elastic Stack. Each file in this directory provides a query tailored for the initial evidence gathering of specific hunts.
 
-These queries are designed for use with the Elastic Security platform, part of the broader Elastic Stack, enabling security teams to proactively hunt for potential threats in their environment.
+Each hunt has a designated TOML and Markdown file, intended to be used either programatically or via copy and pasted. Notes about data considerations, pivoting, exploring data further and more have been added to each hunting query. These queries are designed for use with the Elastic Security platform, part of the broader Elastic Stack, enabling security teams to proactively hunt for potential threats in their environment.
 
-- KQL
-- EQL
-- ES|QL
-- OsQuery
-- YARA
+Note that some hunting files will include a mix of queries with different languages whose sole purpose is to provide optional queries to gather evidence for the hunt.
+
+- [KQL](https://www.elastic.co/guide/en/kibana/current/kuery-query.html)
+- [EQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql.html)
+- [ES|QL](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html)
+- [OsQuery/SQL](https://www.elastic.co/guide/en/kibana/current/osquery.html)
+- [YARA](https://yara.readthedocs.io/en/stable/writingrules.html)
+
+The hunting queries shared in this folder are a mix of the following hunting methods:
+
+- Hypothesis-Driven - Assumed breach method with specific hypothesis of where adversary dwells or where footprints exist.
+- CTI-Driven - Retro-active searches for specific indicators-of-compromise or tactics, techniques and procedures (TTPs) related to adversaries and/or tooling.
+- Data-Driven - Initial evidence collecting query that requires more advanced data analysis to uncover anomalies.
 
 ## How to Contribute
 
 Contributing to the `hunting` folder is a great way to share your expertise and enhance the security community's capabilities. Here’s how you can contribute:
 
 ### Names and Related Queries
 
-All query names should be unique and descriptive. If a query's intent is identical or related to another query, consider 
-adding a suffix with the integration(s) to the name to indicate the relationship and distinguish them from each other. 
+All query names should be unique and descriptive. If a query's intent is identical or related to another query, consider
+adding a suffix with the integration(s) to the name to indicate the relationship and distinguish them from each other.
 Otherwise, the names do not require the integration, since it is already annotated within the `integration` field.
 
-The filename should reflect the query name.
-
-For example:
-- `Detect DLL Hijack via Masquerading as Microsoft Native Libraries - Elastic Defend`
-- `Detect DLL Hijack via Masquerading as Microsoft Native Libraries - Sysmon`
-
 ### Adding New Queries
 - **TOML File Naming and Organization**: Ensure that any new queries are named descriptively and grouped by the type of threat they address. Place your TOML files inside the `queries` folder and ensure they are named in a way that reflects the nature of the threat or behavior they are designed to detect.
 - **TOML Fields**: To ensure the hunt queries are consistent and comprehensive, it's important to structure the threat detection rules with specific fields. When contributing a new rule, please include the following fields in the TOML file to describe and configure the analytic:
   - **author**: The name of the individual or organization authoring the rule.
+  - **description**: The purpose of the hunt with a clear threat explanation and hunting goal.
   - **integration**: The specific integration or data source the rule applies to, such as `aws_bedrock.invocation`.
   - **uuid**: A unique identifier for the rule to maintain version control and tracking.
   - **name**: A descriptive name for the rule that clearly indicates its purpose.
-  - **language**: The query language used in the rule, such as `KQL`, `EQL`, `ES|QL`, `OsQuery`, or `YARA`.
-  - **query**: The actual query or analytic expression written in the appropriate query language that executes the detection logic.
+  - **language**: The query language(s) used in the rule, such as `KQL`, `EQL`, `ES|QL`, `OsQuery`, or `YARA`.
+  - **query**: An array of actual queries or analytic expressions written in the appropriate query language that executes the detection logic.
   - **notes**: An array of strings providing detailed insights into the rationale behind the rule, suggestions for further investigation, and tips on distinguishing false positives from true activity.
   - **mitre**: Reference to applicable MITRE ATT&CK tactics or techniques that the rule addresses, enhancing the contextual understanding of its security implications.
   - **references**: Links to external documents, research papers, or websites that provide additional information or validation for the detection logic.

diff --git a/hunting/generate_markdown.py b/hunting/generate_markdown.py
@@ -32,7 +32,7 @@ class Hunt:
     integration: list[str]
     uuid: str
     name: str
-    language: str
+    language: list[str]
     license: str
     query: list[str]
     notes: Optional[List[str]] = field(default_factory=list)
@@ -85,7 +85,7 @@ def convert_toml_to_markdown(hunt_config: Hunt, file_path: Path) -> str:
     markdown += f"- **Description:** {hunt_config.description}\n"
     markdown += f"- **UUID:** `{hunt_config.uuid}`\n"
     markdown += f"- **Integration:** {", ".join(generate_integration_links(hunt_config.integration))}\n"
-    markdown += f"- **Language:** `{hunt_config.language}`\n\n"
+    markdown += f"- **Language:** `{hunt_config.language}`\n\n".replace("'", "").replace('"', "")
     markdown += "## Query\n\n"
     for query in hunt_config.query:
         markdown += f"```sql\n{query}```\n\n"
@@ -127,7 +127,7 @@ def process_toml_files(base_path: Path) -> None:
         index_content += f"\n\n## {folder}\n"
         for file_path, rule_name, language in sorted(files):
             index_path = "./" + str(file_path)
-            index_content += f"- [{rule_name}]({index_path}) ({language})\n"
+            index_content += f"- [{rule_name}]({index_path}) ({", ".join(language)})\n"
 
     # Write the index file at the base directory level
     index_path = base_path / "index.md"

diff --git a/hunting/index.md b/hunting/index.md
@@ -2,6 +2,39 @@
 
 Here are the queries currently available:
 
+## linux
+- [Network Connections with Low Occurrence Frequency for Unique Agent ID](./linux/docs/command_and_control_via_network_connections_with_low_occurrence_frequency_for_unique_agents.md) (ES|QL)
+- [Unusual File Downloads from Source Addresses](./linux/docs/command_and_control_via_unusual_file_downloads_from_source_addresses.md) (ES|QL)
+- [Defense Evasion via Capitalized Process Execution](./linux/docs/defense_evasion_via_capitalized_process_execution.md) (ES|QL)
+- [Hidden Process Execution](./linux/docs/defense_evasion_via_hidden_process_execution.md) (ES|QL)
+- [Potential Defense Evasion via Multi-Dot Process Execution](./linux/docs/defense_evasion_via_multi_dot_process_execution.md) (ES|QL)
+- [Excessive SSH Network Activity to Unique Destinations](./linux/docs/excessive_ssh_network_activity_unique_destinations.md) (ES|QL)
+- [Uncommon Process Execution from Suspicious Directory](./linux/docs/execution_uncommon_process_execution_from_suspicious_directory.md) (ES|QL)
+- [Logon Activity by Source IP](./linux/docs/login_activity_by_source_address.md) (ES|QL)
+- [Low Volume External Network Connections from Process by Unique Agent](./linux/docs/low_volume_external_network_connections_from_process.md) (ES|QL)
+- [Low Volume GTFOBins External Network Connections](./linux/docs/low_volume_gtfobins_external_network_connections.md) (ES|QL)
+- [Low Volume Modifications to Critical System Binaries by Unique Host](./linux/docs/low_volume_modifications_to_critical_system_binaries.md) (ES|QL)
+- [Low Volume Process Injection-Related Syscalls by Process Executable](./linux/docs/low_volume_process_injection_syscalls_by_executable.md) (ES|QL)
+- [Persistence Through Reverse/Bind Shells](./linux/docs/persistence_reverse_bind_shells.md) (SQL)
+- [Persistence via Cron](./linux/docs/persistence_via_cron.md) (ES|QL, SQL)
+- [Drivers Load with Low Occurrence Frequency](./linux/docs/persistence_via_driver_load_with_low_occurrence_frequency.md) (ES|QL)
+- [Git Hook/Pager Persistence](./linux/docs/persistence_via_git_hook_pager.md) (ES|QL, SQL)
+- [Persistence via Message-of-the-Day](./linux/docs/persistence_via_message_of_the_day.md) (ES|QL, SQL)
+- [Persistence via Packager Manager](./linux/docs/persistence_via_package_manager.md) (ES|QL, SQL)
+- [Persistence via rc.local/rc.common](./linux/docs/persistence_via_rc_local.md) (ES|QL, SQL)
+- [Shell Modification Persistence](./linux/docs/persistence_via_shell_modification_persistence.md) (ES|QL, SQL)
+- [Persistence via SSH Configurations and/or Keys](./linux/docs/persistence_via_ssh_configurations_and_keys.md) (SQL)
+- [Persistence via Systemd (Timers)](./linux/docs/persistence_via_systemd_timers.md) (ES|QL, SQL)
+- [Persistence via Udev](./linux/docs/persistence_via_udev.md) (ES|QL, SQL)
+- [Unusual System Binary Parent (Potential System Binary Hijacking Attempt)](./linux/docs/persistence_via_unusual_system_binary_parent.md) (ES|QL)
+- [Privilege Escalation/Persistence via User/Group Creation and/or Modification](./linux/docs/persistence_via_user_group_creation_modification.md) (SQL)
+- [XDG Persistence](./linux/docs/persistence_via_xdg_autostart_modifications.md) (ES|QL, SQL)
+- [Privilege Escalation Identification via Existing Sudoers File](./linux/docs/privilege_escalation_via_existing_sudoers.md) (SQL)
+- [Process Capability Hunting](./linux/docs/privilege_escalation_via_process_capabilities.md) (ES|QL)
+- [Segmentation Fault & Potential Buffer Overflow Hunting](./linux/docs/privilege_escalation_via_segmentation_fault_and_buffer_overflow.md) (ES|QL)
+- [OSQuery SUID Hunting](./linux/docs/privilege_escalation_via_suid_binaries.md) (SQL)
+
+
 ## llm
 - [AWS Bedrock LLM Denial-of-Service or Resource Exhaustion](./llm/docs/aws_bedrock_dos_resource_exhaustion_detection.md) (ES|QL)
 - [AWS Bedrock LLM Latency Anomalies](./llm/docs/aws_bedrock_latency_anomalies_detection.md) (ES|QL)

diff --git a/...trol_via_network_connections_with_low_occurrence_frequency_for_unique_agents.md b/...trol_via_network_connections_with_low_occurrence_frequency_for_unique_agents.md
@@ -0,0 +1,78 @@
+# Network Connections with Low Occurrence Frequency for Unique Agent ID
+
+---
+
+## Metadata
+
+- **Author:** Elastic
+- **Description:** This hunt identifies network connections with low occurrence frequency for unique agent IDs on Linux systems. It monitors network connection attempts and acceptances, focusing on those initiated by processes that are rarely seen in the environment. By accounting for known low-frequency legitimate binaries (LoLBins) and suspicious directories, this hunt aims to detect unusual network activity that may indicate malicious behavior.
+
+- **UUID:** `ecd84bc7-32ae-474b-93a8-d1d9736c3464`
+- **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
+- **Language:** `[ES|QL]`
+
+## Query
+
+```sql
+from logs-endpoint.events.network-*
+| where @timestamp > now() - 7 day
+| where host.os.type == "linux" and event.type == "start" and event.action in ("connection_attempted", "connection_accepted") and destination.ip IS NOT null and not CIDR_MATCH(destination.ip, "10.0.0.0/8", "127.0.0.0/8", "169.254.0.0/16", "172.16.0.0/12", "192.0.0.0/24", "192.0.0.0/29", "192.0.0.8/32", "192.0.0.9/32", "192.0.0.10/32", "192.0.0.170/32", "192.0.0.171/32", "192.0.2.0/24", "192.31.196.0/24", "192.52.193.0/24", "192.168.0.0/16", "192.88.99.0/24", "224.0.0.0/4", "100.64.0.0/10", "192.175.48.0/24","198.18.0.0/15", "198.51.100.0/24", "203.0.113.0/24", "224.0.0.0/4", "240.0.0.0/4", "::1","FE80::/10", "FF00::/8")
+| stats cc = count(), agent_count = count_distinct(agent.id) by process.executable
+| where agent_count == 1 and cc > 0 and cc <= 3
+| limit 100
+| sort cc asc
+```
+
+```sql
+from logs-endpoint.events.network-*
+| where @timestamp > now() - 7 day
+| where host.os.type == "linux" and event.type == "start" and event.action in ("connection_attempted", "connection_accepted") and (
+    process.name in ("bash", "dash", "sh", "tcsh", "csh", "zsh", "ksh", "fish", "socat", "java", "awk", "gawk", "mawk", "nawk", "openssl", "nc", "ncat", "netcat", "nc.openbsd", "telnet") or
+    process.name like "python*" or
+    process.name like "perl*" or
+    process.name like "ruby*" or
+    process.name like "lua*" or
+    process.name like "php*"
+) and
+destination.ip IS NOT null and not CIDR_MATCH(destination.ip, "10.0.0.0/8", "127.0.0.0/8", "169.254.0.0/16", "172.16.0.0/12", "192.0.0.0/24", "192.0.0.0/29", "192.0.0.8/32", "192.0.0.9/32", "192.0.0.10/32", "192.0.0.170/32", "192.0.0.171/32", "192.0.2.0/24", "192.31.196.0/24", "192.52.193.0/24", "192.168.0.0/16", "192.88.99.0/24", "224.0.0.0/4", "100.64.0.0/10", "192.175.48.0/24","198.18.0.0/15", "198.51.100.0/24", "203.0.113.0/24", "224.0.0.0/4", "240.0.0.0/4", "::1","FE80::/10", "FF00::/8")
+| stats cc = count(), agent_count = count_distinct(agent.id) by process.name
+| where agent_count <= 3 and cc > 0 and cc <= 5
+| limit 100
+| sort cc asc
+```
+
+```sql
+from logs-endpoint.events.network-*
+| where @timestamp > now() - 30 day
+| where host.os.type == "linux" and event.type == "start" and event.action in ("connection_attempted", "connection_accepted") and (
+    process.executable like "./*" or
+    process.executable like "/dev/shm/*" or
+    process.executable like "/var/www/*" or
+    process.executable like "/boot/*" or
+    process.executable like "/srv/*" or
+    process.executable rlike "/tmp/[^/]+" or
+    process.executable rlike "/var/tmp/[^/]+" or
+    process.executable rlike "/run/[^/]+" or
+    process.executable rlike "/var/run/[^/]+"
+) and
+destination.ip IS NOT null and not CIDR_MATCH(destination.ip, "127.0.0.0/8", "169.254.0.0/16", "224.0.0.0/4", "::1")
+| stats cc = count(), agent_count = count_distinct(agent.id) by process.executable
+| where agent_count <= 3 and cc > 0 and cc <= 5
+| limit 100
+| sort cc asc
+```
+
+## Notes
+
+- Monitors for network connections initiated by processes that have low occurrence frequency, focusing on unique agent IDs.
+- Excludes common internal IP ranges to minimize false positives.
+- Accounts for known low-frequency legitimate binaries (LoLBins) to reduce noise.
+- Identifies suspicious directories where processes are executed from, which can indicate malicious activity.
+## MITRE ATT&CK Techniques
+
+- [T1071.001](https://attack.mitre.org/techniques/T1071/001)
+- [T1071.004](https://attack.mitre.org/techniques/T1071/004)
+
+## License
+
+- `Elastic License v2`
diff --git a/...ux/docs/command_and_control_via_unusual_file_downloads_from_source_addresses.md b/...ux/docs/command_and_control_via_unusual_file_downloads_from_source_addresses.md
@@ -0,0 +1,39 @@
+# Unusual File Downloads from Source Addresses
+
+---
+
+## Metadata
+
+- **Author:** Elastic
+- **Description:** This hunt identifies unusual file download activities on Linux systems. It detects instances where commonly used download utilities such as curl and wget are executed with command lines that contain IP addresses, which can indicate potentially suspicious file downloads.
+
+- **UUID:** `0d061fad-cf35-43a6-b9b7-986c348bf182`
+- **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
+- **Language:** `[ES|QL]`
+
+## Query
+
+```sql
+from logs-endpoint.events.process-*
+| where @timestamp > now() - 7 day
+| where host.os.type == "linux" and event.type == "start" and process.name in ("curl", "wget") and process.command_line rlike """.*[0-9]{1,3}(\.[0-9]{1,3}){3}.*"""
+| stats cc = count(), host_count = count_distinct(host.id) by process.command_line, process.executable
+| where cc <= 10 and host_count <= 5
+| sort cc asc
+| limit 100
+```
+
+## Notes
+
+- Detects instances where download utilities like curl and wget are used with IP addresses in their command lines.
+- Monitors for potentially suspicious file downloads, which are often seen in malicious activities.
+- Uses process command line counting in conjunction with host counting to minimize false positives caused by legitimate downloads.
+- The process command line count threshold is set to <= 10, and the host count threshold is set to <= 5 to balance detection and noise.
+## MITRE ATT&CK Techniques
+
+- [T1071.001](https://attack.mitre.org/techniques/T1071/001)
+- [T1071.004](https://attack.mitre.org/techniques/T1071/004)
+
+## License
+
+- `Elastic License v2`
diff --git a/hunting/linux/docs/defense_evasion_via_capitalized_process_execution.md b/hunting/linux/docs/defense_evasion_via_capitalized_process_execution.md
@@ -0,0 +1,41 @@
+# Defense Evasion via Capitalized Process Execution
+
+---
+
+## Metadata
+
+- **Author:** Elastic
+- **Description:** This hunt identifies potential defense evasion techniques via capitalized process execution on Linux systems. It detects processes that have two or more consecutive capital letters within their names, which can indicate an attempt to evade detection. Such naming conventions are often used in malicious payloads to blend in with legitimate processes.
+
+- **UUID:** `9d485892-1ca2-464b-9e4e-6b21ab379b9a`
+- **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
+- **Language:** `[ES|QL]`
+
+## Query
+
+```sql
+from logs-endpoint.events.process-*
+| where @timestamp > now() - 10 day
+| where host.os.type == "linux" and event.type == "start" and event.action == "exec" and (
+  (process.name rlike """[A-Z]{2,}[a-z]{1,}[0-9]{0,}""") or
+  (process.name rlike """[A-Z]{1,}[0-9]{0,}""")
+)
+| stats cc = count(), host_count = count_distinct(host.name) by process.name
+// Alter this threshold to make sense for your environment
+| where cc <= 3 and host_count <= 3
+| limit 100
+```
+
+## Notes
+
+- Detects processes that have two or more consecutive capital letters within their names, with optional digits.
+- This technique is often used in malicious payloads, such as Metasploit payloads, to evade detection.
+- Included a process count of <= 3 and a host count of <= 3 to eliminate common processes across different hosts.
+## MITRE ATT&CK Techniques
+
+- [T1036.004](https://attack.mitre.org/techniques/T1036/004)
+- [T1070](https://attack.mitre.org/techniques/T1070)
+
+## License
+
+- `Elastic License v2`
diff --git a/hunting/linux/docs/defense_evasion_via_hidden_process_execution.md b/hunting/linux/docs/defense_evasion_via_hidden_process_execution.md
@@ -0,0 +1,39 @@
+# Hidden Process Execution
+
+---
+
+## Metadata
+
+- **Author:** Elastic
+- **Description:** This hunt identifies hidden process executions on Linux systems. It detects processes executed from hidden files, which are often used by malicious actors to conceal their activities. By focusing on hidden files rather than directories, this hunt aims to catch stealthy processes while minimizing noise.
+
+- **UUID:** `00461198-9a2d-4823-b4cc-f3d1b5c17935`
+- **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
+- **Language:** `[ES|QL]`
+
+## Query
+
+```sql
+from logs-endpoint.events.process-*
+| where @timestamp > now() - 30 day
+| where host.os.type == "linux" and event.type == "start" and event.action == "exec" and 
+  process.executable rlike "/[^/]+/\\.[^/]+"
+| stats cc = count(), host_count = count_distinct(host.name) by process.executable, process.parent.executable, user.id
+// Alter this threshold to make sense for your environment
+| where cc <= 3 and host_count <= 3
+| sort cc asc
+| limit 100
+```
+
+## Notes
+
+- Included only hidden files, excluding hidden directories, as hidden directories are common in Unix.
+- Included a process or parent process count of <= 3, and a host count of <= 3 to eliminate common processes across different hosts.
+## MITRE ATT&CK Techniques
+
+- [T1036.004](https://attack.mitre.org/techniques/T1036/004)
+- [T1059](https://attack.mitre.org/techniques/T1059)
+
+## License
+
+- `Elastic License v2`