[Bug] Normalize Hunting Index Link Generation (elastic#3872)

* normalizing hunting link generation

* replacing header

* adjusting quotes in f-strings

* added source file to metadata

* removed os dependency

* address bug in source file links

* reverting TOML loading

* change all List type hinting to list

* change all List type hinting to list

* fixed accented characters in queries

* reverted accent character removal; moved macos query and MD to macos folder

hunting/generate_markdown.py

@@ -35,9 +35,9 @@ class Hunt:
     language: list[str]
     license: str
     query: list[str]
-    notes: Optional[List[str]] = field(default_factory=list)
-    mitre: Optional[List[str]] = field(default_factory=list)
-    references: Optional[List[str]] = field(default_factory=list)
+    notes: Optional[list[str]] = field(default_factory=list)
+    mitre: Optional[list[str]] = field(default_factory=list)
+    references: Optional[list[str]] = field(default_factory=list)
 
 
 def load_toml(contents: str) -> Hunt:
@@ -57,7 +57,8 @@ def load_all_toml(base_path: Path) -> List[tuple[Hunt, Path]]:
 
 def validate_link(link: str):
     """Validate and return the link."""
-    response = urllib3.request('get', link)
+    http = urllib3.PoolManager()
+    response = http.request('GET', link)
     if response.status != 200:
         raise ValueError(f"Invalid link: {link}")
 
@@ -84,23 +85,23 @@ def convert_toml_to_markdown(hunt_config: Hunt, file_path: Path) -> str:
     markdown += f"- **Author:** {hunt_config.author}\n"
     markdown += f"- **Description:** {hunt_config.description}\n"
     markdown += f"- **UUID:** `{hunt_config.uuid}`\n"
-    markdown += f"- **Integration:** {", ".join(generate_integration_links(hunt_config.integration))}\n"
-    markdown += f"- **Language:** `{hunt_config.language}`\n\n".replace("'", "").replace('"', "")
-    markdown += "## Query\n\n"
+    markdown += f"- **Integration:** {', '.join(generate_integration_links(hunt_config.integration))}\n"
+    markdown += f"- **Language:** `{hunt_config.language}`\n".replace("'", "").replace('"', "")
+    markdown += f"- **Source File:** [{hunt_config.name}]({(Path('../queries') / file_path.name).as_posix()})\n"
+    markdown += "\n## Query\n\n"
     for query in hunt_config.query:
         markdown += f"```sql\n{query}```\n\n"
 
     if hunt_config.notes:
         markdown += "## Notes\n\n" + "\n".join(f"- {note}" for note in hunt_config.notes)
     if hunt_config.mitre:
-        markdown += "\n## MITRE ATT&CK Techniques\n\n" + "\n".join(
+        markdown += "\n\n## MITRE ATT&CK Techniques\n\n" + "\n".join(
             f"- [{tech}]({ATLAS_URL if tech.startswith('AML') else ATTACK_URL}"
             f"{tech.replace('.', '/') if tech.startswith('T') else tech})"
             for tech in hunt_config.mitre
         )
     if hunt_config.references:
-        markdown += "\n## References\n\n" + "\n".join(f"- {ref}" for ref in hunt_config.references)
-        markdown += f"\n- [{hunt_config.name}]({Path('../queries') / file_path.name})"
+        markdown += "\n\n## References\n\n" + "\n".join(f"- {ref}" for ref in hunt_config.references)
 
     markdown += f"\n\n## License\n\n- `{hunt_config.license}`\n"
     return markdown
@@ -126,8 +127,8 @@ def process_toml_files(base_path: Path) -> None:
     for folder, files in sorted(directories.items()):
         index_content += f"\n\n## {folder}\n"
         for file_path, rule_name, language in sorted(files):
-            index_path = "./" + str(file_path)
-            index_content += f"- [{rule_name}]({index_path}) ({", ".join(language)})\n"
+            index_path = f"./{file_path.as_posix()}"
+            index_content += f"- [{rule_name}]({index_path}) ({', '.join(language)})\n"
 
     # Write the index file at the base directory level
     index_path = base_path / "index.md"

hunting/index.md

@@ -43,6 +43,7 @@ Here are the queries currently available:
 
 
 ## macos
+- [Low Occurrence of Suspicious Launch Agent or Launch Daemon](./macos/docs/persistence_via_suspicious_launch_agent_or_launch_daemon_with_low_occurrence.md) (ES|QL)
 - [Suspicious Network Connections by Unsigned Mach-O](./macos/docs/suspicious_network_connections_by_unsigned_macho.md) (ES|QL)
 
 
@@ -70,7 +71,6 @@ Here are the queries currently available:
 - [PE File Transfer via SMB_Admin Shares by Agent or User](./windows/docs/pe_file_transfer_via_smb_admin_shares_by_agent.md) (ES|QL)
 - [Persistence via Run Key with Low Occurrence Frequency](./windows/docs/persistence_via_run_key_with_low_occurrence_frequency.md) (ES|QL)
 - [Persistence via Startup with Low Occurrence Frequency by Unique Host](./windows/docs/persistence_via_startup_with_low_occurrence_frequency.md) (ES|QL)
-- [Low Occurrence of Suspicious Launch Agent or Launch Daemon](./windows/docs/persistence_via_suspicious_launch_agent_or_launch_daemon_with_low_occurrence.md) (ES|QL)
 - [Egress Network Connections with Total Bytes Greater than Threshold](./windows/docs/potential_exfiltration_by_process_total_egress_bytes.md) (ES|QL)
 - [Rundll32 Execution Aggregated by Command Line](./windows/docs/rundll32_execution_aggregated_by_cmdline.md) (ES|QL)
 - [Scheduled tasks Creation by Action via Registry](./windows/docs/scheduled_task_creation_by_action_via_registry.md) (ES|QL)

hunting/linux/docs/command_and_control_via_network_connections_with_low_occurrence_frequency_for_unique_agents.md

@@ -10,6 +10,7 @@
 - **UUID:** `ecd84bc7-32ae-474b-93a8-d1d9736c3464`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Network Connections with Low Occurrence Frequency for Unique Agent ID](../queries/command_and_control_via_network_connections_with_low_occurrence_frequency_for_unique_agents.toml)
 
 ## Query
 
@@ -68,6 +69,7 @@ destination.ip IS NOT null and not CIDR_MATCH(destination.ip, "127.0.0.0/8", "16
 - Excludes common internal IP ranges to minimize false positives.
 - Accounts for known low-frequency legitimate binaries (LoLBins) to reduce noise.
 - Identifies suspicious directories where processes are executed from, which can indicate malicious activity.
+
 ## MITRE ATT&CK Techniques
 
 - [T1071.001](https://attack.mitre.org/techniques/T1071/001)

hunting/linux/docs/command_and_control_via_unusual_file_downloads_from_source_addresses.md

@@ -10,6 +10,7 @@
 - **UUID:** `0d061fad-cf35-43a6-b9b7-986c348bf182`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Unusual File Downloads from Source Addresses](../queries/command_and_control_via_unusual_file_downloads_from_source_addresses.toml)
 
 ## Query
 
@@ -29,6 +30,7 @@ from logs-endpoint.events.process-*
 - Monitors for potentially suspicious file downloads, which are often seen in malicious activities.
 - Uses process command line counting in conjunction with host counting to minimize false positives caused by legitimate downloads.
 - The process command line count threshold is set to <= 10, and the host count threshold is set to <= 5 to balance detection and noise.
+
 ## MITRE ATT&CK Techniques
 
 - [T1071.001](https://attack.mitre.org/techniques/T1071/001)

hunting/linux/docs/defense_evasion_via_capitalized_process_execution.md

@@ -10,6 +10,7 @@
 - **UUID:** `9d485892-1ca2-464b-9e4e-6b21ab379b9a`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Defense Evasion via Capitalized Process Execution](../queries/defense_evasion_via_capitalized_process_execution.toml)
 
 ## Query
 
@@ -31,6 +32,7 @@ from logs-endpoint.events.process-*
 - Detects processes that have two or more consecutive capital letters within their names, with optional digits.
 - This technique is often used in malicious payloads, such as Metasploit payloads, to evade detection.
 - Included a process count of <= 3 and a host count of <= 3 to eliminate common processes across different hosts.
+
 ## MITRE ATT&CK Techniques
 
 - [T1036.004](https://attack.mitre.org/techniques/T1036/004)

hunting/linux/docs/defense_evasion_via_hidden_process_execution.md

@@ -10,6 +10,7 @@
 - **UUID:** `00461198-9a2d-4823-b4cc-f3d1b5c17935`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Hidden Process Execution](../queries/defense_evasion_via_hidden_process_execution.toml)
 
 ## Query
 
@@ -29,6 +30,7 @@ from logs-endpoint.events.process-*
 
 - Included only hidden files, excluding hidden directories, as hidden directories are common in Unix.
 - Included a process or parent process count of <= 3, and a host count of <= 3 to eliminate common processes across different hosts.
+
 ## MITRE ATT&CK Techniques
 
 - [T1036.004](https://attack.mitre.org/techniques/T1036/004)

hunting/linux/docs/defense_evasion_via_multi_dot_process_execution.md

@@ -10,6 +10,7 @@
 - **UUID:** `11810497-8ce3-4960-9777-9d0e97052682`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Potential Defense Evasion via Multi-Dot Process Execution](../queries/defense_evasion_via_multi_dot_process_execution.toml)
 
 ## Query
 
@@ -28,6 +29,7 @@ from logs-endpoint.events.process-*
 
 - This query identifies processes with executables containing three or more consecutive dots in their names.
 - The process count threshold of <= 10 can be adjusted based on the environment's baseline activity.
+
 ## MITRE ATT&CK Techniques
 
 - [T1036.004](https://attack.mitre.org/techniques/T1036/004)

hunting/linux/docs/excessive_ssh_network_activity_unique_destinations.md

@@ -10,6 +10,7 @@
 - **UUID:** `223f812c-a962-4d58-961d-134d8f8b15da`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Excessive SSH Network Activity to Unique Destinations](../queries/excessive_ssh_network_activity_unique_destinations.toml)
 
 ## Query
 
@@ -30,6 +31,7 @@ from logs-endpoint.events.network-*
 - Monitors network connections to port 22 (SSH) and counts the number of unique destination IP addresses per host and user.
 - A high number of unique destinations can indicate suspicious activity such as discovery or lateral movement.
 - The threshold of 10 unique destinations can be adjusted to suit the environment's baseline activity.
+
 ## MITRE ATT&CK Techniques
 
 - [T1021.004](https://attack.mitre.org/techniques/T1021/004)

hunting/linux/docs/execution_uncommon_process_execution_from_suspicious_directory.md

@@ -10,6 +10,7 @@
 - **UUID:** `0ea47044-b161-4785-ba99-e11f46d6ac51`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Uncommon Process Execution from Suspicious Directory](../queries/execution_uncommon_process_execution_from_suspicious_directory.toml)
 
 ## Query
 
@@ -47,6 +48,7 @@ from logs-endpoint.events.process-*
 - Excluded /tmp, /var/tmp, /run, /var/run subdirectories to reduce noise.
 - Excluded /tmp, /var/tmp files starting or ending with digits to exclude real temporary files.
 - Included a process or parent process count of <= 3, and a host count of <= 3 to eliminate common processes across different hosts.
+
 ## MITRE ATT&CK Techniques
 
 - [T1036.004](https://attack.mitre.org/techniques/T1036/004)

hunting/linux/docs/login_activity_by_source_address.md

@@ -10,6 +10,7 @@
 - **UUID:** `95c1467d-d566-4645-b5f1-37a4b0093bb6`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Logon Activity by Source IP](../queries/login_activity_by_source_address.toml)
 
 ## Query
 
@@ -29,6 +30,7 @@ from logs-system.auth-*
 - Monitors authentication events and counts failed and successful logon attempts by source IP address.
 - A high number of failed logon attempts combined with a low number of successful logons and multiple distinct usernames can indicate a potential brute force or credential stuffing attack.
 - The thresholds for failed attempts, successful logons, and distinct usernames should be adjusted based on the environment's normal logon patterns.
+
 ## MITRE ATT&CK Techniques
 
 - [T1110](https://attack.mitre.org/techniques/T1110)

hunting/linux/docs/low_volume_external_network_connections_from_process.md

@@ -10,6 +10,7 @@
 - **UUID:** `12526f14-5e35-4f5f-884c-96c6a353a544`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Low Volume External Network Connections from Process by Unique Agent](../queries/low_volume_external_network_connections_from_process.toml)
 
 ## Query
 
@@ -40,6 +41,7 @@ from logs-endpoint.events.network-*
 - Monitors for network connections attempted by processes that have a low occurrence frequency (five or fewer connections) and are seen by a unique agent.
 - Excludes common internal IP ranges to minimize false positives.
 - A separate query is included to specifically monitor low volume network connections initiated by the root user, as these can be particularly indicative of malicious activity.
+
 ## MITRE ATT&CK Techniques
 
 - [T1071.001](https://attack.mitre.org/techniques/T1071/001)

hunting/linux/docs/low_volume_gtfobins_external_network_connections.md

@@ -10,6 +10,7 @@
 - **UUID:** `783d6091-b98d-45a8-a880-a07f112a8aa2`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Low Volume GTFOBins External Network Connections](../queries/low_volume_gtfobins_external_network_connections.toml)
 
 ## Query
 
@@ -36,6 +37,7 @@ destination.ip IS NOT null and not CIDR_MATCH(destination.ip, "10.0.0.0/8", "127
 - Excludes common internal IP ranges to minimize false positives.
 - Counts the occurrences of these connections by process name and limits to those seen infrequently (five or fewer connections).
 - This can help identify potentially malicious activity involving the exploitation of GTFOBins.
+
 ## MITRE ATT&CK Techniques
 
 - [T1219](https://attack.mitre.org/techniques/T1219)

hunting/linux/docs/low_volume_modifications_to_critical_system_binaries.md

@@ -10,6 +10,7 @@
 - **UUID:** `c7044817-d9a5-4755-abab-9059e50dab24`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Low Volume Modifications to Critical System Binaries by Unique Host](../queries/low_volume_modifications_to_critical_system_binaries.toml)
 
 ## Query
 
@@ -39,6 +40,7 @@ from logs-endpoint.events.file-*
 - Excludes modifications made by expected update processes such as package managers to reduce false positives.
 - Counts the number of unique files modified by each host and the number of modifications made.
 - Focuses on modifications made by unique hosts to detect unauthorized or suspicious changes.
+
 ## MITRE ATT&CK Techniques
 
 - [T1070.004](https://attack.mitre.org/techniques/T1070/004)

hunting/linux/docs/low_volume_process_injection_syscalls_by_executable.md

@@ -10,6 +10,7 @@
 - **UUID:** `c9931736-d5ec-4c89-b4d2-d71dcf5ca12a`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Low Volume Process Injection-Related Syscalls by Process Executable](../queries/low_volume_process_injection_syscalls_by_executable.toml)
 
 ## Query
 
@@ -28,6 +29,7 @@ from logs-auditd_manager.auditd-*, logs-auditd.log-*, auditbeat-*
 - Monitors for process injection-related syscalls such as ptrace and memfd_create.
 - Counts the occurrences of these syscalls by process executable to identify processes that make these syscalls infrequently.
 - Focuses on low volume occurrences to detect potential malicious activity related to process injection.
+
 ## MITRE ATT&CK Techniques
 
 - [T1055.001](https://attack.mitre.org/techniques/T1055/001)

hunting/linux/docs/persistence_reverse_bind_shells.md

@@ -10,6 +10,7 @@
 - **UUID:** `7422faf1-ba51-49c3-b8ba-13759e6bcec4`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[SQL]`
+- **Source File:** [Persistence Through Reverse/Bind Shells](../queries/persistence_reverse_bind_shells.toml)
 
 ## Query
 
@@ -49,6 +50,7 @@ SELECT pid, address, port, socket, protocol, path FROM listening_ports
 - The third hunt query retrieves information about listening ports on the system.
 - Investigate strange or unexpected open sockets, processes, or listening ports on the system.
 - Use the information from each hunt to pivot and investigate further for potential reverse/bind shells.
+
 ## MITRE ATT&CK Techniques
 
 - [T1059.004](https://attack.mitre.org/techniques/T1059/004)

hunting/linux/docs/persistence_via_cron.md

@@ -10,6 +10,7 @@
 - **UUID:** `e1cffb7c-4acf-4e7a-8d72-b8b7657cf7b8`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL, SQL]`
+- **Source File:** [Persistence via Cron](../queries/persistence_via_cron.toml)
 
 ## Query
 
@@ -101,6 +102,7 @@ SELECT * FROM crontab
 - Uses EVAL to tag potential persistence events and counts occurrences to identify unusual activity.
 - Monitors processes started by cron, fcron, or atd to detect potential persistence mechanisms.
 - OSQuery queries are provided to complement the detection by retrieving detailed file information and crontab entries.
+
 ## MITRE ATT&CK Techniques
 
 - [T1053.003](https://attack.mitre.org/techniques/T1053/003)

hunting/linux/docs/persistence_via_driver_load_with_low_occurrence_frequency.md

@@ -10,6 +10,7 @@
 - **UUID:** `e1f59c9a-7a2a-4eb8-a524-97b16a041a4a`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL]`
+- **Source File:** [Drivers Load with Low Occurrence Frequency](../queries/persistence_via_driver_load_with_low_occurrence_frequency.toml)
 
 ## Query
 
@@ -29,6 +30,7 @@ from logs-auditd_manager.auditd-*, logs-auditd.log-*, auditbeat-*
 - Monitors for kernel modules loaded with syscall 'init_module' or 'finit_module', indicating driver load events.
 - Counts the occurrence of each driver across all hosts and identifies those seen only once on a single host within the past year.
 - Such rare driver loads can indicate potentially malicious activity or the presence of uncommon drivers.
+
 ## MITRE ATT&CK Techniques
 
 - [T1547.006](https://attack.mitre.org/techniques/T1547/006)

hunting/linux/docs/persistence_via_git_hook_pager.md

@@ -10,6 +10,7 @@
 - **UUID:** `2d7bb29d-d53f-47ab-a0b4-1818adb91423`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL, SQL]`
+- **Source File:** [Git Hook/Pager Persistence](../queries/persistence_via_git_hook_pager.toml)
 
 ## Query
 
@@ -80,6 +81,7 @@ WHERE
 - Monitors processes started by Git hooks to detect suspicious activity related to Git configurations.
 - OSQuery query is provided to retrieve detailed file information related to Git configurations and hooks.
 - OSQuery has limited support for wildcard queries, therefore the query includes multiple LIKE conditions for directories. These can be increased and decreased based on the environment's Git configuration paths.
+
 ## MITRE ATT&CK Techniques
 
 - [T1546.004](https://attack.mitre.org/techniques/T1546/004)

hunting/linux/docs/persistence_via_message_of_the_day.md

@@ -10,6 +10,7 @@
 - **UUID:** `5984a354-d76c-43e6-bdd9-228456f1b371`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL, SQL]`
+- **Source File:** [Persistence via Message-of-the-Day](../queries/persistence_via_message_of_the_day.toml)
 
 ## Query
 
@@ -70,6 +71,7 @@ ORDER BY
 - Uses EVAL to tag potential persistence events and counts occurrences to identify unusual activity.
 - Monitors processes started by motd scripts to detect potential persistence mechanisms.
 - OSQuery query is provided to complement the detection by retrieving detailed file information related to motd scripts.
+
 ## MITRE ATT&CK Techniques
 
 - [T1036.005](https://attack.mitre.org/techniques/T1036/005)

hunting/linux/docs/persistence_via_package_manager.md

@@ -10,6 +10,7 @@
 - **UUID:** `2d01a413-8d97-407a-8698-02dfc7119c97`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint)
 - **Language:** `[ES|QL, SQL]`
+- **Source File:** [Persistence via Packager Manager](../queries/persistence_via_package_manager.toml)
 
 ## Query
 
@@ -90,6 +91,7 @@ SELECT * FROM yum_sources
 - Uses EVAL to tag potential persistence events and counts occurrences to identify unusual activity.
 - Monitors processes started by package managers to detect suspicious activity related to package manager configurations.
 - OSQuery queries are provided to retrieve detailed file information related to package manager configurations, as well as sources for APT and YUM.
+
 ## MITRE ATT&CK Techniques
 
 - [T1546.004](https://attack.mitre.org/techniques/T1546/004)

hunting/linux/docs/persistence_via_rc_local.md

@@ -10,6 +10,7 @@
 - **UUID:** `a95f778f-2193-4a3d-bbbe-7b02d5740638`
 - **Integration:** [endpoint](https://docs.elastic.co/integrations/endpoint), [system](https://docs.elastic.co/integrations/system)
 - **Language:** `[ES|QL, SQL]`
+- **Source File:** [Persistence via rc.local/rc.common](../queries/persistence_via_rc_local.toml)
 
 ## Query
 
@@ -71,6 +72,7 @@ WHERE
 - Monitors processes started by rc.local and rc.common scripts to detect potential persistence mechanisms.
 - Syslog hunting query is provided to complement the detection by analyzing syslog entries related to rc.local and rc.common processes.
 - OSQuery queries are provided to retrieve systemd unit states, startup items, and detailed file information related to rc.local and rc.common.
+
 ## MITRE ATT&CK Techniques
 
 - [T1037.004](https://attack.mitre.org/techniques/T1037/004)