Merge pull request #128 from mrrfv/blocklist-summer-cleaning

Blocklist summer cleaning

README.md

@@ -1,6 +1,6 @@
 # Cloudflare Gateway Pi-hole Scripts (CGPS)
 
-![Cloudflare Gateway Analytics screenshot](.github/images/gateway_analytics.png)
+![Cloudflare Gateway Analytics screenshot showing a thousand blocked DNS requests](.github/images/gateway_analytics.png)
 
 Cloudflare Gateway allows you to create custom rules to filter HTTP, DNS, and network traffic based on your firewall policies. This is a collection of scripts that can be used to get a similar experience as if you were using Pi-hole, but with Cloudflare Gateway - so no servers to maintain or need to buy a Raspberry Pi!
 
@@ -19,7 +19,7 @@ Cloudflare Gateway allows you to create custom rules to filter HTTP, DNS, and ne
 - Automatically cleans up filter lists: removes duplicates, invalid domains, comments and more
 - Works **fully unattended**
 - **Allowlist support**, allowing you to prevent false positives and breakage by forcing trusted domains to always be unblocked.
-- Experimental **SNI-based filtering** support that works independently of DNS settings, preventing unauthorized or malicious DNS changes from bypassing the filter.
+- Experimental **SNI-based filtering** that works independently of DNS settings, preventing unauthorized or malicious DNS changes from bypassing the filter.
 - Optional health check: Sends a ping request ensuring continuous monitoring and alerting for the workflow execution, or messages a Discord webhook with progress.
 
 ## Usage
@@ -39,7 +39,7 @@ Cloudflare Gateway allows you to create custom rules to filter HTTP, DNS, and ne
 2. Run `npm install` to install dependencies.
 3. Copy `.env.example` to `.env` and fill in the values.
 4. If this is a subsequent run, execute `node cf_gateway_rule_delete.js` and `node cf_list_delete.js` (in order) to delete old data.
-5. If you haven't downloaded any filters yourself, run the `node download_lists.js` command to download recommended filter lists (about 250 000 domains).
+5. If you haven't downloaded any filters yourself, run the `node download_lists.js` command to download recommended filter lists (about 50 000 domains).
 6. Run `node cf_list_create.js` to create the lists in Cloudflare Gateway. This will take a while.
 7. Run `node cf_gateway_rule_create.js` to create the firewall rule in Cloudflare Gateway.
 8. Profit!
@@ -73,6 +73,10 @@ Please note that the GitHub Action downloads the recommended blocklists and whit
 
 Alternatively, you can install the Cloudflare WARP client and log in to Zero Trust. This method proxies your traffic over Cloudflare servers, meaning it works similarly to a commercial VPN. You need to do this if you want to use the SNI-based filtering feature, as it requires Cloudflare to inspect your raw traffic (HTTPS remains encrypted if "TLS decryption" is disabled).
 
+### Malware blocking
+
+The default filter lists are only optimized for ad & tracker blocking because Cloudflare Zero Trust itself comes with much more advanced security features. It's recommended that you create your own Cloudflare Gateway firewall policies that leverage those features on top of CGPS.
+
 ### Dry runs
 
 To see if e.g. your filter lists are valid without actually changing anything in your Cloudflare account, you can set the `DRY_RUN` environment variable to 1, either in `.env` or the regular way. This will only print info such as the lists that would be created or the amount of duplicate domains to the console.

lib/constants.js

@@ -56,46 +56,42 @@ export const USER_DEFINED_BLOCKLIST_URLS = process.env.BLOCKLIST_URLS
 
 // These are the default blocklists and allowlists that are used by the script if the user doesn't provide any URLs by themselves.
 // The files are dynamically fetched from the internet, therefore it's important to choose only the most reliable sources.
-// Commented out lists are subject to removal. 
+// Commented out lists are subject to removal.
+
+// You can have an unlimited number of allowlists, unlike blocklists.
 export const RECOMMENDED_ALLOWLIST_URLS = [
+  // Torrent trackers
   "https://raw.githubusercontent.com/im-sm/Pi-hole-Torrent-Blocklist/main/all-torrent-trackres.txt",
-  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/banks.txt",
-  //"https://raw.githubusercontent.com/hagezi/dns-blocklists/main/whitelist.txt",
+  // Banks
+  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/banks.txt", 
+  // Official Discord domains
+  "https://raw.githubusercontent.com/Dogino/Discord-Phishing-URLs/main/official-domains.txt",
+  // macOS specific
+  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/mac.txt",
+  // Windows specific
+  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/windows.txt",
+  // URL shorteners
+  "https://raw.githubusercontent.com/boutetnico/url-shorteners/master/list.txt",
+  // Firefox sync, add-ons, etc.
+  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/firefox.txt",
+  // Android apps
+  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/android.txt",
+
+  // General allowlists
   "https://raw.githubusercontent.com/TogoFire-Home/AD-Settings/main/Filters/whitelist.txt",
-  // Older fork of an unmaintained project
-  //"https://raw.githubusercontent.com/freekers/whitelist/master/domains/whitelist.txt",
   "https://raw.githubusercontent.com/DandelionSprout/AdGuard-Home-Whitelist/master/whitelist.txt",
   "https://raw.githubusercontent.com/AdguardTeam/AdGuardSDNSFilter/master/Filters/exclusions.txt",
-  //"https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/optional-list.txt",
   "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/issues.txt",
-  // Broken link
-  //"https://raw.githubusercontent.com/hagezi/dns-blocklists/main/whitelist-referral.txt",
-  "https://raw.githubusercontent.com/mawenjian/china-cdn-domain-whitelist/master/china-cdn-domain-whitelist.txt",
-  // Whitelists doubleclick.net and googleadservices.com
-  //"https://raw.githubusercontent.com/notracking/hosts-blocklists-scripts/master/hostnames.whitelist.txt",
-  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/mac.txt",
-  "https://raw.githubusercontent.com/boutetnico/url-shorteners/master/list.txt",
-  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/windows.txt",
-  "https://raw.githubusercontent.com/Dogino/Discord-Phishing-URLs/main/official-domains.txt",
-  // This one seems okay, but may be too broad for some users. It whitelists small analytics services and domains associated with ad networks.
-  //"https://raw.githubusercontent.com/ookangzheng/blahdns/master/hosts/whitelist.txt",
-  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/android.txt",
-  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/sensitive.txt",
-  // Unmaintained
-  //"https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt",
-  "https://raw.githubusercontent.com/AdguardTeam/HttpsExclusions/master/exclusions/firefox.txt",
-  // Commented out because it whitelists sites including doubleclick.net and ad.atdmt.com
-  // https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/referral-sites.txt,
   // Uncomment the line below to use OISD's most commmonly whitelisted list
   // https://local.oisd.nl/extract/commonly_whitelisted.php,
 ];
 
+// The default blocklist settings are optimized for performance while still blocking a lot.
+// Adding too many blocklists may slow down DNS response times and thus your internet speed.
+// If you'd like to use something larger, consider something like hagezi's Multi LIGHT:
+// https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/domains/light.txt
 export const RECOMMENDED_BLOCKLIST_URLS = [
-  "https://raw.githubusercontent.com/mullvad/dns-blocklists/main/output/doh/doh_adblock.txt",
-  // Commented out as it seems to cause the blocklist to exceed 300,000 domains as of Nov 9, 2023
-  // "https://raw.githubusercontent.com/mullvad/dns-blocklists/main/output/doh/doh_gambling.txt",
-  "https://raw.githubusercontent.com/mullvad/dns-blocklists/main/output/doh/doh_privacy.txt",
-  "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts",
+  "https://small.oisd.nl/",
+  // Only blocks mobile ads and analytics. Very tiny; comment the rest out and only use this one for the absolute best performance.
   "https://adaway.org/hosts.txt",
-  "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts",
 ];