Merge pull request #51 from NeuromatchAcademy/robots.txt-update

Update robots.txt
NeuromatchAcademy · Jun 24, 2024 · 025c09f · 025c09f
2 parents f27a8d9 + fb2be09
commit 025c09f
Showing 1 changed file with 6 additions and 0 deletions.
diff --git a/public/robots.txt b/public/robots.txt
@@ -38,6 +38,12 @@ Disallow: /
 User-agent: Applebot
 Disallow: /
 
+# AI Data Scraper
+# https://darkvisitors.com/agents/applebot-extended
+
+User-agent: Applebot-Extended
+Disallow: /
+
 # AI Data Scraper
 # https://darkvisitors.com/agents/bytespider