Merge pull request #57 from NeuromatchAcademy/robots.txt-update

Update robots.txt
NeuromatchAcademy · Aug 4, 2024 · af91184 · af91184
2 parents b4bdaac + 14a545d
commit af91184
Showing 1 changed file with 6 additions and 0 deletions.
diff --git a/public/robots.txt b/public/robots.txt
@@ -116,6 +116,12 @@ Disallow: /
 User-agent: Meta-ExternalFetcher
 Disallow: /
 
+# AI Search Crawler
+# https://darkvisitors.com/agents/oai-searchbot
+
+User-agent: OAI-SearchBot
+Disallow: /
+
 # AI Data Scraper
 # https://darkvisitors.com/agents/omgili