Create robots.txt

SEO bot crawlers needlessly consuming a lot of resources
mormegil-cz · Jan 23, 2024 · c2c56fc · c2c56fc
1 parent 161a756
commit c2c56fc
Showing 1 changed file with 11 additions and 0 deletions.
diff --git a/KdyPojedeVlak.Web/wwwroot/robots.txt b/KdyPojedeVlak.Web/wwwroot/robots.txt
@@ -0,0 +1,11 @@
+User-agent: SemrushBot
+Disallow: /Train/
+Disallow: /Transits/
+
+User-agent: AhrefsBot
+Disallow: /Train/
+Disallow: /Transits/
+
+User-agent: barkrowler
+Disallow: /Train/
+Disallow: /Transits/