CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent and http_from header. Currently able to detect 1,000's of bots/spiders/crawlers.
composer require jaybizzle/crawler-detect
use Jaybizzle\CrawlerDetect\CrawlerDetect;
$CrawlerDetect = new CrawlerDetect;
// Check the user agent of the current 'visitor'
if($CrawlerDetect->isCrawler()) {
// true if crawler user agent detected
}
// Pass a user agent as a string
if($CrawlerDetect->isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')) {
// true if crawler user agent detected
}
// Output the name of the bot that matched (if any)
echo $CrawlerDetect->getMatches();If you find a bot/spider/crawler user agent that CrawlerDetect fails to detect, please submit a pull request with the regex pattern added to the $data array in Fixtures/Crawlers.php and add the failing user agent to tests/crawlers.txt.
Failing that, just create an issue with the user agent you have found, and we'll take it from there :)
If you would like to use this with Laravel, please see Laravel-Crawler-Detect
To use this library with Symfony 2/3/4, check out the CrawlerDetectBundle.
To use this library with the YII2 framework, check out yii2-crawler-detect.
To use this library with NodeJS or any ES6 application based, check out es6-crawler-detect.
To use this library in a Python project, check out crawlerdetect.
To use this library in a JVM project (including Java, Scala, Kotlin, etc.), check out CrawlerDetect.
To use this library in a .net standard (including .net core) based project, check out NetCrawlerDetect.
To use this library with Ruby on Rails or any Ruby-based application, check out crawler_detect gem.
To use this library with Go, check out the crawlerdetect module.
Parts of this class are based on the brilliant MobileDetect