Crawler Robots Policy Checker (src/CrawlerRobots.php)
Evaluates whether a crawler is permitted to index or follow links on a webpage based on robots.txt headers and meta tags.
- Repository
- crawler
- Language
- php
- SPDX licence
- MIT
- Quality score (dated snapshot)
- 9.6/10
- Category
- scraping
Review source and licence terms before reuse. View on the service