Scrapy AutoThrottle Extension (extensions/throttle.py)
Automatically adjusts Scrapy crawler download speeds based on server load and latency. Reuse to prevent target site overload and respect crawl limits.
- Repository
- scrapy
- Language
- python
- SPDX licence
- BSD-3-Clause
- Quality score (dated snapshot)
- 9.2/10
- Category
- scraping
Review source and licence terms before reuse. View on the service