What is scrapy_proxies?

Scrapy_proxies is a python module (preinstalled in our images), that handles the proxy management in our scrapers made with Scrapy

Our View on scrapy_proxies

Usage Rating of scrapy_proxies

1. BEST CHOICE: This is among the preferred tools we use

It's the easiest way to use an external list of proxies and their management (removing non funcioning proxies, retries and so on).

Settings

You can find the settings needed to make the module work in the github link of the project. We use a slightly modified version where there's another option for PROXY_MODE = -1, meaning don't use proxies.

When to use scrapy_proxies

Whenever and IP gets blocked after several requests.

Reference and documentation

https://github.com/aivarsk/scrapy-proxies

Did you find it helpful? Yes No

Scrapy_proxies (Scrapy module) Print