darkerth / auto_scrapy
Car-ads crawler
Installs: 18
Dependents: 0
Suggesters: 0
Security: 0
Stars: 0
Watchers: 0
Forks: 0
Language:Python
Type:project
This package is not auto-updated.
Last update: 2025-06-30 20:09:12 UTC
README
Crawl todays ads from various Lithuanian car-ads websites based on some parameters.
Requirements
- Python 2.7+
How to run
- Navigate to the spider directory:
cd auto/spiders
- Execute the spider with
python auto_spider.py {filename} {manufacturer} {model} {year_from} {year_to} {price_from} {price_to}
Example: python auto_spider.py response audi a4 2000 2006 0 3000
- this will crawl Audi A4 cars by 2000-2006 years and price 0-3000. Crawled result is saved on {filename}.json
HTTPS support:
In case you need to crawl websites with SSL certificate
sudo apt-get install libssl-dev
pip install pyopenssl --upgrade
To-Do:
- Proxy support
- In-depth ad crawling
- Random delays