darkerth/auto_scrapy

Car-ads crawler

Installs: 18

Dependents: 0

Suggesters: 0

Security: 0

Stars: 0

Watchers: 0

Forks: 0

Language:Python

Type:project

dev-master 2017-05-20 12:40 UTC

This package is not auto-updated.

Last update: 2025-06-30 20:09:12 UTC


README

Crawl todays ads from various Lithuanian car-ads websites based on some parameters.

Requirements

  • Python 2.7+

How to run

  • Navigate to the spider directory: cd auto/spiders
  • Execute the spider with python auto_spider.py {filename} {manufacturer} {model} {year_from} {year_to} {price_from} {price_to}

Example: python auto_spider.py response audi a4 2000 2006 0 3000 - this will crawl Audi A4 cars by 2000-2006 years and price 0-3000. Crawled result is saved on {filename}.json

HTTPS support:

In case you need to crawl websites with SSL certificate

  • sudo apt-get install libssl-dev
  • pip install pyopenssl --upgrade

To-Do:

  • Proxy support
  • In-depth ad crawling
  • Random delays