maxlen/webcrawler

Search engine crawlers

dev-master 2017-03-29 13:46 UTC

Requires

• php: >=5.4.0
• electrolinux/phpquery: dev-master
• guzzlehttp/guzzle: ~6.0

BSD-4-Clause

• Maxim

• extension
• crawler
• search engines

README

Search engines crawlers

Google example:

  $proxy = []; //['host' => '*.*.*.*', 'port' => '', 'login' => '', 'password' => '']
  $params = ['query' => 'test search', 'page' => $page, 'proxy' => $proxy];
  $crawler = new WebCrawler(['strategy' => new GoogleSearch()]);
  print_r($crawler->crawl($params));

Site-parse example:

  $params = ['url' => 'http://your-site.com', 'proxy' => []];
  $crawler = new WebCrawler(['strategy' => new SiteSearch()]);
  print_r($crawler->crawl($params));