macklus/yii2-crawler

Advanced web crawler

dev-master 2016-04-01 06:42 UTC

Requires

• fabpot/goutte: ^3.1
• yiisoft/yii2: *

GPL-3.0+

• Macklus

• extension
• yii2
• crawler
• goutte
• proxy

README

Advanced web crawler module

Features

• Multiple connections on single object
• Define and use multiple proxys
• Define and use multiple identities (usarname/password)

Install

Install using composer:

The preferred way to install this extension is through composer.

Either run

php composer.phar require --prefer-dist macklus/yii2-geoselect "*"

or add

"macklus/yii2-geoselect": "*"

to the require section of your composer.json file.

Usage

Create new object:

<?php

namespace app\controllers;

use macklus\Crawler\Crawler;

class TestController extends Controller
{
    public function actionCraw()
    {
        $crawler = new Crawler();
    }
}

UserAgent

$crawler->setUA($browser)
/*
 * Browser could be chrome, firefox or explorer
 */

Identities

$crawler->setUser($name, $user, $password)
$crawler->setUsers(['user1' => ['username' => 'jhon', 'password' => 'doe']])

Proxys

$crawler->setProxy($name, $string)
$crawler->setProxys([['proxy1' => 'proxystring', 'proxy2' => 'proxy2string]])
/*
 * string should contain all proxy info, like http://username:[e-mail]:8080/
 */