reliqarts / laravel-scavenger
The most integrated web scraper package for Laravel.
Fund package maintenance!
reliq
Installs: 1 474
Dependents: 0
Suggesters: 0
Security: 0
Stars: 153
Watchers: 4
Forks: 9
Open Issues: 0
Requires
- php: ^7.4 || ^8.0
- ext-dom: *
- ext-iconv: *
- ext-json: *
- fabpot/goutte: ^4.0
- illuminate/support: 6 - 10
- monolog/monolog: 1.24 - 3
Requires (Dev)
- orchestra/testbench: 4 - 8
- phpro/grumphp: ^1.0
- phpspec/prophecy-phpunit: ^2.0
- phpunit/phpunit: ^9.3
- symplify/easy-coding-standard: >=8.2
README
The most integrated web scraping package for Laravel.
Top Features
Scavenger provides the following features and more out-the-box.
- Ease of use
- Scavenger is super-easy to configure. Simply publish the config file and set your targets.
- Scrape data from multiple sources at once.
- Convert scraped data into usable Laravel model objects.
- eg. You may scrape an article, have it converted into an object of your choice, and save it in your database. Immediately available to your viewers.
- You can easily perform one or more operations, on each property, of any scraped entity.
- eg. You may call a paraphrase service from a model or package of your choice on data attributes before saving them to your database.
- Data integrity constraints
- Scavenger uses a hashing algorithm of your choice to maintain data integrity. This hash is used to ensure that one scrap (source article) is not converted to multiple output objects (model duplicates).
- Console Command
- Once Scavenger is configured, a simple artisan command launches the seeker. Since this is a console command it is more efficient and timeouts are less likely to occur.
- Artisan command:
php artisan scavenger:seek
- Schedule ready
- Scavenger can easily be set to scrape on a schedule. Hence, creating a somewhat autonomous website is super easy!
- SERP
- Scavenger can be used to flexibly scrape Search Engine Result Pages.
Installation
-
Install via composer; in your terminal:
composer require reliqarts/laravel-scavenger
or require in composer.json:
{ "require": { "reliqarts/laravel-scavenger": "^3.1" } }
then run
composer update
in your terminal to pull it in. -
(Optional) Publish package resources and configuration:
php artisan vendor:publish --provider="ReliqArts\Scavenger\ServiceProvider"
You may opt to publish only configuration by using the scavenger-config
tag:
php artisan vendor:publish --provider="ReliqArts\Scavenger\ServiceProvider" --tag="scavenger-config"
or only the migrations via the scavenger-migrations
tag:
php artisan vendor:publish --provider="ReliqArts\Scavenger\ServiceProvider" --tag="scavenger-migrations"
Configuration
Scavenger is highly configurable. Once configured, the settings will be used for every scrape.
Structure
Below is an example of a typical config file structure, with comments explaining each setting.
<?php return [ // debug mode? 'debug' => false, // whether log file should be written 'log' => true, // How much detail is expected in output, 1 being the lowest, 3 being highest. 'verbosity' => 1, // Set the database config 'database' => [ // Scraps table 'scraps_table' => env('SCAVENGER_SCRAPS_TABLE', 'scavenger_scraps'), ], // Daemon config - used to build daemon user 'daemon' => [ // Model to use for Daemon identification and login 'model' => 'App\\User', // Model property to check for daemon ID 'id_prop' => 'email', // Daemon ID 'id' => 'daemon@scavenger.reliqarts.com', // Any additional information required to create a user: // NB. this is only used when creating a daemon user, there is no "safe" way // to change the daemon's password once he has been created. 'info' => [ 'name' => 'Scavenger Daemon', 'password' => 'pass', ], ], // guzzle settings 'guzzle_settings' => [ 'timeout' => 60, ], // hashing algorithm to use 'hash_algorithm' => 'sha512', // storage 'storage' => [ // This directory will live inside your application's log directory. 'log_dir' => env('SCAVENGER_LOG_DIR', 'scavenger'), ], // different model entities and mapping information 'targets' => [ // NB. the "rooms" target shown below is for example purposes only. It has all posible keys explicitly. 'rooms' => [ 'example' => true, 'serp' => false, 'model' => 'App\\Room', 'source' => 'http://myroomslistingsite.1demo/section/rooms', 'search' => [ // keywords 'keywords' => ['professional'], // form markup 'form' => [ // search form selector (important) 'selector' => '#form', // input element name for search term/keyword 'keyword_input_name' => 'keyword', 'submit_button' => [ // text on submit button (optional) 'text' => null, // submit element id, use if button doesn't have text (optional) 'id' => null, ], ], ], 'pager' => [ // link (a tag) selector 'selector' => 'div.content #page a.pagingnav', ], // max. number of pages to scrape (0 is unlimited) 'pages' => 0, // content markup: actual data to be scraped 'markup' => [ 'title' => 'div.content section > table tr h3', // inside: content to be found upon clicking title link '__inside' => [ 'title' => '#ad-title > h1 > a', 'body' => 'article .adcontent > p[align="LEFT"]:last-of-type', // focus: focus detail on the following section '__focus' => 'section section > .content #ad-detail > article', ], // wrapper/item/result: wrapping selector for each item on single page. // If inside special key is set this key becomes invalid (i.e. inside takes preference) '__result' => null, ], // split single attributes into multiple based on regex 'dissect' => [ 'body' => [ 'email' => '(([eE]mail)*:*\s*\w+\@(\s*\w)*\.(net|com))', 'phone' => '((([cC]all|[[tT]el|[Pp][Hh](one)*)[:\d\-,\sDL\/]*\d)|(\d{3}\-?\d{4}))', 'beds' => '([\d]+[\d\.\/\s]*[^\w]*([Bb]edroom|b\/r|[Bb]ed)s?)', 'baths' => '([\d]+[\d\.\/\s]*[^\w]*([Bb]athroom|bth|[Bb]ath)s?)', // retain: whether details should be left in source attribute after extraction '__retain' => true, ], ], // modify attributes by calling functions 'preprocess' => [ // takes a callable // optional third parameter of array if callable method needs an instance // e.g. ['App\\Item', 'foo', true] or 'bar' 'title' => null, ], // remap entity attributes to model properties (optional) 'remap' => [ 'title' => null, 'body' => null, ], // scraps containing any of these words will be rejected (optional) 'bad_words' => [ 'office', ], ], // Google SERP example: 'google' => [ 'example' => true, 'serp' => true, 'model' => 'App\\GoogleResult', 'source' => 'https://www.google.com', 'search' => [ 'keywords' => ['dog'], 'form' => [ 'selector' => 'form[name="f"]', 'keyword_input_name' => 'q', ], ], 'pages' => 2, 'pager' => [ 'selector' => '#foot > table > tr > td.b:last-child a', ], 'markup' => [ '__result' => 'div.g', 'title' => 'h3 > a', 'description' => '.st', // the 'link' and 'position' attributes make use of some of Scavengers available properties 'link' => '__link', 'position' => '__position', ], ], // Bing SERP example: 'bing' => [ 'example' => true, 'serp' => true, 'model' => 'App\\BingResult', 'source' => 'https://www.bing.com', 'search' => [ 'keywords' => ['dog'], 'form' => [ 'selector' => 'form#sb_form', 'keyword_input_name' => 'q', ], ], 'pages' => 3, 'pager' => [ 'selector' => '.sb_pagN', ], 'markup' => [ '__result' => '.b_algo', 'title' => 'h2 a', 'description' => '.b_caption p', 'link' => '__link', 'position' => '__position', ], ], ], ];
Target Breakdown
The targets
array contains a list of entities (to be scraped from) keyed by a unique target identifier. The structure is as follows.
model
: Laravel DB model to create from target.source
: Source URL to scrape.search
: Search settings. Use if a search is to be performed before target data is shown. (optional)keywords
: Array of keywords to search for.keyword_input
: Keyword input text markup.form_markup
: CSS selector for search form.submit_button_text
: The text on the form's submit button.
pager
: Next link CSS selector. To skip to next page.markup
: Array of attributes to scrape from main list.[attributeName => CSS selector]
__inside
: Sub markup for detail page. Markup for page which shows when article title is clicked/opened. (optional)
dissect
: Split compound attributes into smaller attributes via REGEX. (optional)preprocess
: Array of attributes which need to be preprocessed.[attributeName => callable]
(optional)remap
: Array of attributes which need to be renamed in order to be saved as target objects.[attributeName => newName]
(optional)bad_words
: Any scrap found containing these words will be discarded. (optional)
Glossary of Terms
The following words may appear in context above.
Daemon
: User instance to be used by the Scavenger service.Scrap
: Scraped data before being converted to the target object.Target
: Configured source-model mapping for a single entity.Target Object
: Eloquent model object to be generated from scrape.
Acknowledgements
This library is heavily inspired by, and dependent on, the Guzzle library, although several concepts may have been adjusted.