yoozi/miner

PHP library to extract the metadata from a public web page and/or summarize it.

1.0.3 2014-07-24 08:56 UTC

Requires

• php: >=5.3.0
• illuminate/support: ~4.0
• jeremykendall/php-domain-parser: ~1.3
• kriswallsmith/buzz: ~0.10

Requires (Dev)

• phpunit/phpunit: 3.7.*

MIT

• Saturn HU

• metadata
• webpage
• golem
• yoozi
• summarize
• readability

README


This library is part of Project Golem, see yoozi/golem for more info.

Miner is a PHP library that extracting metadata and interesting text content (like author, summary, and etc.) from HTML pages. It acts like a simplified HTML metadata parser in Apache Tika.

WTF is Miner?

Ta-da! Consider the screenshot taken from LinkedIn below:

When you post a link to your connections on LinkedIn, it will automatically extract the title, summary, and even cover image for you. Miner can be typically used to achieve tasks like this.

Installation

The best and easy way to install the Golem package is with Composer.

1. Open your composer.json and add the following to the require array:

   "yoozi/miner": "1.0.*"
   

2. Run Composer to install or update the new package dependencies.

   php composer install
   

   or

   php composer update
   

Usage

Parsers

• Meta: Summarize a webpage by parsing its HTML meta tags. In most cases it favors Open Graph (OG) markup, and will fall back to standard meta tags if necessary.
• Readability: Summarize a webpage using Arc90's Readability alogrithm. All credit goes to @feelinglucky's PHP Port.
• Hybrid: In combination with the above two parsers, it simply takes Readability as the primary parser, and Meta as its fallback.

Hybrid is enabled by default. You can change parsers to best fit your needs:

// Use the Readability Parser.
$extractor->getConfig()->set('parser', 'readability');

// Or...use the Hybrid Parser.
// $extractor->getConfig()->set('parser', 'hybrid');
// Or...use the Meta Parser.
// $extractor->getConfig()->set('parser', 'meta');

Example

We can parse a remote url and extract its metadata directly.

<?php

use Yoozi\Miner\Extractor;
use Buzz\Client\Curl;

$extractor = new Extractor();

// Use the Hybrid Parser.
$extractor->getConfig()->set('parser', 'hybrid');
// Strip all HTML tags in the description we parsed.
$extractor->getConfig()->set('strip_tags', true);

$meta = $extractor->fromUrl('http://www.example.com/', new Curl)->run();
var_dump($meta);

Data returned:

array(9) {
  ["title"]=>
  string(14) "Example Domain"
  ["author"]=>
  NULL
  ["keywords"]=>
  array(0) {
  }
  ["description"]=>
  string(220) "
    Example Domain
    This domain is established to be used for illustrative examples in documents. You may use this
    domain in examples without prior coordination or asking for permission.
    More information...
"
  ["image"]=>
  NULL
  ["url"]=>
  string(23) "http://www.example.com/"
  ["host"]=>
  string(22) "http://www.example.com"
  ["domain"]=>
  string(11) "example.com"
  ["favicon"]=>
  string(52) "http://www.google.com/s2/favicons?domain=example.com"
}