survos/scraper-bundle

Scrape and cache web pages

1.5.220 2024-04-02 13:26 UTC

Requires

• php: ^8.1
• ext-pdo: *
• symfony/config: ^6.3 || ^7.0
• symfony/dependency-injection: ^6.3 || ^7.0
• symfony/http-client: ^6.3 || ^7.0
• symfony/http-kernel: ^6.3 || ^7.0
• symfony/string: ^7.0

Requires (Dev)

• phpstan/phpstan: ^1.10
• phpunit/phpunit: ^9.5||^10.0||^11.0
• psr/log: ^3.0
• symfony/browser-kit: ^6.3 || ^7.0
• symfony/cache: ^6.3 || ^7.0
• symfony/framework-bundle: ^6.3 || ^7.0
• symfony/phpunit-bridge: ^6.3 || ^7.0
• symfony/twig-bundle: ^6.3 || ^7.0
• symfony/var-dumper: ^6.3 || ^7.0
• twig/twig: ^3.4

MIT

• symfony-ux
• symfony6

README

A Symfony bundle that allows a disk-based web scaper cache.

It also allows a fetch to happen from twig. While this is not a good practice in production, it can speed up prototyping and demos.

Eventually this will be a real cache adapter, but for the moment simply fetching web pages to local storage is sufficient.

After installing the bundle,

Installation

composer req survos/scraper-bundle

If you're not using Flex, enable the bundle by adding the class to bundles.php

// config/bundles.php
<?php

return [
    //...
    Survos\Bundle\SurvosScraperBundle::class => ['all' => true],
    //...
];

Working Demo

Cut and paste the following to see it in action.

symfony new --webapp scraper-bundle-demo && cd scraper-bundle-demo
composer req survos/scraper-bundle
symfony console make:controller AppController
sed -i "s|/app|/|" src/Controller/AppController.php 

cat <<'EOF' > templates/app/index.html.twig
{% extends 'base.html.twig' %}
{% block body %}
    {% set url = 'https://jsonplaceholder.typicode.com/users' %}
    {% set users = request_data(url) %}
    <ul>
        {% for row in users %}
            <li>{{ row.name }} / {{ row.website }}</li>
        {% endfor %}
    </ul>
{% endblock %}
EOF
symfony server:start -d
symfony open:local

When you refresh the page, it will use the cached data and be much faster. To see the fetch in the debug toolbar, clear the cache and reload.

bin/console cache:pool:clear --all
symfony open:local

To use in a service or controller, inject the cache.

    public function index(ScraperService $scraper): Response
    {
        $data = $scraper->fetchData('https://jsonplaceholder.typicode.com/albums', asData: 'object');
        
    }