crwlr / crwl-extension-utils
Utils for extension packages for the crwl.io app.
Fund package maintenance!
otsch
Installs: 1 111
Dependents: 2
Suggesters: 0
Security: 0
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Requires
- php: ^8.1
- crwlr/crawler: ^1.8|^2.0
- illuminate/support: ^9.27|^10.0|^11.0
Requires (Dev)
- friendsofphp/php-cs-fixer: ^3.57
- orchestra/testbench: ^8.21|^9.0
- pestphp/pest: ^2.4|^3.0
- pestphp/pest-plugin-laravel: ^2.2|^3.0
- phpstan/phpstan: ^1.10
README
These utilities enable you to develop extensions for the crwl.io web crawling and scraping app.
How to Package Your Custom Steps for the crwl.io App
The crwlr.software documentation provides a detailed explanation of how to build your own steps for the crwlr/crawler library.
Once you've successfully created a working step, the most challenging part of your task is already complete. To integrate your custom steps into the crwl.io app, follow these simple steps:
- Package it as a Composer package in a GitHub repository.
- Create a
StepBuilder
- Create a laravel
ServiceProvider
and register your extension steps using theExtensionPackageManager
included in this package.
Composer Package Setup
Ensure your package repository's composer.json
file resembles the following template:
{ "name": "my-vendor/my-crwl-extension", "description": "Extension package with custom steps for the crwl.io app", "type": "library", "autoload": { "psr-4": { "MyVendor\\MyCrwlExtension\\": "src/" } }, "require": { "crwlr/crawler": "^1.5", "illuminate/support": "^9.27|^10.0", "crwlr/crwl-extension-utils": "^1.0" } }
Feel free to customize it according to your preferences, but ensure that it includes dependencies on the following three packages: crwlr/crawler
, crwlr/crwl-extension-utils
and illuminate/support
.
For a well-organized project structure, we recommend the following folder arrangement:
/
├─ src/
│ ├─ StepBuilders/
│ ├─ Steps/
│ ├─ ServiceProvider.php
├─ .gitignore
├─ composer.json
├─ README.md
In your .gitignore
, include at least the following entries:
/vendor/ composer.lock
Step Builders
To create a StepBuilder
for your step, follow the example below:
namespace MyVendor\MyCrwlExtension\StepBuilders; use Crwlr\Crawler\Steps\StepInterface; use Crwlr\Crawler\Steps\StepOutputType; use Crwlr\CrwlExtensionUtils\ConfigParam; use Crwlr\CrwlExtensionUtils\StepBuilder; use MyVendor\MyCrwlExtension\Steps\MyStep; class MyStepBuilder extends StepBuilder { public function stepId(): string { return 'my-extension.my-step'; } public function label(): string { return 'This step does X.'; } public function configToStep(array $stepConfig): StepInterface { $fooConfigValue = $this->getValueFromConfigArray('foo', $stepConfig); $barConfigValue = $this->getValueFromConfigArray('bar', $stepConfig); $bazConfigValue = $this->getValueFromConfigArray('baz', $stepConfig); return new MyStep($fooConfigValue, $barConfigValue); } public function configParams(): array { return [ ConfigParam::string('foo') ->inputLabel('Your foo'), ConfigParam::int('bar') ->default(5) ->inputLabel('Number of bar') ->description('Provide the number of bar, so the step can do X.'), ConfigParam::bool('baz') ->inputLabel('Baz?'), ]; } /** * Define the possible output type. More info at * https://www.crwlr.software/packages/crawler/v1.10/steps-and-data-flow/custom-steps#step-output-types * If the underlying step can yield different output types based on the configuration, * just return StepOutputType::Mixed. */ public function outputType(): StepOutputType { return StepOutputType::AssociativeArrayOrObject; } }
If your step requires configuration, define the necessary parameters in the configParams()
method. The crwl.io app's crawler creation/editing form will display corresponding inputs for these config options. When the crawler runs, the StepBuilder::configToStep()
method is invoked with the user-saved configuration data. In this method, construct your custom step with the configured values and return it.
Currently, the available config param types are string
, int
, and bool
. Optionally, you can specify a default value (default()
), an input label (inputLabel()
), and a description text (description()
) for each configuration parameter.
If your step doesn't require any settings, the StepBuilder
looks rather minimalistic:
namespace MyVendor\MyCrwlExtension\StepBuilders; use Crwlr\Crawler\Steps\StepInterface; use Crwlr\Crawler\Steps\StepOutputType;use Crwlr\CrwlExtensionUtils\StepBuilder; use MyVendor\MyCrwlExtension\Steps\MyStep; class MyStepBuilder extends StepBuilder { public function stepId(): string { return 'my-extension.my-step'; } public function label(): string { return 'This step does X.'; } public function configToStep(array $stepConfig): StepInterface { return new MyStep(); } public function outputType(): StepOutputType { return StepOutputType::Scalar; } }
If your step needs a filesystem path, where it can store files, you can use $this->fileStoragePath
inside the builder. The crwl.io app sets this path for all step builders before building any steps.
public function configToStep(array $stepConfig): StepInterface { return new MyStep($this->fileStoragePath); }
When creating a step builder for an HTTP loading step, please implement the following method:
public function isLoadingStep(): bool { return true; }
ServiceProvider and Registering Package and Steps
To make your steps accessible in the crwl.io app, the final step is to register an extension package and all your steps using the ExtensionPackageManager
included in this package. Since crwl.io is a Laravel application, this is accomplished through a ServiceProvider
class:
namespace MyVendor\MyCrwlExtension; use Crwlr\CrwlExtensionUtils\ExtensionPackageManager; use MyCrwlExtension\StepBuilders\FooStepBuilder; use MyCrwlExtension\StepBuilders\BarStepBuilder; use MyCrwlExtension\StepBuilders\BazStepBuilder; class ServiceProvider extends \Illuminate\Support\ServiceProvider { public function register() { $this->app->make(ExtensionPackageManager::class) ->registerPackage('my-vendor-name/my-crwl-extension') ->registerStep(FooStepBuilder::class) ->registerStep(BarStepBuilder::class) ->registerStep(BazStepBuilder::class); } }
To complete the setup, add the ServiceProvider
to the extra
section in the composer.json
file:
{ "name": "my-vendor/my-crwl-extension", "description": "Extension package with custom steps for the crwl.io app", "type": "library", "autoload": { "psr-4": { "MyVendor\\MyCrwlExtension\\": "src/" } }, "require": { "crwlr/crawler": "^1.4", "illuminate/support": "^9.27|^10.0", "crwlr/crwl-extension-utils": "^1.0" }, "extra": { "laravel": { "providers": [ "MyVendor\\MyCrwlExtension\\ServiceProvider" ] } } }
With these configurations in place, your extension package is ready for use. If your extension is private, ensure you grant access to the crwlrsoft GitHub organization. As a super-user on your crwl.io instance, you can then install your extension via the extensions page in the app.
Custom Steps Performing HTTP Requests Without the crwlr/crawler HttpLoader
In scenarios where your custom steps need to execute HTTP requests that cannot leverage the HttpLoader
from the crwlr/crawler
package—such as when utilizing a REST API SDK to retrieve data from an API—you'll need to ensure that every HTTP request is tracked when executing your custom steps within the crwl.io app.
To accomplish this, you have two options:
- Use a Guzzle Client instance generated by the
TrackingGuzzleClientFactory
. - Alternatively, manually invoke the
trackHttpResponse()
ortrackHeadlessBrowserResponse()
methods of theRequestTracker
following each request.
Using a guzzle Client instance
If you want to use a Guzzle Client
instance for the requests (it's common practice for PHP API SDKs to let you provide your own Guzzle instance), use the TrackingGuzzleClientFactory
from this package:
use Crwlr\CrwlExtensionUtils\TrackingGuzzleClientFactory; // Let the factory be resolved by the laravel service container. $factory = app()->make(TrackingGuzzleClientFactory::class); $client = $factory->getClient();
You can also pass your custom Guzzle configuration as an argument:
$client = $factory->getClient(['allow_redirects' => false]);
Using the RequestTracker
In scenarios where utilizing a Guzzle Client
instance for requests is not feasible, you need to call either RequestTracker::trackHttpResponse()
or if your request was executed using a headless browser RequestTracker::trackHeadlessBrowserResponse()
.
use Crwlr\CrwlExtensionUtils\RequestTracker; // Let the tracker be resolved by the laravel service container. $tracker = app()->make(RequestTracker::class); // Execute your request however you want... $tracker->trackHttpResponse(); // or $tracker->trackHeadlessBrowserResponse();
If you can provide request/response instances implementing the PSR-7 RequestInterface
and/or ResponseInterface
, please do so:
$tracker->trackHttpResponse($request, $response); // or $tracker->trackHeadlessBrowserResponse($request, $response);