Search by

spidra / spidra-php

olawanlejoel

Official PHP SDK for Spidra — AI-powered web scraping and crawling

v0.2.0 2026-10-27 22:14 UTC

This package is auto-updated.

Last update: 2026-10-27 22:15:59 UTC


README

The official PHP SDK for Spidra — AI-native web scraping, multi-engine search, file & document parsing, deep site crawling, ready-made extraction templates, saved presets, and scheduled change monitors.

Packagist Version License: MIT PHP Version

Features

  • Web Scraping: Fast raw content extraction (Markdown, HTML, text, screenshots) or structured AI extraction with guaranteed JSON schemas.
  • Browser Actions: Automated interactions: click, type, wait for selectors, infinite scroll, take element screenshots, and multi-step workflows.
  • Web & Social Search: Multi-source web search (Google, Reddit, YouTube, TikTok, Hacker News, and more) with optional automatic page scraping.
  • Domain Mapping: Fast URL discovery via sitemaps and site graphs with subdomain and robots.txt filtering.
  • Document & Media Parsing: Parse PDFs, DOCX, spreadsheets, audio, and video files with AI or fast layout modes.
  • Deep Site Crawling: Recursive crawling with regex/glob link matching, page limits, parallel workers, crawl reports, and per-page retries.
  • Batch Scraping: Concurrent scraping of hundreds or thousands of URLs with automatic chunking and retry support.
  • Ready-Made Templates: 100+ turnkey extraction templates for platforms like Google Maps, Amazon, flights, LinkedIn, and real estate.
  • Saved Presets: Save and rerun common tool and template configurations with a single identifier.
  • Scheduled Monitors: Watch pages for changes, evaluate CSS or AI conditions, and send webhook/email/Slack notifications.
  • Webhook Verification: Secure HMAC-SHA256 signature verification helper.
  • Account & Logs: Check credit balances, inspect proxy usage, and review detailed execution logs.

Requirements

  • PHP >= 8.1
  • Composer
  • ext-curl and ext-json

Installation

composer require spidra/spidra-php

Get your API key at app.spidra.io under Settings → API Keys.

Quick Start

use Spidra\SpidraClient;

$spidra = new SpidraClient('spd_YOUR_API_KEY');

// Scrape structured data with AI
$result = $spidra->scrape([
    'urls'   => ['https://news.ycombinator.com'],
    'prompt' => 'Extract the top 5 articles with title, points, author, and URL',
    'output' => 'json',
]);

print_r($result['result']['content']);

Table of Contents

Client Initialization

use Spidra\SpidraClient;

$spidra = new SpidraClient(
    apiKey: 'spd_YOUR_API_KEY',
    baseUrl: 'https://api.spidra.io/api', // optional custom API URL
    timeout: 60                           // request timeout in seconds
);

Scraping

Simple scrape

Pass a URL string or an array of URLs:

$result = $spidra->scrape('https://example.com');
echo $result['result']['content']['markdown'];

Structured AI extraction with JSON schema

When you need a deterministic response structure, provide a JSON schema. Missing values are returned as null rather than hallucinated:

$result = $spidra->scrape([
    'urls'   => ['https://jobs.example.com/staff-engineer'],
    'prompt' => 'Extract the job listing details',
    'output' => 'json',
    'schema' => [
        'type'       => 'object',
        'required'   => ['title', 'company', 'remote'],
        'properties' => [
            'title'      => ['type' => 'string'],
            'company'    => ['type' => 'string'],
            'remote'     => ['type' => ['boolean', 'null']],
            'salary_min' => ['type' => ['number', 'null']],
            'salary_max' => ['type' => ['number', 'null']],
            'skills'     => ['type' => 'array', 'items' => ['type' => 'string']],
        ],
    ],
]);

print_r($result['result']['content']);

Browser actions & interactions

Execute user actions before content is extracted:

$result = $spidra->scrape([
    'urls' => [
        [
            'url' => 'https://example.com/search',
            'actions' => [
                ['type' => 'type', 'selector' => 'input[name="q"]', 'text' => 'MacBook Pro'],
                ['type' => 'click', 'selector' => 'button[type="submit"]'],
                ['type' => 'wait_for_selector', 'selector' => '.product-card', 'timeout' => 5000],
                ['type' => 'scroll', 'direction' => 'down', 'amount' => 500],
            ],
        ],
    ],
    'prompt' => 'Extract the first 5 search results',
    'output' => 'json',
]);

Geo-targeted scraping

Route requests through residential or mobile proxies in specific countries:

$result = $spidra->scrape([
    'urls'         => ['https://www.amazon.de/gp/bestsellers'],
    'useProxy'     => true,
    'proxyCountry' => 'DE', // DE, US, GB, FR, JP, CA, etc.
    'prompt'       => 'Extract top 10 best-selling items',
    'output'       => 'json',
]);

Raw formats (Markdown, HTML, Screenshot)

Extract raw content directly without running AI prompts:

$result = $spidra->scrape([
    'urls'       => ['https://news.ycombinator.com'],
    'scrapeMode' => 'raw',
    'formats'    => ['markdown', 'html', 'screenshot'],
]);

$item = $result['data'][0];
echo "Markdown: " . substr($item['markdown'], 0, 200) . "\n";
echo "Screenshot URL: " . $item['screenshot'] . "\n";

Search

Basic web search

Search across the web or specific channels:

$search = $spidra->search([
    'query'       => 'latest breakthroughs in quantum computing',
    'resultLimit' => 5,
]);

foreach ($search['results'] as $item) {
    echo "Title: {$item['title']}\n";
    echo "URL:   {$item['url']}\n";
    echo "Snippet: {$item['snippet']}\n\n";
}

Search with automatic page scraping

Fetch the full content of search results in the same call:

$search = $spidra->search([
    'query'         => 'OpenAI o3 model architecture',
    'resultLimit'   => 3,
    'scrapeContent' => true,
    'scrapeOptions' => [
        'formats'    => ['markdown'],
        'maxResults' => 3,
    ],
]);

foreach ($search['results'] as $item) {
    echo "URL: " . $item['url'] . "\n";
    echo "Scraped Markdown: " . substr($item['scrapedContent']['markdown'] ?? '', 0, 150) . "...\n\n";
}

Domain filtering & specialized sources

$search = $spidra->search([
    'query'          => 'best mechanical keyboards for coding',
    'sources'        => ['google', 'reddit'],
    'includeDomains' => ['reddit.com', 'geekhack.org'],
    'timeRange'      => 'past_month', // past_24h, past_week, past_month, past_year
    'country'        => 'US',
]);

Map

Discover all URLs on a domain using sitemaps and site link graphs:

$map = $spidra->map([
    'url'               => 'https://docs.github.com',
    'search'            => 'rest api',      // filter links matching substring
    'urlLimit'          => 100,             // maximum URLs to return
    'includeSubdomains' => false,
    'respectRobots'     => true,
]);

foreach ($map['links'] as $link) {
    echo $link . "\n";
}

Document Parsing

Extract clean markdown and metadata from PDF, DOCX, XLSX, CSV, audio, or video files:

// Parse via URL or uploaded document
$job = $spidra->parse([
    'url'    => 'https://example.com/reports/financial-q3.pdf',
    'mode'   => 'ai', // 'ai' for smart reasoning / OCR or 'fast' for high-speed layout
    'prompt' => 'Extract revenue, gross margin, and operating expenses by segment',
]);

echo $job['markdown'] ?? '';
print_r($job['structuredData'] ?? []);

You can also estimate credit costs before running a parse job:

$estimate = $spidra->parse->estimate([
    'url'  => 'https://example.com/large-book.pdf',
    'mode' => 'ai',
]);

echo "Estimated credits: {$estimate['credits']}\n";

Crawling

Start and wait for a crawl

Recursively crawl an entire website:

$crawl = $spidra->crawl([
    'url'            => 'https://docs.stripe.com',
    'maxPages'       => 25,
    'maxDepth'       => 2,
    'allowPatterns'  => ['https://docs.stripe.com/api/.*'],
    'ignorePatterns' => ['.*\\.pdf$', '.*/changelog/.*'],
    'prompt'         => 'Extract the endpoint title, HTTP method, and path',
    'output'         => 'json',
]);

echo "Crawl completed: {$crawl['status']}\n";
echo "Pages crawled:   {$crawl['progress']['pagesCrawled']}\n";

// Fetch crawled pages
$pages = $spidra->crawl->pages($crawl['jobId']);
foreach ($pages['pages'] as $page) {
    echo "{$page['url']} ({$page['status']})\n";
}

Crawl reports and page retries

// Download a diagnostic report of skipped URLs and errors
$report = $spidra->crawl->report($crawl['jobId']);
print_r($report['skipped']);

// Retry a specific page that previously failed
$retry = $spidra->crawl->retryPage($crawl['jobId'], 'page-uuid-here');

Batch Scraping

Scrape up to thousands of URLs with automatic chunking and retry resilience:

$batch = $spidra->batchScrape([
    'urls'     => [
        'https://example.com/product/1',
        'https://example.com/product/2',
        'https://example.com/product/3',
    ],
    'prompt'   => 'Extract the product name, price, and rating',
    'output'   => 'json',
    'useProxy' => true,
]);

foreach ($batch['items'] as $item) {
    echo "URL: {$item['url']} - Status: {$item['status']}\n";
    print_r($item['result']);
}

Ready-Made Templates

Run pre-built turnkey scrapers for popular platforms:

// List available templates
$templates = $spidra->templates->list();
foreach ($templates['templates'] as $tpl) {
    echo "{$tpl['slug']} — {$tpl['title']}\n";
}

// Run Google Maps reviews template
$run = $spidra->runTemplate('google-maps-reviews', [
    'place_id'    => 'ChIJN1t_tDeuEmsRUsoyG83frY4',
    'max_reviews' => 20,
]);

print_r($run['data']);

Saved Presets

Save extraction configurations to run again with a single identifier:

// Create a preset
$presetId = $spidra->presets->create([
    'kind'        => 'tool',
    'name'        => 'Daily TechCrunch Scrape',
    'description' => 'Extracts top news from TechCrunch homepage',
    'tool'        => 'scrape',
    'config'      => [
        'urls'   => ['https://techcrunch.com'],
        'prompt' => 'Extract top headlines with author and summary',
        'output' => 'json',
    ],
]);

// Execute the preset anytime
$result = $spidra->runPreset($presetId);
print_r($result);

Scheduled Monitors

Monitor pages for changes, evaluate CSS or AI conditions, and send alerts:

// Create a scheduled monitor
$monitor = $spidra->monitors->create([
    'name'     => 'Competitor Pricing Watch',
    'schedule' => [
        'cron'     => '0 9 * * *', // daily at 9am
        'timezone' => 'America/New_York',
    ],
    'sources'  => [
        [
            'type' => 'url',
            'url'  => 'https://competitor.com/pricing',
        ],
    ],
    'rule'     => [
        'type'        => 'ai',
        'instruction' => 'Alert if the Pro plan price drops below $49/mo',
    ],
    'notify'   => [
        'channels' => [
            [
                'type' => 'webhook',
                'url'  => 'https://api.mycompany.com/alerts/pricing',
            ],
            [
                'type'  => 'email',
                'email' => 'alerts@mycompany.com',
            ],
        ],
    ],
]);

// Trigger an immediate check
$check = $spidra->monitors->checkNow($monitor['id']);

Account & Usage

// Get plan, remaining credits, and proxy bandwidth
$account = $spidra->account->get();
echo "Plan: {$account['plan']['name']}\n";
echo "Credits Remaining: {$account['credits']['remaining']}\n";
echo "Proxy Bandwidth Remaining: {$account['proxyBandwidth']['remainingMb']} MB\n";

// Check usage history
$usage = $spidra->account->usage('30d');

// Check tool-by-tool breakdown
$breakdown = $spidra->account->breakdown('7d');

Execution Logs

Inspect historical jobs and execution logs:

// Scrape logs
$scrapeLogs = $spidra->logs->scrape(searchTerm: 'amazon.com', limit: 10);

// Search logs
$searchLogs = $spidra->logs->search(limit: 10);

// Map logs
$mapLogs = $spidra->logs->map(limit: 10);

// Parse logs
$parseLogs = $spidra->logs->parse(limit: 10);

Webhook Verification

Verify incoming webhooks from Spidra using HMAC-SHA256:

use Spidra\Webhook;

$payload = file_get_contents('php://input');
$signature = $_SERVER['HTTP_X_SPIDRA_SIGNATURE'] ?? '';
$secret = 'whsec_YOUR_WEBHOOK_SECRET';

if (!Webhook::verify($payload, $signature, $secret)) {
    http_response_code(401);
    exit('Invalid signature');
}

$event = json_decode($payload, true);
// Process event...

Error Handling

Every API error throws a typed exception inheriting from SpidraException:

use Spidra\Exceptions\AuthenticationException;
use Spidra\Exceptions\InsufficientCreditsException;
use Spidra\Exceptions\RateLimitException;
use Spidra\Exceptions\ServerException;
use Spidra\Exceptions\ValidationException;
use Spidra\Exceptions\SpidraException;

try {
    $result = $spidra->scrape('https://example.com');
} catch (AuthenticationException $e) {
    // 401: Invalid or missing API key
    echo "Auth error: {$e->getMessage()}\n";
} catch (InsufficientCreditsException $e) {
    // 402/403: Monthly quota exceeded
    echo "Out of credits: {$e->getMessage()}\n";
} catch (ValidationException $e) {
    // 422: Request parameter invalid
    echo "Validation error: {$e->getMessage()}\n";
} catch (RateLimitException $e) {
    // 429: Too many requests
    echo "Rate limit reached. Retry after delay.\n";
} catch (ServerException $e) {
    // 500/503: Spidra server error
    echo "Server error: {$e->getMessage()}\n";
} catch (SpidraException $e) {
    // Base catch-all for any other SDK exception
    echo "Spidra error [{$e->statusCode}]: {$e->getMessage()}\n";
}

License

MIT © Spidra