spidra / spidra-php
Official PHP SDK for Spidra — AI-powered web scraping and crawling
Requires
- php: >=8.1
- guzzlehttp/guzzle: ^7.0
Requires (Dev)
- phpunit/phpunit: ^11.0
Suggests
None
Provides
None
Conflicts
None
Replaces
None
README
The official PHP SDK for Spidra — AI-native web scraping, multi-engine search, file & document parsing, deep site crawling, ready-made extraction templates, saved presets, and scheduled change monitors.
Features
- Web Scraping: Fast raw content extraction (Markdown, HTML, text, screenshots) or structured AI extraction with guaranteed JSON schemas.
- Browser Actions: Automated interactions: click, type, wait for selectors, infinite scroll, take element screenshots, and multi-step workflows.
- Web & Social Search: Multi-source web search (Google, Reddit, YouTube, TikTok, Hacker News, and more) with optional automatic page scraping.
- Domain Mapping: Fast URL discovery via sitemaps and site graphs with subdomain and robots.txt filtering.
- Document & Media Parsing: Parse PDFs, DOCX, spreadsheets, audio, and video files with AI or fast layout modes.
- Deep Site Crawling: Recursive crawling with regex/glob link matching, page limits, parallel workers, crawl reports, and per-page retries.
- Batch Scraping: Concurrent scraping of hundreds or thousands of URLs with automatic chunking and retry support.
- Ready-Made Templates: 100+ turnkey extraction templates for platforms like Google Maps, Amazon, flights, LinkedIn, and real estate.
- Saved Presets: Save and rerun common tool and template configurations with a single identifier.
- Scheduled Monitors: Watch pages for changes, evaluate CSS or AI conditions, and send webhook/email/Slack notifications.
- Webhook Verification: Secure HMAC-SHA256 signature verification helper.
- Account & Logs: Check credit balances, inspect proxy usage, and review detailed execution logs.
Requirements
- PHP >= 8.1
- Composer
ext-curlandext-json
Installation
composer require spidra/spidra-php
Get your API key at app.spidra.io under Settings → API Keys.
Quick Start
use Spidra\SpidraClient; $spidra = new SpidraClient('spd_YOUR_API_KEY'); // Scrape structured data with AI $result = $spidra->scrape([ 'urls' => ['https://news.ycombinator.com'], 'prompt' => 'Extract the top 5 articles with title, points, author, and URL', 'output' => 'json', ]); print_r($result['result']['content']);
Table of Contents
- Client Initialization
- Scraping
- Search
- Map
- Document Parsing
- Crawling
- Batch Scraping
- Ready-Made Templates
- Saved Presets
- Scheduled Monitors
- Account & Usage
- Execution Logs
- Webhook Verification
- Error Handling
Client Initialization
use Spidra\SpidraClient; $spidra = new SpidraClient( apiKey: 'spd_YOUR_API_KEY', baseUrl: 'https://api.spidra.io/api', // optional custom API URL timeout: 60 // request timeout in seconds );
Scraping
Simple scrape
Pass a URL string or an array of URLs:
$result = $spidra->scrape('https://example.com'); echo $result['result']['content']['markdown'];
Structured AI extraction with JSON schema
When you need a deterministic response structure, provide a JSON schema. Missing values are returned as null rather than hallucinated:
$result = $spidra->scrape([ 'urls' => ['https://jobs.example.com/staff-engineer'], 'prompt' => 'Extract the job listing details', 'output' => 'json', 'schema' => [ 'type' => 'object', 'required' => ['title', 'company', 'remote'], 'properties' => [ 'title' => ['type' => 'string'], 'company' => ['type' => 'string'], 'remote' => ['type' => ['boolean', 'null']], 'salary_min' => ['type' => ['number', 'null']], 'salary_max' => ['type' => ['number', 'null']], 'skills' => ['type' => 'array', 'items' => ['type' => 'string']], ], ], ]); print_r($result['result']['content']);
Browser actions & interactions
Execute user actions before content is extracted:
$result = $spidra->scrape([ 'urls' => [ [ 'url' => 'https://example.com/search', 'actions' => [ ['type' => 'type', 'selector' => 'input[name="q"]', 'text' => 'MacBook Pro'], ['type' => 'click', 'selector' => 'button[type="submit"]'], ['type' => 'wait_for_selector', 'selector' => '.product-card', 'timeout' => 5000], ['type' => 'scroll', 'direction' => 'down', 'amount' => 500], ], ], ], 'prompt' => 'Extract the first 5 search results', 'output' => 'json', ]);
Geo-targeted scraping
Route requests through residential or mobile proxies in specific countries:
$result = $spidra->scrape([ 'urls' => ['https://www.amazon.de/gp/bestsellers'], 'useProxy' => true, 'proxyCountry' => 'DE', // DE, US, GB, FR, JP, CA, etc. 'prompt' => 'Extract top 10 best-selling items', 'output' => 'json', ]);
Raw formats (Markdown, HTML, Screenshot)
Extract raw content directly without running AI prompts:
$result = $spidra->scrape([ 'urls' => ['https://news.ycombinator.com'], 'scrapeMode' => 'raw', 'formats' => ['markdown', 'html', 'screenshot'], ]); $item = $result['data'][0]; echo "Markdown: " . substr($item['markdown'], 0, 200) . "\n"; echo "Screenshot URL: " . $item['screenshot'] . "\n";
Search
Basic web search
Search across the web or specific channels:
$search = $spidra->search([ 'query' => 'latest breakthroughs in quantum computing', 'resultLimit' => 5, ]); foreach ($search['results'] as $item) { echo "Title: {$item['title']}\n"; echo "URL: {$item['url']}\n"; echo "Snippet: {$item['snippet']}\n\n"; }
Search with automatic page scraping
Fetch the full content of search results in the same call:
$search = $spidra->search([ 'query' => 'OpenAI o3 model architecture', 'resultLimit' => 3, 'scrapeContent' => true, 'scrapeOptions' => [ 'formats' => ['markdown'], 'maxResults' => 3, ], ]); foreach ($search['results'] as $item) { echo "URL: " . $item['url'] . "\n"; echo "Scraped Markdown: " . substr($item['scrapedContent']['markdown'] ?? '', 0, 150) . "...\n\n"; }
Domain filtering & specialized sources
$search = $spidra->search([ 'query' => 'best mechanical keyboards for coding', 'sources' => ['google', 'reddit'], 'includeDomains' => ['reddit.com', 'geekhack.org'], 'timeRange' => 'past_month', // past_24h, past_week, past_month, past_year 'country' => 'US', ]);
Map
Discover all URLs on a domain using sitemaps and site link graphs:
$map = $spidra->map([ 'url' => 'https://docs.github.com', 'search' => 'rest api', // filter links matching substring 'urlLimit' => 100, // maximum URLs to return 'includeSubdomains' => false, 'respectRobots' => true, ]); foreach ($map['links'] as $link) { echo $link . "\n"; }
Document Parsing
Extract clean markdown and metadata from PDF, DOCX, XLSX, CSV, audio, or video files:
// Parse via URL or uploaded document $job = $spidra->parse([ 'url' => 'https://example.com/reports/financial-q3.pdf', 'mode' => 'ai', // 'ai' for smart reasoning / OCR or 'fast' for high-speed layout 'prompt' => 'Extract revenue, gross margin, and operating expenses by segment', ]); echo $job['markdown'] ?? ''; print_r($job['structuredData'] ?? []);
You can also estimate credit costs before running a parse job:
$estimate = $spidra->parse->estimate([ 'url' => 'https://example.com/large-book.pdf', 'mode' => 'ai', ]); echo "Estimated credits: {$estimate['credits']}\n";
Crawling
Start and wait for a crawl
Recursively crawl an entire website:
$crawl = $spidra->crawl([ 'url' => 'https://docs.stripe.com', 'maxPages' => 25, 'maxDepth' => 2, 'allowPatterns' => ['https://docs.stripe.com/api/.*'], 'ignorePatterns' => ['.*\\.pdf$', '.*/changelog/.*'], 'prompt' => 'Extract the endpoint title, HTTP method, and path', 'output' => 'json', ]); echo "Crawl completed: {$crawl['status']}\n"; echo "Pages crawled: {$crawl['progress']['pagesCrawled']}\n"; // Fetch crawled pages $pages = $spidra->crawl->pages($crawl['jobId']); foreach ($pages['pages'] as $page) { echo "{$page['url']} ({$page['status']})\n"; }
Crawl reports and page retries
// Download a diagnostic report of skipped URLs and errors $report = $spidra->crawl->report($crawl['jobId']); print_r($report['skipped']); // Retry a specific page that previously failed $retry = $spidra->crawl->retryPage($crawl['jobId'], 'page-uuid-here');
Batch Scraping
Scrape up to thousands of URLs with automatic chunking and retry resilience:
$batch = $spidra->batchScrape([ 'urls' => [ 'https://example.com/product/1', 'https://example.com/product/2', 'https://example.com/product/3', ], 'prompt' => 'Extract the product name, price, and rating', 'output' => 'json', 'useProxy' => true, ]); foreach ($batch['items'] as $item) { echo "URL: {$item['url']} - Status: {$item['status']}\n"; print_r($item['result']); }
Ready-Made Templates
Run pre-built turnkey scrapers for popular platforms:
// List available templates $templates = $spidra->templates->list(); foreach ($templates['templates'] as $tpl) { echo "{$tpl['slug']} — {$tpl['title']}\n"; } // Run Google Maps reviews template $run = $spidra->runTemplate('google-maps-reviews', [ 'place_id' => 'ChIJN1t_tDeuEmsRUsoyG83frY4', 'max_reviews' => 20, ]); print_r($run['data']);
Saved Presets
Save extraction configurations to run again with a single identifier:
// Create a preset $presetId = $spidra->presets->create([ 'kind' => 'tool', 'name' => 'Daily TechCrunch Scrape', 'description' => 'Extracts top news from TechCrunch homepage', 'tool' => 'scrape', 'config' => [ 'urls' => ['https://techcrunch.com'], 'prompt' => 'Extract top headlines with author and summary', 'output' => 'json', ], ]); // Execute the preset anytime $result = $spidra->runPreset($presetId); print_r($result);
Scheduled Monitors
Monitor pages for changes, evaluate CSS or AI conditions, and send alerts:
// Create a scheduled monitor $monitor = $spidra->monitors->create([ 'name' => 'Competitor Pricing Watch', 'schedule' => [ 'cron' => '0 9 * * *', // daily at 9am 'timezone' => 'America/New_York', ], 'sources' => [ [ 'type' => 'url', 'url' => 'https://competitor.com/pricing', ], ], 'rule' => [ 'type' => 'ai', 'instruction' => 'Alert if the Pro plan price drops below $49/mo', ], 'notify' => [ 'channels' => [ [ 'type' => 'webhook', 'url' => 'https://api.mycompany.com/alerts/pricing', ], [ 'type' => 'email', 'email' => 'alerts@mycompany.com', ], ], ], ]); // Trigger an immediate check $check = $spidra->monitors->checkNow($monitor['id']);
Account & Usage
// Get plan, remaining credits, and proxy bandwidth $account = $spidra->account->get(); echo "Plan: {$account['plan']['name']}\n"; echo "Credits Remaining: {$account['credits']['remaining']}\n"; echo "Proxy Bandwidth Remaining: {$account['proxyBandwidth']['remainingMb']} MB\n"; // Check usage history $usage = $spidra->account->usage('30d'); // Check tool-by-tool breakdown $breakdown = $spidra->account->breakdown('7d');
Execution Logs
Inspect historical jobs and execution logs:
// Scrape logs $scrapeLogs = $spidra->logs->scrape(searchTerm: 'amazon.com', limit: 10); // Search logs $searchLogs = $spidra->logs->search(limit: 10); // Map logs $mapLogs = $spidra->logs->map(limit: 10); // Parse logs $parseLogs = $spidra->logs->parse(limit: 10);
Webhook Verification
Verify incoming webhooks from Spidra using HMAC-SHA256:
use Spidra\Webhook; $payload = file_get_contents('php://input'); $signature = $_SERVER['HTTP_X_SPIDRA_SIGNATURE'] ?? ''; $secret = 'whsec_YOUR_WEBHOOK_SECRET'; if (!Webhook::verify($payload, $signature, $secret)) { http_response_code(401); exit('Invalid signature'); } $event = json_decode($payload, true); // Process event...
Error Handling
Every API error throws a typed exception inheriting from SpidraException:
use Spidra\Exceptions\AuthenticationException; use Spidra\Exceptions\InsufficientCreditsException; use Spidra\Exceptions\RateLimitException; use Spidra\Exceptions\ServerException; use Spidra\Exceptions\ValidationException; use Spidra\Exceptions\SpidraException; try { $result = $spidra->scrape('https://example.com'); } catch (AuthenticationException $e) { // 401: Invalid or missing API key echo "Auth error: {$e->getMessage()}\n"; } catch (InsufficientCreditsException $e) { // 402/403: Monthly quota exceeded echo "Out of credits: {$e->getMessage()}\n"; } catch (ValidationException $e) { // 422: Request parameter invalid echo "Validation error: {$e->getMessage()}\n"; } catch (RateLimitException $e) { // 429: Too many requests echo "Rate limit reached. Retry after delay.\n"; } catch (ServerException $e) { // 500/503: Spidra server error echo "Server error: {$e->getMessage()}\n"; } catch (SpidraException $e) { // Base catch-all for any other SDK exception echo "Spidra error [{$e->statusCode}]: {$e->getMessage()}\n"; }
License
MIT © Spidra