README

Channel-agnostic import engine for SilverShop (CMS 6). A two-phase stage → apply pipeline that thin per-system connectors plug into without the core knowing any system specifics — the same core + satellite seam as silvershop-suite/address-lookup (→ myparcel/pro6pp) and silvershop-suite/product-feed (→ product-feed-google, the mirror-image direction).

The core ships no connector: a fresh install imports nothing until a connector module registers a source. The reference connectors are:

silvershop-suite/import-excel — the MVP connector: spreadsheet in/out, including the export→correct→re-apply roundtrip.
silvershop-suite/connector-k3 — a real ERP driver (K3 kassa cloud SOAP).

Writing your own connector for another provider (ERP, POS, PIM, marketplace) is a small, well-defined job — see Writing a connector.

Architecture
Install
Core concepts
- ImportProfile · FieldMapping · Domains · Key strategy · Write mode & missing policy · Staging, idempotency, delta · Runs & audit
Source of truth (field ownership)
Pseudo-target fields
Running
Writing a connector
- The contract · A minimal connector · Registering it · The safety contract · Delta / cursor · Testing
Services & extension points
Security
Requirements

Architecture

  CONNECTOR (yours)                 CORE ENGINE (this module)
  ────────────────                  ─────────────────────────
  ImportSource::fetch() ──yields──► SourceRecord
                                        │  stage
                                        ▼
                                    ImportStagingRecord  (keyed, hashed, cursored)
                                        │  apply
                                        ▼
                                    MappingResolver → FieldValidator → BuyableResolver
                                        │
                                        ▼
                                    Product / Variation / stock / price   +   ImportRun audit

Stage — a connector's ImportSource::fetch() yields SourceRecords, written to ImportStagingRecord rows (keyed, hashed, cursored).
Apply — staged rows are mapped through the profile's FieldMappings and upserted onto Product / Variation / stock / price.

Separating the two phases buys dry-run (stage without apply), delta (only changed rows apply), idempotency (re-applying an unchanged row is a no-op via PayloadHash), and retry (a failed apply leaves the row to retry).

Install

composer require silvershop-suite/import
# plus at least one connector, e.g.
composer require silvershop-suite/import-excel

Then vendor/bin/sake db:build flush=all. The engine does nothing on its own — install a connector and enable its source key (see Registering it).

Core concepts

ImportProfile

An ImportProfile (managed in ImportAdmin → Imports) is the unit of configuration — it binds a source to the domains, mappings, write rules and schedule for one feed. Many profiles may use one source (two suppliers' Excel layouts; K3 stock-every-15-min vs catalogue-nightly).

Field	Meaning
`Title`	Human label.
`ProfileKey`	Stable key for CLI/scheduling (`--profile=<key>`).
`SourceKey`	Which registered source to use (`excel`, `k3`, …).
`SourceLocation`	Source-specific locator (a file path for Excel; unused for env-configured ERPs).
`Domains`	Comma list to sync: `Product,Variation,Stock,Price`.
`WriteMode`	`upsert` · `add-only` · `update-only`.
`MissingPolicy`	`none` · `deactivate` · `delete` — what to do with records absent from a pull.
`KeyStrategy`	Comma list: `sku`, `gtin` (tried in order).
`OwnershipPreset`	`project-default` · `pos-price` · `stock-only` · `full-erp`.
`ScheduleIntervalMins`	Minutes between background syncs (0 = manual only).
`Enabled`	Master on/off.

FieldMapping

Each FieldMapping (child of a profile) maps one source field → one target field, scoped to a domain, with a Transform, optional ValidationRules, and an Ownership override. Target fields are either real $db fields on the buyable (InternalItemID, Title, BasePrice, GTIN, Price) or pseudo-targets (Category, Brand, Quantity, ProductSKU, Option{N}…).

Transforms: trim · number · integer · boolean · uppercase · lowercase · date.

If a connector already yields silvershop target field names (as K3 does), the mapping is identity (InternalItemID → InternalItemID). If it yields source-native names (as Excel does — column headers), the mapping translates them. Either is fine; the mapping is where it's reconciled.

Domains

Product · Variation · Stock · Price · Category (reserved). A profile syncs one or more; each run processes a single domain. Products resolve by SKU, variations by GTIN, stock against variation-then-product.

Key strategy

KeyStrategy is a comma list of how to find the existing record to upsert onto: sku (core's InternalItemID) → gtin (via silvershop-suite/product-identity, guarded). sku,gtin tries SKU first, then GTIN — the usual choice for a mixed catalogue.

Write mode & missing policy

WriteMode — upsert (create + update), add-only (create, never touch existing), update-only (update, never create).
MissingPolicy — for records present in the shop but absent from a pull: none (leave them — the safe default for partial sheets), deactivate (flip AllowPurchase off / zero stock), delete.

Staging, idempotency, delta

Staged rows carry a PayloadHash (a stable hash of the mapped field set). Re-staging an identical record is a no-op; only changed rows apply. A per-source ImportCursor records the high-water mark so a scheduled run fetches only what changed since last time (a manual run is always a full pull).

Runs & audit

Every run writes an append-only ImportRun header (status + counts: created/updated/skipped/failed + summary) and, optionally, per-record ImportRunLines. A dry-run resolves and diffs every record, writes nothing, and still produces the report — the "Check Data" gate. All surfaced in ImportAdmin.

Source of truth (field ownership)

Who wins on a field — the ERP or the CMS — differs per project, so it resolves in layers (most-specific wins): per-FieldMapping → ImportProfile preset → project YAML default → shipped fallback. Presets: pos-price (ERP owns price + stock), stock-only (ERP owns stock only), full-erp (ERP owns all). Shipped fallback is do-no-harm — stock is ERP-authoritative, every other field is import-once (seeded on create, never overwritten). Per-field ownership values: erp-authoritative, import-once, cms-owns, inherit. Project default:

SilverShop\Import\Service\OwnershipResolver:
  default_ownership:
    BasePrice: erp-authoritative
    Title: erp-authoritative

Pseudo-target fields

These target names aren't database fields — the engine routes them to resolvers:

Target	Routed to	Effect
`Category`	`CategoryResolver`	`A > B \| C` → nested paths; primary = `ParentID`, all = memberships; missing created
`Brand`	`BrandResolver`	match/create a brand by `Title`, set `BrandID` (needs `silvershop-suite/brands`)
`Quantity` / `Stock`	`StockWriter`	`setStockLevel()` (needs `silvershop/stock` + a warehouse)
`ProductSKU`	engine	on a Variation record, the parent product's SKU (resolves/creates the parent)
`Option{N}Name/Value`	`AttributeResolver`	variation axes (`AttributeType` + `AttributeValue`), up to `max_axes` (3)

Fixed/identity fields (ID, ClassName, Version, Created, LastEdited) are never writable from a feed, even if mapped.

Running

# one run of a profile (full pull)
sake tasks:SilverShop-Import-Run --profile=<id-or-key>

# preview only (stage + simulate apply, writes nothing)
sake tasks:SilverShop-Import-Run --profile=<id-or-key> --dry-run

# only one domain
sake tasks:SilverShop-Import-Run --profile=<id-or-key> --domain=Stock

# arm the recurring background sync (per ScheduleIntervalMins)
sake tasks:SilverShop-Import-Run --profile=<id-or-key> --schedule

The recurring job runs on symbiote/silverstripe-queuedjobs; process the queue from OS cron:

*/1 * * * * php /path/to/site/vendor/bin/sake tasks:ProcessJobQueue

Writing a connector

A connector is a small module that teaches the engine to read one system. You implement one interface, yield plain value objects, and register the class in YAML. The core handles mapping, keying, validation, upsert, staging, dry-run, scheduling and audit — you only fetch.

The contract

interface ImportSource
{
    public function getKey(): string;        // 'acme' — stable, lower-case
    public function getName(): string;        // 'Acme ERP (REST)' — for CMS/CLI
    public function isConfigured(): bool;      // env/keys present? inert until true
    public function supportsProducts(): bool;  // capability flags: what the SYSTEM can provide
    public function supportsStock(): bool;
    public function supportsPrices(): bool;
    public function fetch(ImportContext $context): iterable; // yield SourceRecord — a generator
}

fetch() receives an ImportContext (->profile, ->domain, ->dryRun, ->since, ->triggeredBy) and yields SourceRecords for the requested domain only:

new SourceRecord(
    string  $externalKey,      // the source's stable id for this row (SKU / EAN / …)
    string  $domain,           // one of SilverShop\Import\Support\Domain
    array   $fields,           // source-native field name => value (pre-mapping)
    ?string $sourceTimestamp = null, // the row's change time, for the delta cursor
    mixed   $raw = null,       // the untouched payload, kept for the audit trail
);

The connector's job ends at producing records — it never touches Product/Variation. The $fields keys are whatever you choose; the profile's FieldMappings translate them to silvershop targets (yield silvershop target names directly for identity mapping, or source-native names and map them — both work).

A minimal connector

namespace Acme\Import;

use RuntimeException;
use SilverShop\Import\Source\ImportContext;
use SilverShop\Import\Source\ImportSource;
use SilverShop\Import\Source\SourceRecord;
use SilverShop\Import\Support\Domain;
use SilverStripe\Core\Environment;
use SilverStripe\Core\Injector\Injectable;

class AcmeImportSource implements ImportSource
{
    use Injectable;

    public function getKey(): string  { return 'acme'; }
    public function getName(): string { return 'Acme ERP (REST)'; }

    public function isConfigured(): bool
    {
        return (string) Environment::getEnv('ACME_API_URL') !== '';
    }

    public function supportsProducts(): bool { return true; }
    public function supportsStock(): bool    { return true; }
    public function supportsPrices(): bool   { return false; }

    public function fetch(ImportContext $context): iterable
    {
        return match ($context->domain) {
            Domain::PRODUCT => $this->fetchProducts($context->since),
            Domain::STOCK   => $this->fetchStock(),
            default         => [],
        };
    }

    private function fetchProducts(?string $since): iterable
    {
        foreach ($this->call('/products', ['since' => $since]) as $row) {
            $sku = trim((string) ($row['sku'] ?? ''));
            if ($sku === '') {
                continue;
            }
            // Yield silvershop target names → the profile's mapping is identity.
            yield new SourceRecord($sku, Domain::PRODUCT, [
                'InternalItemID' => $sku,
                'Title'          => $row['name'] ?? '',
                'BasePrice'      => $row['price'] ?? '',
            ], $row['modified'] ?? null, $row);
        }
    }

    private function fetchStock(): iterable
    {
        foreach ($this->call('/stock') as $row) {
            $ean = trim((string) ($row['ean'] ?? ''));
            if ($ean === '') {
                continue;
            }
            yield new SourceRecord($ean, Domain::STOCK, [
                'GTIN'     => $ean,
                'Quantity' => (int) ($row['qty'] ?? 0),
            ], null, $row);
        }
    }

    /** @return array<int, array<string, mixed>> */
    private function call(string $path, array $query = []): array
    {
        // Your HTTP call. On an unreachable/refused/timeout response: throw a RuntimeException
        // (the engine soft-fails the run and KEEPS existing data). On an empty-but-valid
        // response: return [] (yield nothing) — never signal "everything is gone".
        throw new RuntimeException('Implement me');
    }
}

Registering it

A connector declares its source in YAML (key → Injector service) and the shop enables it:

SilverShop\Import\Registry\ImportSourceRegistry:
  sources:
    acme: '%$Acme\Import\AcmeImportSource'
  enabled_sources:
    - acme

Then create an ImportProfile with SourceKey = acme, pick the domains, and add the field mappings (identity here: InternalItemID → InternalItemID, Title → Title, BasePrice → BasePrice, and Quantity → Quantity for the Stock domain). Done — the engine can now run it, preview it, schedule it and audit it.

The safety contract

Learned from a production catalogue-blanking incident — your fetch() must honour these:

Throw on unreachable. A timeout/refusal/transport error throws a RuntimeException; the engine soft-fails the run and keeps existing data. Never return a partial or empty list on failure.
Empty-but-valid yields nothing. A genuinely empty feed yields zero records — it must never read as "deactivate/delete everything". MissingPolicy only acts after a successful, non-empty pull.
Stream, don't buffer. fetch() is a generator so a 100k-row feed flows row by row.
Fail fast on connect, be patient overall. Short connect timeout (don't hang workers), generous total timeout (bulk feeds are big).

Delta / cursor

Set SourceRecord::$sourceTimestamp to the row's change time. On a scheduled run the engine passes the last high-water mark as ImportContext::$since (null on a manual full pull) — use it to ask the source for only what changed, and the per-source ImportCursor advances automatically. If your source has no usable timestamp (e.g. a flat spreadsheet), leave it null and rely on PayloadHash for idempotency.

Testing a connector

Unit-test the source in isolation by asserting the SourceRecords it yields (mock the transport). To test end-to-end against a real Product, register a TestOnly fake source and run the engine — the pattern this module uses in tests/php/:

Config::modify()->set(ImportSourceRegistry::class, 'sources', ['fake' => FakeImportSource::class]);
$run = ImportEngine::create()->run(new ImportContext($profile, 'Product'));
$this->assertSame(1, (int) $run->NumUpdated);

Gate on phpcs + phpstan (silverstan L5) + phpunit (SQLite), as the suite connectors do.

Services & extension points

All are Injectable — override via Injector or subclass:

Service	Responsibility
`ImportEngine`	Orchestrates stage + apply; safety invariants; chunked memory reset
`MappingResolver`	Source fields → target fields (transform + validation + ownership)
`FieldValidator`	Per-field validation rules (incl. guarded GTIN check-digit)
`BuyableResolver`	Find the existing Product/Variation to upsert onto, by key strategy
`StockWriter`	Guarded write to `silvershop/stock`
`CategoryResolver`	Nested category paths, find/create
`BrandResolver`	Brand by title, find/create (guarded on `silvershop-suite/brands`)
`AttributeResolver`	Variation option axes → attribute types/values
`OwnershipResolver`	Layered field-ownership resolution

Security

No dynamic execution — no eval/unserialize/call_user_func; transforms are a fixed whitelist.
Parameterized ORM throughout; staging payloads are JSON, not PHP serialize.
Per-row isolation — one bad row fails only itself (logged); the run continues.
Fixed-field guard — a feed can't write ID/ClassName/Version/… even if mapped.
Connectors handling credentials should read them from env, never commit them, and never log them.

Requirements

PHP ^8.3 · silverstripe/framework ^6 · silverstripe/cms ^6 · silverstripe/admin ^3
silvershop/core ^6.2
symbiote/silverstripe-queuedjobs ^6

Optional (guarded): silvershop/stock (stock write target), silvershop-suite/product-identity (GTIN keying), silvershop-suite/brands (brand assignment), silvershop-suite/catalog-data, silvershop-suite/ledger.

Credits

Authored by Remy Vaartjes and the SilverShop-Suite Contributors. Generalised from the production viva-import two-phase framework. Licensed BSD-3-Clause.

silvershop-suite / import

Package info

Security

Statistics

README

Contents

Architecture

Install

Core concepts

ImportProfile

FieldMapping

Domains

Key strategy

Write mode & missing policy

Staging, idempotency, delta

Runs & audit

Source of truth (field ownership)

Pseudo-target fields

Running

Writing a connector

The contract

A minimal connector

Registering it

The safety contract

Delta / cursor

Testing a connector

Services & extension points

Security

Requirements

Credits