README

Structured Data Toolkit
_{JSON-LD and schema.org for PHP}

This library provides several tools to work with JSON-LD and schema.org in PHP. It includes:

an implementation of the W3C JSON-LD algorithms described in the JSON-LD 1.1 Processing Algorithms and API Recommendation, published on July 16, 2020;
a schema.org validator;
a Google validator, which tells you whether your structured data is eligible for Google Rich Results;
an extractor for JSON-LD, microdata and RDFa embedded in HTML documents.

Installation

Install the library with Composer:

composer require jolicode/structured-data

Remote @context resolution is off by default, and the default setup needs no extra dependency. If you opt in (see Loading remote contexts), install an HTTP client as well:

composer require symfony/http-client

Dependencies

This library requires:

PHP >= 8.4, with the dom, json and libxml extensions
(optional, only for remote @context resolution) an implementation of symfony/http-client-contracts, such as symfony/http-client
(optional) the PHP task runner Castor, used for the tooling and the CLI interface. The development tooling additionally needs the ZipArchive PHP extension to download the W3C test suites.

Validating a JSON-LD document

To validate a JSON-LD document, use the JoliCode\StructuredData\Validator class.

Accepted inputs

audit() takes the document itself, as a string - never a URL, never a file path.

This library deliberately does not guess what a string is, and never fetches anything on your behalf. Guessing is a security hazard: an application that forwards user input to a validator would silently offer an attacker a way to reach its internal network (http://127.0.0.1:9200/, cloud metadata endpoints), or to read local files through a path or a stream wrapper (/var/www/.env, file://, phar://).

Whether a document may be fetched, from where, and under which restrictions is a decision only your application can make. So your application is the one that fetches:

// From a local file - the path comes from you, not from a user
$document = file_get_contents('/path/to/document.html');

// From the network - your HTTP client, your allow-list, your timeouts
$document = $httpClient->request('GET', $trustedUrl)->getContent();

$audit = $validator->audit($document);

The same rule applies to the @context URLs found inside a document: see Loading remote contexts.

The validator accepts the following data formats:

JSON-LD
microdata
RDFa (schema.org-style RDFa only)

Using the validator

Basic usage

The validator exposes a single validation method: audit(). It returns a JoliCode\StructuredData\Audit\Audit object holding the validation result.

To quickly check the result, use isValid() or isFullyValid():

isValid() returns true if no errors are detected
isFullyValid() returns true if no errors, no warnings, and no malformed (hence unusable) data structures were detected

Keep in mind that a schema.org type is considered valid even with warnings!

To access the messages themselves, use the getDiagnostic() method, which will return an array of diagnostic messages (errors and warnings).

A typical usage example looks like this:

use JoliCode\StructuredData\Validator;

$validator = new Validator();

$document = file_get_contents('/path/to/a-page.html');
$audit = $validator->audit($document);

if ($audit->isValid()) {
    echo 'The document contains valid structured data!';
} else {
    echo 'The provided document contains invalid schema.org data!';

    // Returns an array of string diagnostic messages
    $diagnostic = $audit->getDiagnostic();

    foreach ($diagnostic as $message) {
        // Messages look like this:
        // [Google warning] DataFeed.dataFeedElement.workExample: Missing recommended property: "sameAs" for the type "Book"
        // [Google error] DataFeed.dataFeedElement.workExample.potentialAction.expectsAcceptanceOf: Missing required property: "price" for the type "Offer" when "category" is "purchase" or "rental".
        echo $message;
    }
}

If you are only interested in the results of a single validator, call setValidator() before auditing:

use JoliCode\StructuredData\Validator;
use JoliCode\StructuredData\Vocabularies\Validators\Google\GoogleValidator;

$validator = new Validator();
$validator->setValidator(GoogleValidator::class);

$validator->audit($document);

Terms hosted under pending.schema.org are still under development and may change or be removed. Using them is legitimate, so the validator accepts them silently by default; opt in to $reportPendingVocabularyUsage to get a warning for each of these usages instead:

$audit = $validator->audit($document, reportPendingVocabularyUsage: true);

Advanced usage

The getDiagnostic() method accepts an optional parameter: a JoliCode\StructuredData\Audit\AuditOptions object, which lets you filter or group the results, or change the return format. See the PHPDoc on JoliCode\StructuredData\Audit\AuditOptions for more details.

Finally, if you want to access the full parsed PHP tree, use getTypes() and inspect the underlying MappedType objects directly. These are low-level objects, but they are the most detailed information you can get, and they preserve the type hierarchy of the document. For an idea of what you can do with these objects, look at the output of the castor validate command.

Command line interface

A command is available to quickly validate a JSON-LD document from the command line or in CI: castor check. Use castor validate to get a nicely parsed and colored full audit (it can be pretty verbose!). Both return a meaningful exit code, for use in scripts and CI.

castor check <file-or-url>
castor validate <file-or-url>

Unlike the Validator::audit() API, these CLI commands do accept a file path or a URL, and will read or fetch it for you. That is safe here precisely because the argument comes from the operator running the command, not from a document being processed - exactly the distinction drawn in Accepted inputs.

You can validate using a specific validator:

castor validate <file-or-url> google

castor validate <file-or-url> schema-org

Sample result of the validate command:

Using the JSON-LD algorithms

The currently available algorithms are:

Conformance

Each algorithm is validated against the official W3C test suites (json-ld-api and json-ld-framing), pinned to a known-good upstream commit and re-run weekly against main to surface drift.

The full suites pass, covering expansion, compaction, flattening and framing. The only skipped fixtures are a handful that target JSON-LD 1.0-specific behavior (declared specVersion: json-ld-1.0 in the upstream manifest), which this library does not implement; each skip is documented in the corresponding test. Serialization to and from RDF (toRdf / fromRdf) is out of scope and not implemented.

To use them, create an instance of JoliCode\StructuredData\JsonLd\Algorithms\Expand\Expander, JoliCode\StructuredData\JsonLd\Algorithms\Flatten\Flattener, JoliCode\StructuredData\JsonLd\Algorithms\Compact\Compactor or JoliCode\StructuredData\JsonLd\Algorithms\Frame\Framer, and pass it the JSON-LD document you want to convert.

For instance, expanding a JSON-LD document looks like this:

use JoliCode\StructuredData\JsonLd\Algorithms\Expand\Expander;

$jsonString = '{
  "@context": "https://schema.org",
  "@type": "Person",
  "name": "John Doe"
}';

$expander = new Expander();
$result = $expander->expand($jsonString);

The result will be a JSON string containing the expanded JSON-LD document:

[
  {
    "@type": [
      "http://schema.org/Person"
    ],
    "http://schema.org/name": [
      {
        "@value": "John Doe"
      }
    ]
  }
]

If you want a PHP value (an array or a stdClass) instead of a JSON string, set the encodeResult parameter to false when calling expand(). You can also pass a ProcessorOptions object holding the JSON-LD options if you want to modify the default behavior of the algorithms:

use JoliCode\StructuredData\JsonLd\Algorithms\Expand\Expander;
use JoliCode\StructuredData\JsonLd\Algorithms\JsonLd\ProcessorOptions;

$jsonString = '{
  "@context": "https://schema.org",
  "@type": "Person",
  "name": "John Doe"
}';

$options = new ProcessorOptions(
  ordered: true,
  frameExpansion: true,
);

$expander = new Expander();
$result = $expander->expand($jsonString, options: $options, encodeResult: false);

Loading remote contexts

By default, nothing goes out

A JSON-LD document may point its @context at a URL, and the specification requires that URL to be resolved before the document can be expanded. This library resolves https://schema.org (and its http and trailing-slash variants) from the vocabulary files it ships with, so the most common case by far is covered without a single outbound request.

Every other remote context is refused. Validator::audit() and the four algorithms issue no network request and read no file unless you say otherwise, and a refused context raises the error the specification mandates:

loading remote context failed

Why unbounded resolution is dangerous

The @context URL comes from the document being processed. As soon as that document is not fully under your control, the URL is attacker-controlled, and a loader that resolves anything hands them:

Request forgery. http://127.0.0.1:9200/, http://169.254.169.254/latest/meta-data/, or any host on your internal network, reachable from your server.
Network mapping. Even without seeing the responses, the difference between a refusal, a timeout, and a success tells them which internal ports are open.
Exfiltration. If the response body of a failed fetch ever finds its way back into an error message. This is why the message above is opaque: it discloses neither the body, nor the status code, nor the URL that was tried.
Denial of service. Through a response that never ends or never arrives, or through a chain of contexts that each pull more contexts (@import, alternate locations, Link rel="…json-ld#context" headers).
Local file reads. If a non-http scheme is allowed to reach the PHP stream wrappers: file:///var/www/.env, or phar://, which deserializes archive metadata on a mere stat call.

Widening the policy, safely

If your documents legitimately reference contexts you trust, allow those hosts, and nothing else:

use JoliCode\StructuredData\JsonLd\Algorithms\Http\HttpDocumentLoader;
use JoliCode\StructuredData\JsonLd\Algorithms\Http\RemoteContextPolicy;
use JoliCode\StructuredData\Validator;
use Symfony\Component\HttpClient\HttpClient;
use Symfony\Component\HttpClient\NoPrivateNetworkHttpClient;

// 1. Which hosts do you trust? An explicit list, over https only.
$policy = RemoteContextPolicy::allowHosts('schema.org', 'www.w3.org', 'json-ld.org')
    ->withTimeouts(timeout: 2.0, maxDuration: 5.0)
    ->withMaxResponseBytes(512 * 1024)
    ->withMaxRedirects(3);

// 2. A second barrier, at the transport level: no private, loopback or link-local
//    address, even if a hostile DNS answer points an allowed host at 169.254.169.254.
$httpClient = new NoPrivateNetworkHttpClient(HttpClient::create());

// 3. A single injection point covers the whole chain.
$validator = new Validator(documentLoader: new HttpDocumentLoader($policy, $httpClient));
$audit = $validator->audit($document);

The same constructor argument is available on Expander, Compactor, Flattener and Framer:

$expander = new Expander(documentLoader: new HttpDocumentLoader($policy, $httpClient));

Host matching is exact, so allowing schema.org does not allow evil.schema.org.example. A URL carrying userinfo (https://user:pass@schema.org/) or a non-default port (https://schema.org:8080/) is refused. Only http and https may ever be allowed, and http requires an explicit withSchemes('http', 'https'). The policy is re-checked on every hop: the URL you asked for, each intermediate redirect, each alternate location, each Link header, and the URL a response was ultimately served from.

The @context URL is not the only document-controlled value that can reach the loader: Expander::expand() (and, through it, Validator::audit()) also accepts a bare IRI as its input and will resolve it. That path is bound by the very same policy, so the default deny-all loader refuses it too - but keep it in mind when you widen the allow-list.

Writing your own loader

Implement JoliCode\StructuredData\JsonLd\Algorithms\Http\DocumentLoaderInterface to resolve contexts your own way, for instance from a local mirror or a PSR-6 cache:

interface DocumentLoaderInterface
{
    public function load(string $url): \stdClass;

    public function getCacheNamespace(): string;
}

Processed contexts are cached for the lifetime of the process, and getCacheNamespace() partitions that cache. Return a value that identifies what your loader is willing to resolve, so that a context obtained under a permissive strategy can never be served to a restrictive one. Signal every failure with new ContextProcessingException('loading remote context failed'), and never put anything from the remote response in that message.

Checklist

List the allowed hosts explicitly, and keep the list short.
Stay on https unless a context you genuinely must load is only available over http.
Wrap your client in NoPrivateNetworkHttpClient.
Set a timeout, a max duration, a response size cap and a redirect cap.
Never return the body of a remote response to your users.

Command line interface

Commands are also available to use the algorithms from the CLI:

castor json-ld:expand <file>
castor json-ld:flatten <file>
castor json-ld:compact <file> <context-file>
castor json-ld:frame <file> <frame-file>

Each prints its output to the console.

Testing and QA commands

All the tasks are defined in the .castor directory, one file per namespace.

The following commands are available to run the QA checks:

Command	Description
`castor qa:install`	Installs the QA tooling
`castor qa:update`	Updates the QA tooling
`castor qa:cs`	Fixes coding standards
`castor qa:phpstan`	Runs PHPStan
`castor qa:all`	Runs all QA tasks

The following commands are available to run the tests:

Command	Description
`castor qa:phpunit:prepare`	Downloads the W3C tests suite
`castor qa:phpunit:run`	Runs PHPUnit
`castor qa:phpunit:coverage`	Runs PHPUnit with code coverage (requires the pcov or xdebug extension)
`castor qa:infection`	Runs Infection mutation testing on the validator and mapper layers (requires the pcov or xdebug extension)

The W3C test suite is pinned to a known-good upstream commit (see W3C_TEST_SUITE_REF in .castor/qa.php). To re-download it, or to test against the upstream main branch:

castor qa:phpunit:prepare --force
castor qa:phpunit:prepare --force --ref main

Additional commands are available to run the benchmarks:

Command	Description
`castor qa:bench:all`	Runs all the benchmarks
`castor qa:bench:algorithms`	Runs the JSON-LD manipulation algorithms benchmark
`castor qa:bench:validators`	Runs the validators benchmark
`castor qa:bench:validators -d`	Runs the detailed and slow validators benchmark

Vocabulary generation commands

The validation classes shipped in src/Vocabularies/Generated are generated from the vocabulary definitions. These commands refresh them:

Command	Description
`castor schema-org:update-version`	Bump the schema.org version to the latest release published on GitHub
`castor schema-org:download`	Download the schema.org types definition file
`castor schema-org:generate`	Generate the schema.org validation classes
`castor schema-org:download-examples`	Refresh the schema.org examples used by the tests
`castor google:download`	Crawl the Google structured-data documentation
`castor google:generate`	Generate the Google validation classes
`castor google:check`	Check the Google documentation coverage against the curated manifest

The full procedures are described in the schema.org and Google upgrade guidelines.

Contributing

See the CONTRIBUTING.md file for more information.

Working on the library

Clone the repository, then install the library's dependencies and the QA tooling:

composer install   # the library's own dependencies (Composer, not Castor)
castor qa:install  # the QA tooling: php-cs-fixer, phpstan, phpunit, phpbench, infection

Upgrading schema.org

Upgrades are driven by the official schema.org release definition file. The complete upgrade process is documented in: resources/schema.org/UPGRADE_GUIDELINES.md

When upgrading, always review the schema.org release notes and check that the changes to the schema-org-baseline.json test fixture reflect real schema.org changes.

Upgrading Google

The Google validator tracks the Google structured-data documentation, which evolves continuously. The complete upgrade process is documented in: resources/google/UPGRADE_GUIDELINES.md

License

This library is released under the MIT License. See the bundled LICENSE file for details.

jolicode / structured-data

Maintainers

Package info

Fund package maintenance!

Statistics

Security

README

Structured Data Toolkit
_{JSON-LD and schema.org for PHP}

JSON-LD and schema.org for PHP

Installation

Dependencies

Validating a JSON-LD document

Accepted inputs

Using the validator

Basic usage

Advanced usage

Command line interface

Using the JSON-LD algorithms

Conformance

Loading remote contexts

By default, nothing goes out

Why unbounded resolution is dangerous

Widening the policy, safely

Writing your own loader

Checklist

Command line interface

Testing and QA commands

Vocabulary generation commands

Contributing

Working on the library

Upgrading schema.org

Upgrading Google

License

jolicode / structured-data

Maintainers

Package info

Fund package maintenance!

Statistics

Security

README

Structured Data Toolkit JSON-LD and schema.org for PHP

JSON-LD and schema.org for PHP

Installation

Dependencies

Validating a JSON-LD document

Accepted inputs

Using the validator

Basic usage

Advanced usage

Command line interface

Using the JSON-LD algorithms

Conformance

Loading remote contexts

By default, nothing goes out

Why unbounded resolution is dangerous

Widening the policy, safely

Writing your own loader

Checklist

Command line interface

Testing and QA commands

Vocabulary generation commands

Contributing

Working on the library

Upgrading schema.org

Upgrading Google

License

Structured Data Toolkit
_{JSON-LD and schema.org for PHP}