salsadigitalauorg/merlin-framework

There is no license information available for the latest version (1.1.0) of this package.

Open source migration framework (markup to structured data)

1.1.0 2022-10-28 04:36 UTC

Requires

• chuyskywalker/rolling-curl: ^3.1
• consolidation/comments: ^1.0
• dompdf/dompdf: ^2.0.0
• league/uri: ^6.0
• masterminds/html5: ^2.5
• mustangostang/spyc: ^0.6.2
• myclabs/deep-copy: ^1.9
• php-curl-class/php-curl-class: ^8.6
• ramsey/uuid: ^3.8
• samchristy/piechart: ^2.0
• spatie/browsershot: ^3.32
• spatie/crawler: ^4.4
• symfony/console: ~4.4.0
• symfony/css-selector: ~4.4.0
• symfony/dom-crawler: ~4.4.0
• symfony/yaml: ~4.4.0
• twig/twig: ^2.0

Requires (Dev)

• phpunit/phpunit: ^7.5
• squizlabs/php_codesniffer: ^3.4

Unknown License

• Stuart Rowlands
• Steve Worley
• Nick Georgiou
• Andrew Rowlands

• migration
• merlin

README

Welcome to Merlin a simple tool to aid content migration from an arbitrary source to a structured format ready for consumption by another system.

Detailed documentation can be found https://salsadigitalauorg.github.io/merlin-framework/.

Requirements

• PHP > 7.2
• Composer (optional)

Installing

The Merlin framework is expected to be used as a standalone executable, this can be localised to your project or installed globally and added to your path. To download, visit the release page and download the latest bundled .phar executable.

curl -s https://github.com/salsadigitalauorg/merlin-framework/releases \
| grep "merlin-framework" \
| cut -d : -f 2,3 \
| tr -d \" \
| wget -qi -

Composer dependency

Merlin can be installed as a composer dependency as well, this changes how the application is excuted for your project.

Add the repository

"repositories": [
  {
    "type": "vcs",
    "url": "https://github.com/salsadigitalauorg/merlin-framework"
  }
]

Add the dependency

composer require salsadigitalauorg/merlin-framework

Usage

There are two primary commands: crawl and generate.

• crawl will run crawl a domain and find URLs on a domain for migration. Read the crawler docs and check the example for more information.
• generate will generate structured output based on mapping configuration. Read the migration docs and check the example for more information.

To run the framework you need to specify a command (e.g crawl or generate), a configuration yaml file, and a path to the output, e.g:

merlin crawl -c <path/to/crawler-config.yml> -o <path/to/output>
merlin generate -c <path/to/migrate-config.yml> -o <path/to/output>

Configuration files

The configuration file should be treated as a schema file, this contains the paths, domains and mapping information to transform a HTML representation of content into structured JSON.

Example configuration files can be found in the examples.

Testing

The automated testing suite tests standard configuration values against representative HTML structure to make sure that the tool can correctly build the JSON structure.

Running the tests

./vendor/bin/phpunit

Support

We encourage you to file issues with the github issue queue.

License

• GPL3.0