lolautruche/sqliimport

eZ Publish legacy extension for importing external content

v1.3.0 2015-11-18 08:18 UTC

Requires

• ezsystems/ezpublish-legacy-installer: *

GPL-2.0

• Jerome Vieilledent

README

Table of contents

• 1   PRESENTATION
  • 1.1   LICENCE
  • 1.2   CREDITS
• 2   USAGE
  • 2.1   Immediate & Scheduled
    • 2.1.1   Immediate
    • 2.1.2   Scheduled
  • 2.2   Runtime Options
  • 2.3   Import Interruption
    • 2.3.1   From admin interface
    • 2.3.2   From CLI
• 3   CLI SCRIPT
• 4   PERFORMANCE SETTINGS
  • 4.1   ViewCaching
  • 4.2   ObjectIndexing
  • 4.3   Content objects update
    • 4.3.1   Datatypes for which a specific diff handler is provided :
• 5   HANDLER DEVELOPMENT
  • 5.1   Simplified content API
    • 5.1.1   Options
      • 5.1.1.1   SQLIContentOptions
      • 5.1.1.2   SQLIContentPublishOptions
  • 5.2   Source Handlers
• 6   DIFF HANDLER DEVELOPMENT

1   PRESENTATION

SQLIImport is an extension allowing to import external content in eZ Publish. It provides a framework for content manipulation and a simple interface for developing import handlers, so as a GUI to administrate your imports in the admin interface Import handlers are classes you need to develop in order to process external data (XML, CSV, ...) and import it into eZ Publish (see Handler Development section).

1.1   LICENCE

This eZ Publish extension is provided as is, in GPL v2 (see LICENCE).

1.2   CREDITS

This extension has been written by Jérôme Vieilledent and sponsored by SQLi.

Oracle adaptation has been made by Simone Tellini.

2   USAGE

You can manage your imports via the admin interface. After installing SQLIImport, a new Import Management tab appears (you'll need to have access to sqliimport module or to simply be administrator to be able to see it from eZ Publish 4.3). Click on it to start managing your imports :

Imports list

2.1   Immediate & Scheduled

There are two types of imports :

• Immediate
• Scheduled

Each import is stored in the database as pending and awaits for the cronjob to process it. Pending imports can be cancelled while the cronjob has not processed it.

2.1.1   Immediate

Immediate imports are one-shot, meaning that they will not repeat in time, contrary to scheduled imports. To add an immediate import, click on Request a new immediate import in the left menu. Choose your import handler and eventually add options (see Runtime Options section below).

2.1.2   Scheduled

Scheduled imports will be launched at chosen start date. They can be one-shot (Frequency = none) or recurring. You can add a label to the scheduled import and deactivate it :

Add scheduled import form

To add a scheduled import, go to Scheduled import(s) by clicking the link in the left menu, and click Add a scheduled import. Choose your import handler and eventually add options (see Runtime Options section below).

2.2   Runtime Options

If your import handler supports Runtime options (see Handler Development section), you can add them from the admin interface. You can only add one option per line with format optionName=optionValue. Options will be passed to the import handler at runtime (in the handler constructor).

2.3   Import Interruption

Running imports are safely interruptable from the admin interface or from the CLI.

2.3.1   From admin interface

From the import list, you can interrupt a running import by clicking the Interrupt link of the import. Please note that you will need to have access to the manageimports policy function for the current import handler.

2.3.2   From CLI

From version 1.2.0, SQLI Import catches SIGTERM and SIGINT signals. This is made possible thanks to PCNTL extension (won't work on Windows). You can thus safely interrupt a running import with kill command :

kill -2 <import_script_pid>
kill -15 <import_script_pid>

Please note that kill -9 (SIGKILL) signal cannot be caught, so always prefer using SIGTERM (kill -15) or SIGINT (kill -2). You can also ask for import interruption by pressing Ctrl+C, which sends a SIGINT signal.

3   CLI SCRIPT

SQLIImport provides both a cronjob and a regular CLI script. The cronjob is used to process imports added from the admin interface (immediate and scheduled). The regular CLI script can be used to trigger a quick one shot import, without having to go into the admin interface.

Usage : php extension/sqliimport/bin/php/sqlidoimport.php [OPTION]...

Options :

4   PERFORMANCE SETTINGS

Several performance settings are set in sqliimport.ini configuration file. For more details, read the inline comments in the INI file.

4.1   ViewCaching

View caching is disabled by default for performance reasons. It's disabled only for the import script. ViewCache is cleared once import has been done, via sqliimport_cleanup cronjob (launched after sqliimport_run)

4.2   ObjectIndexing

Same as for ViewCaching above. Import will be much faster with ObjectIndexing set to disabled. Will just activate site.ini SearchSettings.DelayedIndexing for current import script. Content objects will be indexed once import has been done, via sqliimport_cleanup cronjob.

4.3   Content objects update

If bundled content manipulation framework is used, the system will do comparisons in order to check if it is really necessary to create a new content object version. By default it compares the string representation of each attribute content, but the diff system is extendable. It is thus possible to define new diff handlers for each datatype.

For more information, please read the DIFF HANDLER DEVELOPMENT section.

4.3.1   Datatypes for which a specific diff handler is provided :

• ezimage
• ezbinaryfile

5   HANDLER DEVELOPMENT

To import external content into eZ Publish with SQLIImport, you need to develop a handler that understands the external source (whatever it is) and maps it with your eZ Publish content structure.

Developing an import handler is fairly easy. You just need to create a PHP class that extends SQLIImportAbstractHandler and implements ISQLIImportHandler. You will also need to declare it in an override of sqliimport.ini by creating a dedicated section (please read inline INI comments for further details).

UML diagram for import handlers

Handler method call order :

• __construct() - You'll need to call the parent constructor in it
• initialize() - Main method called to init your handler. Here you may read your external data source
• getProcessLength() - Get the number of iterations needed to complete the process
• getHandlerName()
• getHandlerIdentifier()
• getNextRow() - Must return next row to process or false when import process is finished for the handler
• process() - Called in a loop mastered by getNextRow(). Result of getNextRow is passed to this method
• getProgressionNotes() - Called inside the loop, after process()
• cleanup() Final method called at the end of the handler process

A full working example is provided (SQLIRSSImportHandler). Check it to understand the mechanism.

Note that all configuration set in your INI handler block in sqliimport.ini will be available in your handler in $this->handlerConfArray.

5.1   Simplified content API

A framework is provided to manage eZ Publish content without hassle (please read examples in the API doc) :

<?php
  $cli->notice( 'Creation of a new "comment" object' );
  $options = new SQLIContentOptions( array(
      'class_identifier'      => 'comment',
      'remote_id'             => 'my_ubber_cool_remote_id',
      'language'              => 'fre-FR'
  ) );
  $comment = SQLIContent::create( $options );
  $cli->notice( 'Current version : '.$comment->current_version );
  $comment->fields->subject = 'Mon super sujet';
  $comment->fields->author = 'Moi !';
  $comment->fields->message = 'Le commentaire de la mort';

  $comment->addTranslation( 'eng-US' );
  $comment->fields['eng-US']->subject = 'My great subject';
  $comment->fields['eng-US']->author = 'Batman';
  $comment->fields['eng-US']->message = 'Death comment';

  $comment->addLocation( SQLILocation::fromNodeID( 2 ) );
  $comment->addLocation( SQLILocation::fromNodeID( 43 ) );

  $publisher = SQLIContentPublisher::getInstance();
  $publisher->publish( $comment );

  $cli->notice( 'Current version : '.$comment->current_version );

  // Loop against locations
  foreach( $comment->locations as $nodeID => $location )
  {
      // Regular node attributes are available as virtual properties
      $cli->notice( $nodeID.' => '.$location->path_string.
                    ' ('.$comment->locations[$nodeID]->path_identification_string.')' );
  }

  // Now free memory.
  // unset() on SQLIContent triggers eZContentObject::clearCache()
  // and eZContentObject::resetDataMap()
  unset( $comment );
?>

SQLIContent framework relies on string representation of content attributes. It makes use of datatypes fromString() / toString() methods, implemented in every kernel datatypes since eZ Publish 3.9. So if you use custom datatypes, make sure they implement these methods for better result. If they are not present, the framework will use data_text instead.

For more information about string representation of kernel datatypes, please read fromString.txt appendix.

Note about HTML content :

A shorthand method is available to handle HTML content import. It is available in Import Handlers and outside import handlers :

<?php
// Code below is available in an import handler
$content->fields->intro = $this->getRichContent( $myHTMLContent );

// Code below is available everywhere
$content->fields->intro = SQLIContentUtils::getRichContent( $myHTMLContent );
?>

For more examples, please check scripts located in the stubs/ directory.

You can also have a look at the UML diagram below.

UML diagram for SQLIContent framework

5.1.1   Options

When manipulating SQLIContent framework, you can tweak several options. All of these options are packaged into objects, usually passed to a constructor/singleton (SQLIContent, SQLIContentPublisher).

5.1.1.1   SQLIContentOptions

These options are used when instantiating an SQLIContent object :

• remote_id => RemoteID of your content. Use it to retrieve/update your content later. Default is NULL.
• section_id => SectionID of your content. Default is 0 (let the eZ Publish decide regarding tree inheritance).
• creator_id => CreatorID of your content. Default is false (currently logged in user).
• language => Content language. Default is your default language (cf eZContentObject::defaultLanguage().
• class_identifier => Class identifier of your content. Default is NULL. Mandatory

5.1.1.2   SQLIContentPublishOptions

These options are used when instantiating an SQLIContentPublisher object. As the publisher is a singleton object, you can alter its options with its setOptions() method.

• parent_node_id => DEPRECATED, use content locations instead. Defines ParentNodeID for content.
• modification_check => If true, makes the publisher to do check if at least one attribute has been modified before publishing. If not, the content is not published. Default is true.
• update_null_field => If true, will update all attributes in DB, even if data is not set (null), and set it to null. Default is false.

5.2   Source Handlers

2 source handlers are provided :

• SQLIXMLParser - Catches parse errors and fetches XML string. Transforms PHP errors into exceptions. Works with DOM or SimpleXML (example in stubs/xml.php).
• SQLICSVDoc - Set of classes to manage CSV structures as easily as with SimpleXML (example in stubs/csv.php).

UML diagram for CSVDoc classes

6   DIFF HANDLER DEVELOPMENT

When updating a content, SQLIContentPublisher only publishes really modified content by default. It makes a diff between already stored content and new content. This is done via diff handlers.

3 diff handlers are provided by default :

• SQLIDefaultDiffHandler - will basically compare attributes string representation
• SQLIImageDiffHandler
• SQLIBinaryFileDiffHandler

You can develop your own diff handler for your datatypes by creating a class implementing ISQLIDiffHandler interface. Only one static method is needed : contentIsModified(). Please read interface PHPDoc for further information. You can also check the code of provided handlers for examples.

Page ###Page### / ###Total###