mkjpryor/transform

Library for on-the-fly source transformation in PHP

0.1 2013-09-11 12:15 UTC

Requires

• php: >=5.3.0
• doctrine/cache: *

Requires (Dev)

• phpunit/phpunit: *

MIT

• Matt Pryor

• source
• library
• transform
• transformation

README

Transform is an experimental PHP library for performing source-to-source transformations on loaded files.

Source-to-source transformations are useful for a variety of purposes. For example:

• There are situations in which the lack of metaprogramming capabilities in PHP restrict what can be done using the language. The use of source-to-source transformations can mitigate this.
• Source-to-source transformations could be used to transform source code written in a completely different language into PHP as they are loaded. However, note that transformations can only be applied to a single file at once.

Concepts

At the heart of Transform are source transformers. A source transformer takes an input string, processes it and returns an output string. It is possible to pipeline several transformers so that the output from one becomes the input for the next.

Internally, transformers can use whatever technique they like to transform the code as long as the input and output are strings. When combined with PHP-Token-Reflection or PHP-Parser, this becomes quite powerful.

It is possible to cache the results of transformations so that they are only re-applied if the original source changes. This means that the expense of applying transformations is not too much of an issue.

Loading of the transformed code into PHP is handled by a loader object, which is given a source transformer to apply to the contents of the files it loads. The loader object is designed to function as a class autoloader, and also provides functionality to load individual files (as a replacement for include or require) where autoloading is not appropriate (e.g. a file containing function definitions).

Installation

Transform is installed via composer:

#!json

{
    "require" : {
        "mkjpryor/transform" : "0.1"
    }
}

Source Transformers

A source transformer is any object that implements the Transform\Transformer\SourceTransformer interface:

<?php

namespace Transform\Transformer;


interface SourceTransformer {
    /**
     * Apply the source transformation to the given source and return the
     * transformed source
     * 
     * Metadata can be injected by source transformers, but is guaranteed to have
     * the key 'originalFile' containing the fully qualified path to the original
     * source file
     * ArrayObject is used rather than a plain array to get pass-by-reference
     * semantics without worrying about passing by reference...
     * 
     * @param string $source
     * @param \ArrayObject $metadata
     * @return string
     */
    public function apply($source, \ArrayObject $metadata);
}

Source transformers are pipelined using Transform\Transformer\Pipeline, which is itself a transformation:

<?php

$transformer = new \Transform\Transformer\Pipeline([
    new SomeSourceTransformer(), new AnotherSourceTransformer()
]);

// Pass $transformer to a loader object...

In this case, the output from SomeSourceTransformer::apply is used as the input for AnotherSourceTransformer::apply, and the output from AnotherSourceTransformer::apply becomes the output of the pipeline.

The only source transformer provided by Transform that performs any code modification is Transform\Transformer\MagicConstantTransformer. MagicConstantTransformer takes any valid PHP code and replaces the magic constants __FILE__ and __DIR__ with strings appropriate for the original file that the code came from. This transformation is not applied by default - to apply it to loaded code, it must be configured like any other transformer.

The Transforming Loader

Source transformations are applied to code as it is loaded by Transform\ClassLoader\TransformingLoader instances.

Including individual files

Individual files can have transformations applied as they are loaded using TransformingLoader::includeFile. This should be used in place of include or require if transformations are required:

<?php

// Set $transformer to the required transformer (e.g. a pipeline of other transformers)

$loader = new \Transform\ClassLoader\TransformingLoader($transformer);

$loader->includeFile('/path/to/file/to/include.php');

This will apply the given transformation to the given file and evaluate the result as PHP code. Note that if the transformed code contains classes/functions etc. that are already defined, this will result in an error.

Autoloading classes (and traits/interfaces)

The TransformingLoader extends the Composer autoloader to provide its autoloading capabilities. The loader must be told which classes it is responsible for loading and transforming and then registered as an autoloader:

<?php

// Set $transformer to the required transformer (e.g. a pipeline of other transformers)

$loader = new \Transform\ClassLoader\TransformingLoader($transformer);

// Tell the loader to apply transformations to classes in a particular namespace
$loader->add('Some\\Namespace', '/path/to/package/src');
// Or using PEAR naming conventions
$loader->add('Some_Namespace_', '/path/to/package/src');
// Or using a class map
$loader->addClassMap([
    'Some\\Namespace\\SomeClass' => '/path/to/class/file.php',
    'Some\\Other\\Namespace\\OtherClass' => '/path/to/other/class/file.php'
]);

// Register the autoloader - it is important to set the $prepend argument to true
// so that it gets executed before other autoloaders
$loader->register(true);

// The autoloader will then be triggered when a class it knows about is used
$obj = new \Some\Other\Namespace\OtherClass();

Obviously, the transformed code must still contain the class definition, otherwise PHP will issue an error.

Using TransformingLoader with Composer

If you are using Composer to manage your project's dependencies, some effort must be made to get it to play nice with TransformingLoader.

Suppose that we have an existing project that doesn't currently use any transformations that we want to apply transformations to. We are currently using Composer to autoload classes for our dependencies and our project, as well as including a file containing some functions.

#!javascript

// composer.json

{
    "require" : {
        "some/component" : "*",
        "some/other-component" : "*"
    },

    "autoload" : {
        "files" : [ "src/functions/file_with_functions.php" ],
        "psr-0" : {
            "MyNamespace" : "src/"
        }
    }
}

In order to apply transformations to loaded files, we must instead tell Composer to include a bootstrap file where we configure and register a suitable TransformingLoader:

#!javascript

// composer.json

{
    "require" : {
        "some/component" : "*",
        "some/other-component" : "*"
    },

    "autoload" : {
        "files" : [ "src/bootstrap.php" ]
    }
}

What src/bootstrap.php looks like depends on what we want to achieve.

Applying transformations to project files only (i.e. not dependencies)

To apply transformations to our project's files only, we just need to create a TransformingLoader, add our project's namespace, register it as an autoloader and include the functions file:

<?php

/**
 * src/bootstrap.php
 */

// Set $transformer to the required transformer (e.g. a pipeline of other transformers)

// Create a transforming loader
$loader = new \Transform\Transformer\TransformingLoader($transformer);

// Add our project's namespace (we use __DIR__ since this file sits in src
$loader->add("MyNamespace", __DIR__);

// Register the transforming loader as an autoloader
$loader->register(true);

// Include our functions file
$loader->includeFile(__DIR__ . "/functions/file_with_functions.php");

Applying transformations to all classes loaded by Composer (i.e. completely replace the Composer autoloader)

It is also possible to completely replace the Composer autoloader for all future class loads (unfortunately, there is nothing we can do about classes that have already been loaded or files that have already been included...):

<?php

/**
 * src/bootstrap.php
 */

// Get the Composer autoloader responsible for loading dependencies
$composerLoader = require __DIR__ . '/../vendor/autoload.php';

// Set $transformer to the required transformer (e.g. a pipeline of other transformers)

// Create a transforming loader
$loader = new \Transform\Transformer\TransformingLoader($transformer);

// Copy the configuration of the Composer autoloader to our transforming loader
foreach( $composerLoader->getPrefixes() as $prefix => $path ) {
    $loader->add($prefix, $path);
}
$loader->addClassMap($composerLoader->getClassMap());
$loader->setUseIncludePath($composerLoader->getUseIncludePath());

// Add our project's namespace (as above)
$loader->add("MyNamespace", __DIR__);

// Unregister the Composer autoloader and register the transforming loader
$composerLoader->unregister();
$loader->register(true);

// Include our functions file
$loader->includeFile(__DIR__ . "/functions/file_with_functions.php");

Caching for increased performance

Since some transformations will likely be expensive, it is possible to cache the results of transformations using Transform\Tranformer\CachingTransformer. The CachingTransformer implements Transform\Transformer\SourceTransformer, and uses the decorator pattern to wrap an existing transformer and caches the results in a Doctrine Cache:

<?php

// Set $transformer to the required transformer (e.g. a pipeline of other transformers)

// Create a cache
$cache = new \Doctrine\Common\Cache\FilesystemCache('/path/to/cache/dir');

// Wrap $transformer in a CachingTransformer
$transformer = new \Transform\Transformer\CachingTransformer($transformer, $cache);

// Pass $transformer on to a transforming loader

The advantage of using the Doctrine Cache component is that it enables the use of many different caches depending on the situation. During development, \Doctrine\Common\Cache\ArrayCache would most likely be used (i.e. transformations will be only be cached while the script is running). In production, you might use \Doctrine\Common\Cache\FilesystemCache to persist the results of transformations. If you want to share cached transformations across many machines, the Doctrine Cache component also provides implementations for many distributed caches/databases (e.g. Memcached, Redis, Riak).

Warning: Cache size

The CachingTransformer works by taking a hash of the source it is given and using that hash as a cache key (returning the cached version if the key exists and applying the wrapped transformer and caching the result if the key does not exist).

Because of this, the CachingTransformer has no concept of the previous source and hence doesn't remove old keys from the cache, which can lead to the cache growing large if the pre-transformed source changes frequently. Unfortunately, the only solution to this is to purge the cache and start again.

On the flip side, since the cache uses a hash of the source it means that the transformation is only re-calculated if the source actually changes, not just if the modified time of the source file is updated (e.g. a change being made and then reverted).