README

Kite: Make your projects fly

Kite is a build and automation tool inspired by TYPO3.Surf, written in PHP and utilizing PHP for configuration. It's...

easy to use
- Kite ships with several preconfigured tasks, workflows and presets - just use a preset and be done.
- Jobs and workflows are available on the command line and there's also --help available for each of them.
flexible
- The configuration can be completely done by arrays, fiddling your tasks together or by workflow classes that have all the tasks available as methods or both.
- Jobs, workflows and tasks can easily be reused at any point - new jobs can be composed of any of them.
- The variable system provides a JavaScript like variable inheritance: Sub tasks can access variables from parent but can set them on their own as well
- Jobs and workflows can expose variables as command line options and arguments.
- Advanced logic during execution possible by using Symfony Expression Language
node based
- Unlimited number of remote targets possible
- Nodes can be set globally or only for specific (sub) tasks
- Remote tasks operate on all current nodes
safe
- Everything can be --dry-run to preview what happens (yet the tasks to include need to be configured)
- The complete debug output of previous tasks can be viewed with kite log

Appendix

Task and workflow reference

Installation

You can install kite globally (recommended) or per project

Prerequesites

Locally

PHP 5.4+ (for the composer installation, the application can have lower versions)
Linux shell (windows might work but is untested)
SSH/SCP installed (for working on nodes)
git installed (if you use git tasks)
composer installed (if you use composer tasks)

Remote

PHP
SSH access

Global installation

composer global require "netresearch/kite"
~/.composer/vendor/bin/kite -V

Per project installation

cd /var/www/project
composer require "netresearch/kite"
vendor/bin/kite -V

Configuration

Concepts

Tasks
- Smallest, predefined steps
- See the task reference for tasks shipped with kite
Workflows
- Special kind of task that allows to composer it's subtasks in a class
- Top level workflows can expose command line arguments and options
- See the workflow reference for workflows shipped with kite
Jobs
- Outermost kind of task
- Available as commands on command line
- Set of tasks and/or workflows defined in arrays (in arbitrary depth)
- Configurable command line arguments and options
Presets
- Configuration presets (including f.i. common jobs)
Configuration file (typo3conf/Kite.php, app/etc/kite.php, kite.php)
- Defines the jobs; can load and override presets

Variables

The fact that all of the configured tasks are to be ran automated, introduces the need for a variable system that allows you to read from dynamic configurations or change it. Kite provides a basic syntax to access those variables from within strings (all options of tasks, nodes etc.):

Each string inside curly braces inside an option string are evaluated as Symfony Expression Language expressions - f.i.

<?php
$this['messages'] = (object) ['default' => 'Hello world'];
$this['jobs']['echo'] = [
    'description' => 'Output the default message from (\{config["messages"].default\})',
    'task' => [
        'type' => 'output',
        'message' => '{config["messages"].default}'
    ]
];

As you see above, by quoting the braces, you can avoid that the expression is evaluated. Please see the Symfony Expression Language Syntax for help on how to use the expressions.

Variable scopes

The variable scopes are very similar to those in JavaScript. This means that you can access all variables from the parent scope within the current scope unless you have a variable in the current scope that's name is the same. To disambiguate you can use the special variables this and parent.

Task or workflow options are always bound to the scope of the task

This means, that they have to explicitly be set for the task or workflow and can not be read from parent tasks (like jobs or workflows). However sub tasks of those tasks can access those options without prefix when they don't have an option with the same name or with parent prefix otherwise.

Global variables

Additionally to the options from the current and parent tasks there are some global variables available:

job
- The current job object (instance of \Netresearch\Kite\Job)
kite
- Object with some information about the kite package (path and relative dir)
config
- The config array object (as in configuration file)
composer
- Composer service object providing keys packages and rootPackage

Special variables

this
- By using this you can force the variable to be not looked up in the parent scopes
- but only within the current.
parent
- Points to the parent object

Available functions

Kite ships with the following expression language functions:

isset(variable) and empty(variable)
- Behave just like their PHP equivalents. Only available for variable objects, such as
- tasks, nodes, workflows or jobs and their objects (f.i. not for configuration arrays)
get(variableName, variableValue) and set(variableName, variableValue)
- Get or set the variables (f.i. set('job.owner', node.user)
answer(question)
- Let the (command line) user answer a question and return the result
answer(question)
- Let the (command line) user answer a confirmation question and return the result
select(question, options)
- Let the (command line) select from an array of options
any PHP function
- Lets you call PHP functions as you are used to in PHP - e.g. str_replace('\\\\', '/', config['somePath']) (Note the four back slashes which are required to pass a single escaped backslash to Expression Language)

Kite configuration file

You need a file called "Kite.php" to set up config (where to deploy). For TYPO3 projects it should be placed here: typo3conf/Kite.php, for Magento app/etc/kite.php and for all other applications just kite.php. A basic example could be

<?php
// Example for a project without a staging environment

// This loads configuration with common jobs
$this->loadPreset('common');

// This configuration is loaded on execution of deploy or rollout job
$this['stages']['staging']['node'] = array(
    'host' => 'set host here',
    'deployPath' => 'set path on host here',
    'webUrl' => 'set url here',
    'php' => 'php56',
);

// no staging is available
unset($this['stages']['production']);

?>

Jobs

Jobs are to be configured in the key jobs in the configuration. They can contain a single task, an array of tasks or a workflow (always only one of them).

<?php
// Job, running a single task
$this['jobs']['echo'] = [
    'description' => 'Output a message',
    'arguments' => [
        'message' => [
            'type' => 'string',
            'required' => true,
            'label' => 'The message to output'
        ]
    ],
    'task' => [
        'type' => 'output',
        'message' => '{job.message}'
    ]
];

// Job, running a workflow
$this['jobs']['diagnose'] = [
    'description' => 'Show status of packages',
    'workflow' => 'Netresearch\Kite\Workflow\Composer\Diagnose'
    // can written as follows also:
    // 'workflow' => 'composer-diagnose'
];

Nodes

Whenever you set a key named node or nodes on a job, workflow or task it's value will be mapped to an aggregate of node models. Those models have the following default configuration:

<?php
array(
    'user' => '',
    'pass' => '',
    'port' => '',
    'url' => '{(this.user ? this.user ~ "@" : "") ~ this.host}', // SCP/SSH URL
    'sshOptions' => ' -A{this.port ? " -p " ~ this.port : ""}{this.pass ? " -o PubkeyAuthentication=no" : ""}',
    'scpOptions' => '{this.port ? " -P " ~ this.port : ""}{this.pass ? " -o PubkeyAuthentication=no" : ""}',
    'php' => 'php', // PHP executable
    'webRoot' => '{this.deployPath}/current',
    // No default values, required to be set:
    // 'webUrl' => 'http://example.com',
    // 'host' => 'example.com',
    // 'deployPath' => '/var/www'
);

Development packages

The default kite jobs checkout and merge work only on packages which are white listed as development packages. As deploy and rollout utilize those jobs they stick to this behaviour as well.

You can white list packages in three ways: by package name, by git remote url or by package path. When none of them is given all git packages will be used as development packages. You can change the white lists in your Kite configuration file:

<?php
// The following whitelist types are available (evaluated by OR)
// ... for the package names
$this['composer']['whitelistNames'] = 'netresearch/.*';
// ... for the git remote urls
$this['composer']['whitelistRemotes'] = 'git@github.com:netresearch/.*';
// ... for the package paths
$this['composer']['whitelistPaths'] = 'vendor/netresearch/.*';

The default configuration is as follows:

<?php
$this['composer']['whitelistNames'] = $this['composer']['whitelistPaths'] = null;
$this['composer']['whitelistRemotes'] = '{preg_quote(preg_replace("#/[^/]+$#s", "", composer.rootPackage.remote), "#")}/.+';

This means that only packages which share the part before the last occurrence of / with the application remote will be considered as development packages (f.i. if your application git remote is https://github.com/netresearch/kite.git then only packages with a remote like https://github.com/netresearch/* will match).

Deployment configuration

Stages

As you saw in the example in Kite configuration file, there is a top level configuration element named stages. They are set by the common preset and hold configuration only used for each of it's keys (such as staging and production by default). They are evaluated by workflows based on the stage-select workflow, which takes the stage(s) to use from either command line or a select question. After a stage was selected ALL of it's values are set to the corresponding task (such as deploy).

The stages have no special meaning and are not handled in a special way - they only play together with the stage based tasks (deploy and rollout from the common preset and ccr from the typo3 preset) because those are configured so.

Deployment

The deployment workflow deploys your application to exactly one stage (whereas the rollout just runs the deployment workflow for each until the selected stage). Thereby it does the following steps:

Run kite composer diagnose to assert that your application is at a defined state (nothing uncommited, unpushed, unpulled, lock file up to date etc.)

Run composer checkout with the parameters you provided for the stage:

branch - The branch to checkout. In common preset they are configured as follows:

<?php
$this['stages'] = [
    'staging' => [
        'branch' => '{replace("/", "-", composer.rootPackage.name)}-staging',
        'merge' => true,
        'createBranch' => '{config["composer"]["whitelistNames"] || config["composer"]["whitelistRemotes"] || config["composer"]["whitelistPaths"]}'
        // add nodes or node in your config
    ],
    'production' => [
        'branch' => 'master',
        // add nodes or node in your config
    ]
];

merge - Whether to merge the currently checked out branch into the branch to checkout
createBranch - Whether to create the branch if it doesn't exist. This is by default set to true for the staging stage, when Development packages are white listed (which are by default).
rsync - configuration for rsync task invoked (f.i. with excludes option)

Creates a new release from the current release on each node {deployPath}/releases
Rsync the current local state to the new release dir on each node

Symlink shared directories and files (shared means shared between the releases) -the shared directories and files are expected to be at {deployPath}/shared. They can be configured as seen in the typo3 preset:

<?php
$this->merge(
    $this['jobs']['deploy']['task'],
    [
        'shared' => [
            'dirs' => ['fileadmin', 'uploads', 'typo3temp']
        ]
    ]
);

To illustrate the behaviour of the stage configuration here's an example setting the shared directories differently for each stage:

<?php
$this->merge(
    $this['stages'],
    [
        'staging' => [
            'shared' => [
                'dirs' => ['shared_dir_1', 'shared_dir_2'],
                'files' => ['file1', 'file2']
            ]
        ],
        'production' => [
            'shared' => [
                'dirs' => '{config["stages"]["staging"]["shared"]["dirs"]}',
                'file' => 'file'
            ]
        ]
    ]
);

Switch the previous release pointer ({deploypath}/previous) to the current release.
Switch the current release pointer ({deploypath}/current) to the new release.

Invoke the onReady task if any. F.i.:

<?php
$this->merge(
    $this['jobs']['deploy']['task'],
    [
        'onReady' => [
            'type' => 'shell',
            'command' => 'mail',
            'optArg' => ['s' => 'Deployed to {stage}', 'user@example.com']
        ],
    ]
);

And to once again demonstrate that each of the stages can override any option on the deployment workflow:

<?php
$this->merge(
    $this['stages']['production'],
    [
        'onReady' => [
            'type' => 'shell',
            'command' => 'mail',
            'optArg' => ['s' => '[IMPORTANT] Deployed to {stage}', 'user@example.com']
        ],
    ]
);

When you invoke the deployment or rollout jobs with the rollback (--rollback or -r) option, it

switches the next release pointer ({deploypath}/next) to the current release
switches the current release pointer ({deploypath}/current) to the previous release
invokes the onReady task if any.

When you invoke the deployment or rollout jobs with the rollback (--activate or -a) option, it invokes the last three steps of the deployment (switch symlinks, and invoke onReady).

Usage

As stated above, all jobs are available as kite sub commands (kite job-name). You can list the available commands by running

kite [list]

By running

kite help command
#or
kite command --help

you can show help for a specific job/command.

Common commands

kite [help [command]]
- Gives a list of all available commands (jobs) or shows help for the given one
kite --workflow=<workflow-name-or-class>
- Runs a workflow class without requiring it to be inside a job
kite --workflow=<workflow-name-or-class> --help
- Shows the docs (php class doc), arguments and options for a workflow
kite log [-l]
- Shows the last (default), specific (use with caret like ^2 shows the 2nd least log or with a timestamp from kite log -l) or a list of the available log records

Common jobs

update [branch]
- Checks out branch when given
- Runs git pull followed by composer update
kite checkout [--merge] branch
- Goes through all Development packages and checks out the branch there if it’s available
- After checking out the branch on a package it goes through all Development packages requiring it and updates the version constraint to that branch
- When --merge is passed, the currently checked out branch is merged into the branch to checkout
- When -c is passed, the branch will be created in all Development packages when it doesn't exist there (you can use -p option to limit the operation to certain packages)
kite merge [--squash] [--message=”Message”] branch
- Goes through all composer packages and merges the branch into the currently checked out
kite package-foreach [--git] command
- Runs a command for each composer package (optionally only --git packages)
kite cc, kite ccr [stage]
- Clears caches locally (cc) or on all nodes of a specific stage (contained in TYPO3 preset only for now)

Development workflow

The default kite jobs are designed to ease the development of composer based applications which tends to be pretty complicated when it comes to working with development branches across several packages. What kite can do for you, is to manage the branches in the Development packages along with dependencies on them.

In the following you'll find the kite commands for the common workflow that you do your development within feature (eventually derived from topic branches) or bug fix branches, which you will merge into the master as soon as their changes were reviewed and tested.

Ensure application is on master and up to date
```
kite update master
```
Create the feature branch in the package(s) you want
This will create the FEATURE branch in package vendor/branch and all installed dependent packages and make them require it in dev-FEATURE (as your application directly or indirectly requires the package it will be checked out in FEATURE branch as well).
```
kite checkout -c -p vendor/package FEATURE
```

Develop

cd vendor/package
echo '<?php phpinfo(); ?>' > i.php
git add i.php
git commit -m 'Added php info'
git push
cd ../..

(Work on another existing branch)
```
kite checkout ANOTHER_FEATURE
```

Deploy FEATURE branch to staging

kite checkout FEATURE
kite deploy staging

(Work on another existing branch)
```
kite checkout ANOTHER_FEATURE
```
FEATURE was reviewed, merge it into master and delete it
```
kite checkout master
kite merge --delete FEATURE
```
Roll out master (now including FEATURE) over staging to production
```
kite rollout production
```

Deployment jobs

kite deploy [stage]
- Runs the deployment for all nodes on the given or selected stage
kite rollout [stage]
- Runs the deployment for all nodes for each stage until (including) the given stage

Use public key authentication

To prevent you to have to type your password several times during deployment you should set your public key on your server. Usually this is located here: "~/.ssh/authorized_keys".

Trouble shooting

Every task that's executed including it's output will be logged to a log file inside your home directory. This includes f.i. each command ran on the local and remote shells, their output, debug messages and a lot more. Basically it holds the output, you would get by adding -vvv to your kite command.

Just run kite log when a job failed and you want to know exactly what went wrong.

netresearch / kite

Maintainers

Details