netresearch/kite

Yet another build and deployment tool - inspired by TYPO3.Surf

Maintainers

Details

github.com/netresearch/kite

Homepage

Source

Issues

Installs: 335

Dependents: 0

Suggesters: 0

Security: 0

Stars: 5

Watchers: 12

Forks: 0

Open Issues: 0

1.8.0 2021-06-01 14:08 UTC

Requires

MIT 13118999014278812e0a1f6db0d13e06bba29db5

  • Christian Opitz <christian.opitz.woop@netresearch.de>

This package is auto-updated.

Last update: 2024-10-29 04:31:08 UTC

README

Kite

Kite: Make your projects fly

http://img.shields.io/travis/netresearch/kite.svg?style=flat-square https://img.shields.io/packagist/v/netresearch/kite.svg?style=flat-square https://img.shields.io/scrutinizer/g/netresearch/kite.svg?style=flat-square

Kite is a build and automation tool inspired by TYPO3.Surf, written in PHP and utilizing PHP for configuration. It's...

  • easy to use
    • Kite ships with several preconfigured tasks, workflows and presets - just use a preset and be done.
    • Jobs and workflows are available on the command line and there's also --help available for each of them.
  • flexible
    • The configuration can be completely done by arrays, fiddling your tasks together or by workflow classes that have all the tasks available as methods or both.
    • Jobs, workflows and tasks can easily be reused at any point - new jobs can be composed of any of them.
    • The variable system provides a JavaScript like variable inheritance: Sub tasks can access variables from parent but can set them on their own as well
    • Jobs and workflows can expose variables as command line options and arguments.
    • Advanced logic during execution possible by using Symfony Expression Language
  • node based
    • Unlimited number of remote targets possible
    • Nodes can be set globally or only for specific (sub) tasks
    • Remote tasks operate on all current nodes
  • safe
    • Everything can be --dry-run to preview what happens (yet the tasks to include need to be configured)
    • The complete debug output of previous tasks can be viewed with kite log

Contents

Appendix

Installation

You can install kite globally (recommended) or per project

Prerequesites

Locally

  • PHP 5.4+ (for the composer installation, the application can have lower versions)
  • Linux shell (windows might work but is untested)
  • SSH/SCP installed (for working on nodes)
  • git installed (if you use git tasks)
  • composer installed (if you use composer tasks)

Remote

  • PHP
  • SSH access

Global installation

composer global require "netresearch/kite"
~/.composer/vendor/bin/kite -V

Per project installation

cd /var/www/project
composer require "netresearch/kite"
vendor/bin/kite -V

Configuration

Concepts

  • Tasks
    • Smallest, predefined steps
    • See the task reference for tasks shipped with kite
  • Workflows
    • Special kind of task that allows to composer it's subtasks in a class
    • Top level workflows can expose command line arguments and options
    • See the workflow reference for workflows shipped with kite
  • Jobs
    • Outermost kind of task
    • Available as commands on command line
    • Set of tasks and/or workflows defined in arrays (in arbitrary depth)
    • Configurable command line arguments and options
  • Presets
    • Configuration presets (including f.i. common jobs)
  • Configuration file (typo3conf/Kite.php, app/etc/kite.php, kite.php)
    • Defines the jobs; can load and override presets

Variables

The fact that all of the configured tasks are to be ran automated, introduces the need for a variable system that allows you to read from dynamic configurations or change it. Kite provides a basic syntax to access those variables from within strings (all options of tasks, nodes etc.):

Each string inside curly braces inside an option string are evaluated as Symfony Expression Language expressions - f.i.

<?php
$this['messages'] = (object) ['default' => 'Hello world'];
$this['jobs']['echo'] = [
    'description' => 'Output the default message from (\{config["messages"].default\})',
    'task' => [
        'type' => 'output',
        'message' => '{config["messages"].default}'
    ]
];

As you see above, by quoting the braces, you can avoid that the expression is evaluated. Please see the Symfony Expression Language Syntax for help on how to use the expressions.

Variable scopes

The variable scopes are very similar to those in JavaScript. This means that you can access all variables from the parent scope within the current scope unless you have a variable in the current scope that's name is the same. To disambiguate you can use the special variables this and parent.

Task or workflow options are always bound to the scope of the task

This means, that they have to explicitly be set for the task or workflow and can not be read from parent tasks (like jobs or workflows). However sub tasks of those tasks can access those options without prefix when they don't have an option with the same name or with parent prefix otherwise.

Global variables

Additionally to the options from the current and parent tasks there are some global variables available:

  • job
    • The current job object (instance of \Netresearch\Kite\Job)
  • kite
    • Object with some information about the kite package (path and relative dir)
  • config
    • The config array object (as in configuration file)
  • composer
    • Composer service object providing keys packages and rootPackage

Special variables

  • this
    • By using this you can force the variable to be not looked up in the parent scopes
    • but only within the current.
  • parent
    • Points to the parent object

Available functions

Kite ships with the following expression language functions:

  • isset(variable) and empty(variable)
    • Behave just like their PHP equivalents. Only available for variable objects, such as
    • tasks, nodes, workflows or jobs and their objects (f.i. not for configuration arrays)
  • get(variableName, variableValue) and set(variableName, variableValue)
    • Get or set the variables (f.i. set('job.owner', node.user)
  • answer(question)
    • Let the (command line) user answer a question and return the result
  • answer(question)
    • Let the (command line) user answer a confirmation question and return the result
  • select(question, options)
    • Let the (command line) select from an array of options
  • any PHP function
    • Lets you call PHP functions as you are used to in PHP - e.g. str_replace('\\\\', '/', config['somePath']) (Note the four back slashes which are required to pass a single escaped backslash to Expression Language)

Kite configuration file

You need a file called "Kite.php" to set up config (where to deploy). For TYPO3 projects it should be placed here: typo3conf/Kite.php, for Magento app/etc/kite.php and for all other applications just kite.php. A basic example could be

<?php
// Example for a project without a staging environment

// This loads configuration with common jobs
$this->loadPreset('common');

// This configuration is loaded on execution of deploy or rollout job
$this['stages']['staging']['node'] = array(
    'host' => 'set host here',
    'deployPath' => 'set path on host here',
    'webUrl' => 'set url here',
    'php' => 'php56',
);

// no staging is available
unset($this['stages']['production']);

?>

Jobs

Jobs are to be configured in the key jobs in the configuration. They can contain a single task, an array of tasks or a workflow (always only one of them).

<?php
// Job, running a single task
$this['jobs']['echo'] = [
    'description' => 'Output a message',
    'arguments' => [
        'message' => [
            'type' => 'string',
            'required' => true,
            'label' => 'The message to output'
        ]
    ],
    'task' => [
        'type' => 'output',
        'message' => '{job.message}'
    ]
];

// Job, running a workflow
$this['jobs']['diagnose'] = [
    'description' => 'Show status of packages',
    'workflow' => 'Netresearch\Kite\Workflow\Composer\Diagnose'
    // can written as follows also:
    // 'workflow' => 'composer-diagnose'
];

Nodes

Whenever you set a key named node or nodes on a job, workflow or task it's value will be mapped to an aggregate of node models. Those models have the following default configuration:

<?php
array(
    'user' => '',
    'pass' => '',
    'port' => '',
    'url' => '{(this.user ? this.user ~ "@" : "") ~ this.host}', // SCP/SSH URL
    'sshOptions' => ' -A{this.port ? " -p " ~ this.port : ""}{this.pass ? " -o PubkeyAuthentication=no" : ""}',
    'scpOptions' => '{this.port ? " -P " ~ this.port : ""}{this.pass ? " -o PubkeyAuthentication=no" : ""}',
    'php' => 'php', // PHP executable
    'webRoot' => '{this.deployPath}/current',
    // No default values, required to be set:
    // 'webUrl' => 'http://example.com',
    // 'host' => 'example.com',
    // 'deployPath' => '/var/www'
);

Development packages

The default kite jobs checkout and merge work only on packages which are white listed as development packages. As deploy and rollout utilize those jobs they stick to this behaviour as well.

You can white list packages in three ways: by package name, by git remote url or by package path. When none of them is given all git packages will be used as development packages. You can change the white lists in your Kite configuration file:

<?php
// The following whitelist types are available (evaluated by OR)
// ... for the package names
$this['composer']['whitelistNames'] = 'netresearch/.*';
// ... for the git remote urls
$this['composer']['whitelistRemotes'] = 'git@github.com:netresearch/.*';
// ... for the package paths
$this['composer']['whitelistPaths'] = 'vendor/netresearch/.*';

The default configuration is as follows:

<?php
$this['composer']['whitelistNames'] = $this['composer']['whitelistPaths'] = null;
$this['composer']['whitelistRemotes'] = '{preg_quote(preg_replace("#/[^/]+$#s", "", composer.rootPackage.remote), "#")}/.+';

This means that only packages which share the part before the last occurrence of / with the application remote will be considered as development packages (f.i. if your application git remote is https://github.com/netresearch/kite.git then only packages with a remote like https://github.com/netresearch/* will match).

Deployment configuration

Stages

As you saw in the example in Kite configuration file, there is a top level configuration element named stages. They are set by the common preset and hold configuration only used for each of it's keys (such as staging and production by default). They are evaluated by workflows based on the stage-select workflow, which takes the stage(s) to use from either command line or a select question. After a stage was selected ALL of it's values are set to the corresponding task (such as deploy).

The stages have no special meaning and are not handled in a special way - they only play together with the stage based tasks (deploy and rollout from the common preset and ccr from the typo3 preset) because those are configured so.

Deployment

The deployment workflow deploys your application to exactly one stage (whereas the rollout just runs the deployment workflow for each until the selected stage). Thereby it does the following steps:

  1. Run kite composer diagnose to assert that your application is at a defined state (nothing uncommited, unpushed, unpulled, lock file up to date etc.)

  2. Run composer checkout with the parameters you provided for the stage:

    1. branch - The branch to checkout. In common preset they are configured as follows:

      <?php
$this['stages'] = [
    'staging' => [
        'branch' => '{replace("/", "-", composer.rootPackage.name)}-staging',
        'merge' => true,
        'createBranch' => '{config["composer"]["whitelistNames"] || config["composer"]["whitelistRemotes"] || config["composer"]["whitelistPaths"]}'
        // add nodes or node in your config
    ],
    'production' => [
        'branch' => 'master',
        // add nodes or node in your config
    ]
];

    2. merge - Whether to merge the currently checked out branch into the branch to checkout

    3. createBranch - Whether to create the branch if it doesn't exist. This is by default set to true for the staging stage, when Development packages are white listed (which are by default).

    4. rsync - configuration for rsync task invoked (f.i. with excludes option)

  3. Creates a new release from the current release on each node {deployPath}/releases

  4. Rsync the current local state to the new release dir on each node

  5. Symlink shared directories and files (shared means shared between the releases) - the shared directories and files are expected to be at {deployPath}/shared. They can be configured as seen in the typo3 preset:

    <?php
$this->merge(
    $this['jobs']['deploy']['task'],
    [
        'shared' => [
            'dirs' => ['fileadmin', 'uploads', 'typo3temp']
        ]
    ]
);

    To illustrate the behaviour of the stage configuration here's an example setting the shared directories differently for each stage:

    <?php
$this->merge(
    $this['stages'],
    [
        'staging' => [
            'shared' => [
                'dirs' => ['shared_dir_1', 'shared_dir_2'],
                'files' => ['file1', 'file2']
            ]
        ],
        'production' => [
            'shared' => [
                'dirs' => '{config["stages"]["staging"]["shared"]["dirs"]}',
                'file' => 'file'
            ]
        ]
    ]
);

  6. Switch the previous release pointer ({deploypath}/previous) to the current release.

  7. Switch the current release pointer ({deploypath}/current) to the new release.

  8. Invoke the onReady task if any. F.i.:

    <?php
$this->merge(
    $this['jobs']['deploy']['task'],
    [
        'onReady' => [
            'type' => 'shell',
            'command' => 'mail',
            'optArg' => ['s' => 'Deployed to {stage}', 'user@example.com']
        ],
    ]
);

    And to once again demonstrate that each of the stages can override any option on the deployment workflow:

    <?php
$this->merge(
    $this['stages']['production'],
    [
        'onReady' => [
            'type' => 'shell',
            'command' => 'mail',
            'optArg' => ['s' => '[IMPORTANT] Deployed to {stage}', 'user@example.com']
        ],
    ]
);

When you invoke the deployment or rollout jobs with the rollback (--rollback or -r) option, it

  1. switches the next release pointer ({deploypath}/next) to the current release
  2. switches the current release pointer ({deploypath}/current) to the previous release
  3. invokes the onReady task if any.

When you invoke the deployment or rollout jobs with the rollback (--activate or -a) option, it invokes the last three steps of the deployment (switch symlinks, and invoke onReady).

Usage

As stated above, all jobs are available as kite sub commands (kite job-name). You can list the available commands by running

kite [list]

By running

kite help command
#or
kite command --help

you can show help for a specific job/command.

Common commands

  • kite [help [command]]
    • Gives a list of all available commands (jobs) or shows help for the given one
  • kite --workflow=<workflow-name-or-class>
    • Runs a workflow class without requiring it to be inside a job
  • kite --workflow=<workflow-name-or-class> --help
    • Shows the docs (php class doc), arguments and options for a workflow
  • kite log [-l]
    • Shows the last (default), specific (use with caret like ^2 shows the 2nd least log or with a timestamp from kite log -l) or a list of the available log records

Common jobs

  • update [branch]
    • Checks out branch when given
    • Runs git pull followed by composer update
  • kite checkout [--merge] branch
    • Goes through all Development packages and checks out the branch there if it’s available
    • After checking out the branch on a package it goes through all Development packages requiring it and updates the version constraint to that branch
    • When --merge is passed, the currently checked out branch is merged into the branch to checkout
    • When -c is passed, the branch will be created in all Development packages when it doesn't exist there (you can use -p option to limit the operation to certain packages)
  • kite merge [--squash] [--message=”Message”] branch
    • Goes through all composer packages and merges the branch into the currently checked out
  • kite package-foreach [--git] command
    • Runs a command for each composer package (optionally only --git packages)
  • kite cc, kite ccr [stage]
    • Clears caches locally (cc) or on all nodes of a specific stage (contained in TYPO3 preset only for now)

Development workflow

The default kite jobs are designed to ease the development of composer based applications which tends to be pretty complicated when it comes to working with development branches across several packages. What kite can do for you, is to manage the branches in the Development packages along with dependencies on them.

In the following you'll find the kite commands for the common workflow that you do your development within feature (eventually derived from topic branches) or bug fix branches, which you will merge into the master as soon as their changes were reviewed and tested.

  1. Ensure application is on master and up to date

    kite update master

  2. Create the feature branch in the package(s) you want

    This will create the FEATURE branch in package vendor/branch and all installed dependent packages and make them require it in dev-FEATURE (as your application directly or indirectly requires the package it will be checked out in FEATURE branch as well).

    kite checkout -c -p vendor/package FEATURE

  3. Develop

    cd vendor/package
echo '<?php phpinfo(); ?>' > i.php
git add i.php
git commit -m 'Added php info'
git push
cd ../..

  4. (Work on another existing branch)

    kite checkout ANOTHER_FEATURE

  5. Deploy FEATURE branch to staging

    kite checkout FEATURE
kite deploy staging

  1. (Work on another existing branch)

    kite checkout ANOTHER_FEATURE

  2. FEATURE was reviewed, merge it into master and delete it

    kite checkout master
kite merge --delete FEATURE

  3. Roll out master (now including FEATURE) over staging to production

    kite rollout production

Deployment jobs

  • kite deploy [stage]
    • Runs the deployment for all nodes on the given or selected stage
  • kite rollout [stage]
    • Runs the deployment for all nodes for each stage until (including) the given stage

Use public key authentication

To prevent you to have to type your password several times during deployment you should set your public key on your server. Usually this is located here: "~/.ssh/authorized_keys".

Trouble shooting

Every task that's executed including it's output will be logged to a log file inside your home directory. This includes f.i. each command ran on the local and remote shells, their output, debug messages and a lot more. Basically it holds the output, you would get by adding -vvv to your kite command.

Just run kite log when a job failed and you want to know exactly what went wrong.