gamegos/consul-imex

Import/Export Tool for Consul Key/Value Storage

1.0.0-rc3 2018-11-14 07:55 UTC

Requires

• php: >=5.5.9
• ext-json: ~1.3
• guzzlehttp/guzzle: ^6.3
• symfony/console: ^3.0

Requires (Dev)

• gamegos/php-code-sniffer: *

MIT

• Safak Ozpinar

• consul importer
• consul exporter
• json to consul
• consul to json

README

Consul-Imex is a simple import/export tool for Consul key/value storage over Consul HTTP API.

• Export all the keys in a prefix, save it as a json file.
• Import the keys from a json file to Consul under a specified prefix.
• Copy keys:
  • from a prefix to another prefix.
  • from a server to another server under a specified prefix.

Installation

You can install Consul Imex in several ways:

• Executable phar file (requires PHP >=5.5.9)
• Docker image (requires Docker engine)
• Composer dependency (requires Git, Composer and PHP >=5.5.9)
• Single PHP project (requires Git, Composer and PHP >=5.5.9)

Install as an executable phar file

Download the phar file from https://github.com/gamegos/consul-imex/releases/download/1.0.0-rc1/consul-imex.phar, then assign execute permission for it.

Example:

wget -O /usr/local/bin/consul-imex https://github.com/gamegos/consul-imex/releases/download/1.0.0-rc1/consul-imex.phar
chmod +x /usr/local/bin/consul-imex

Install as a docker image

docker pull sozpinar/consul-imex

Install as a Composer dependency

composer require gamegos/consul-imex

Install as a PHP project

Install via composer:

composer create-project gamegos/consul-imex

Or clone/download and install manually:

git clone https://github.com/gamegos/consul-imex.git
cd consul-imex
composer install

Usage

Run as an executable phar file:

consul-imex <command> [options] [arguments]

Run as a docker container:

See Notices for Docker Usage for detailed information of docker usage.

docker run -t -v `pwd`:/consul-imex sozpinar/consul-imex <command> [options] [arguments]

Run as a composer vendor binary:

composer exec -- consul-imex <command> [options] [arguments]

Run as a PHP script:

php scripts/consul-imex.php <command> [options] [arguments]

Export

Usage:

consul-imex export [options] <file>

Arguments:

• file: Output data file.

Options:

• --url (-u): Consul server url.
• --prefix (-p): Path prefix.
• --token (-c): Consul token.

Import

Usage:

consul-imex import [options] <file>

Arguments:

• file: Input data file.

Options:

• --url (-u): Consul server url.
• --prefix (-p): Path prefix.
• --token (-c): Consul token.

Copy

Usage:

consul-imex copy [options] <source> <target>

Arguments:

• source: Source prefix.
• target: Target prefix.

Options:

• --source-server (-s): Source server URL.
• --target-server (-t): Target server URL. If omitted, source server is used as target server.
• --source-server-token (-c): Source server Consul token.
• --target-server-token: Target server Consul token. If omitted, source server token is used as target server token.

Examples

Export

$ consul-imex export -u http://localhost:8500 -p /old/prefix my-data.json
93 keys are fetched.

Export using token key:

$ consul-imex export -c 4874874a-2a3s-7892-123e-597a4e1v87e1 -u http://localhost:8500 -p /old/prefix my-data.json
93 keys are fetched.

Import

$ consul-imex import -u http://localhost:8500 -p /new/prefix my-data.json
93 keys are stored. (25 new directories are created.)

Import using token key:

$ consul-imex import -c 4874874a-2a3s-7892-123e-597a4e1v87e1 -u http://localhost:8500 -p /new/prefix my-data.json
93 keys are stored. (25 new directories are created.)

Copy

Copy keys from /old/prefix to /new/prefix:

$ consul-imex copy -s http://localhost:8500 /old/prefix /new/prefix
93 keys are fetched.
93 keys are stored. (25 new directories are created.)
Operation completed.

Copy keys from /old/prefix to /new/prefix using token:

$ consul-imex copy -s http://localhost:8500 -c 4874874a-2a3s-7892-123e-597a4e1v87e1 /old/prefix /new/prefix
93 keys are fetched.
93 keys are stored. (25 new directories are created.)
Operation completed.

Copy keys under /my/prefix to another server:

$ consul-imex copy -s http://localhost:8500 -t http://anotherhost:8500 /my/prefix /my/prefix
93 keys are fetched.
93 keys are stored. (25 new directories are created.)
Operation completed.

Copy keys under /my/prefix to another server using token:

$ consul-imex copy -s http://localhost:8500 -c 4874874a-2a3s-7892-123e-597a4e1v87e1 -t http://anotherhost:8500 --target-server-token 6242c15a-9w4x-2318-61a2-8as8c47317 /my/prefix /my/prefix
93 keys are fetched.
93 keys are stored. (25 new directories are created.)
Operation completed.

Copy all keys to another server:

$ consul-imex copy -s http://localhost:8500 -t http://anotherhost:8500
492 keys are fetched.
492 keys are stored. (108 new directories are created.)
Operation completed.

Notices for Docker Usage

Input/Output File Location

To use import and export commands with docker, the input/output files must be accessible by the container. The default working directory of the image is /consul-imex and input/output files are placed under this directory by default.

Examples for export command:

Example 1: Mount a host directory to the container for export operation, then the container will create my-data.json file in the host directory.

$ docker run -it -v `pwd`:/consul-imex sozpinar/consul-imex export -u 192.168.1.20:8500 -p /foo/bar my-data.json
93 keys are fetched.

Example 2: Copy the output file to your working directory after export operation.

$ docker run -it sozpinar/consul-imex export -u 192.168.1.20:8500 -p /foo/bar my-data.json
93 keys are fetched.
$ docker cp `docker ps -ql`:/consul-imex/my-data.json .

Examples for import command:

Example 1: Mount a host directory to the container for import operation, then the container will read my-data.json file from the host directory.

$ docker run -it -v `pwd`:/consul-imex sozpinar/consul-imex import -u 192.168.1.20:8500 -p /new/prefix my-data.json
93 keys are stored. (25 new directories are created.)

Example 2: Mount a file to the container and use it for import operation. This method does not require the input file to be placed in the default working directory.

$ docker run -it -v `pwd`/my-data.json:/my-data.json sozpinar/consul-imex import -u 192.168.1.20:8500 -p /new/prefix -v /my-data.json
93 keys are stored. (25 new directories are created.)

Network Configuration

If your Consul service is in a private network or does not have a public URL, you may have to set up a custom network configuration for the docker container.

Example

docker run -it --net=host sozpinar/consul-imex copy -s http://localhost:8500 -t http://anotherhost:8500

If the default docker network type is bridge then the running container does not recognize 'localhost'. So we simply add --net=host argument to make the container to use the network of the host machine.

Known Issues

Directories with values

Consul key/value storage allows a directory to have a value like an ordinary key. If a directory has a value, Consul Imex will ignore the value or the sub-keys; this depends how the keys are ordered.