Infra: powertools for managing your infrastructure
Infra allows you to define all objects in your infrastructure in a way that you can load it into a GraphQL service. For example: Hosts, HostGroups, Users, Dns, Monitoring, BackupRules, Deployments, etc, and all of their relationships.
Accessing your infrastructure as a graph allows you to do a couple of cool things:
- Generate multiple inventory definitions from a single source (ansible, puppet, rundeck, etc)
- Generate configuration files for backup and monitoring services
- Generate complex firewall scripts
- Configure objects in context. I.e. configure dns records as part of your Deployment, or monitoring rules as part of your HostGroup
- Generate dashboard configurations based on infra objects
- Generate comprehensive infrastructure documentation
- Create scripts that operate on your infrastructure data in language agnostic ways
- As your configuration consists of YAML files only, everything is heavily scriptable (for both importing and exporting data from all relevant systems) and managable through version control.
You define your infrastructure as a set of Resources in YAML files. The format is heavily inspired by Kubernetes. Here's an example:
--- kind: Host metadata: name: my-app-server description: This my application server spec: os: Ubuntu 16.04 publicIp: 192.168.1.1 privateIp: 10.0.1.1 hostGroups: app-servers --- kind: HostGroup metadata: name: app-servers description: All my application server labels: color: green spec: ~
For a more comprehensive example, see the
example/ directory in this repository.
You can define your resources in standalone YAML files, or (as in this example) configure multiple resources in one file using YAML's multiple document feature
If you're familiar with Kubernetes, you'll feel right at home.
Each resource specifies it's type using the
kind key. A list of available resource types is provided below.
Resources contain a
metadata key that allows you to specify the resource name, description, a set of labels and annotations (more on metadata below).
spec key configures the resource, and it's available keys depend on the
kind of resource you're creating.
cd /opt git clone email@example.com:linkorb/infra.git cd infra composer install # See https://getcomposer.org/download/ if you don't have composer installed yet
NOTE: It is strongly recommended to add
bin/ to your environment's
PATH variable, so you can easily invoke the
infra command-line tool from anywhere on your system.
If you're using Bash, you can do this by adding the following line to your
Don't forget to open a new bash session for the changes to take effect (close your terminal, or just run
Infra decides where to load it's infrastructure data based on the
INFRA_CONFIG environment variable. It should contain a path to directory that contains a set of YAML files describing your infrastructure.
You can define the variable wherever you like. For example in your
~/.bashrc, or a
.env file in the root of the infra directory (i.e.
.env.dist file is provided.
INFRA_CONFIG is undefined, it will point to the
example/ directory as a default.
example/ directory contains an example infrastructure configuration (You can view
example/README.md for it's details). To load it, make sure your
INFRA_CONFIG variable points to the
example/ directory (absolute path required), or simply leave
INFRA_CONFIG undefined: the example project is loaded by default.
It is recommended to explore the example project first, before using infra to configure your own infrastructure. The example configuration will contain examples for all relevant resource types, configured in a sensible way to show you all available functionality.
Once you're ready to define your own infrastructure, simply point
INFRA_CONFIG to a new directory (ideally a git repository), and start creating configuration files there.
You can get a list of available commands by simply typing
infra list or just
infra get command lets you explore your infrastructure configuration. Type
infra get --help for it's options. All arguments are optional, and the more you specify, the more specific your response will be. Some examples:
infra get: returns a list of available resource types (and their aliases)
infra get hosts: returns all defined objects of that resource type.
infra get hg: you can use any of the available resource type aliases as shortcuts: hg, hostGroup, HostGroup, hostGroups, HostGroups are all equivalent here.
infra get Host app0: returns a resource definition by type and name.
infra get HostGroup db hosts: returns a calculated property from a resource by type, name and propertyname.
infra query command let's you perform a GraphQL query against your infrastructure configuration.
It reads the query from
stdin and outputs the response (as json) to
infra query < /opt/infra/example/graphql/example.graphql
Introspection is also enabled, so you can execute the following command to get a full export of all supported types with detailed information about available fields:
infra query < /opt/infra/example/graphql/introspection.graphql
infra host:list <hosts> allows you to list a set of hosts. The list you can provide is quite flexible. It's using the host expansion algorithm which is used in all places in infra where you can specify a set of hosts. Some of the command-line utilities allow you to specify a list of hosts, but also linking to hosts from
spec sections in your resource definitions follow this algorithm. This command helps you to test this. Some examples:
app0: (i.e., a host name) returns an list containing only the app0 host
db: (i.e., a hostgroup name) returns an list containing all hosts in this hostgroup.
app0,db0m: (i.e., two host name) returns a list containing these specific hosts.
app0,db: (i.e., a host name and a host group name) returns a list containing hosts in the
dbhost group, plus the specific
db0m,db: (i.e., a host group name and a host name already in that host group) returns a list containing hosts in the
dbhost, not listing the host
prod-east: (i.e., a child host group name of
prod) returns a list in the
prod-west: (i.e., a child host group name of
prod) returns a list in the
prod: (i.e., a parent host group name) returns a list of all hosts linked to the
prodhost group directly, or through one of it's child host groups (
As you can see, you can freely mix and match host names, host group names, and take advantage of hierarchy in host groups.
infra host:exec <hosts> <command> command lets you perform a standard shell command one one or more hosts in bulk.
The first argument is the list of hosts you want to perform the command on (following the host expansion algorithm described earlier), and the second argument is the actual command.
Infra can generate complex firewall configurations based on very simple rule defintions. For example, it allows you to set up a rule that allows access to a specified port from all hosts in group X to all hosts in group Y over public or private IP addresses. It will dynamically generate the required rules based on your infra graph. You can leverage the full power of the host expansion algorithm (described earlier) to specify source and remote target host lists.
infra firewall:show <hosts> (or short-hand notation
infra f:s <hosts>) to let infra generate firewall rules for the specified host.
infra f:s db0m will output the generated iptables rules to the console like this:
#### iptables boilerplate removed for brevity #### # rule=allow-mysql hosts='db' remoteHosts='app' -A INPUT -d 10.0.2.1 -s 10.0.1.1 -p tcp --dport 3306 -j ACCEPT -m comment --comment "host='db1m' remoteHost='app1' rule='allow-mysql'" -A INPUT -d 10.0.2.1 -s 10.0.1.2 -p tcp --dport 3306 -j ACCEPT -m comment --comment "host='db1m' remoteHost='app2' rule='allow-mysql'" # rule=allow-ssh hosts='prod' remoteHosts='' -A INPUT -d 10.0.2.1 -s 192.168.99.99 -p tcp --dport 22 -j ACCEPT -m comment --comment "host='db1m' rule='allow-ssh'" COMMIT
Compare these to the definitions in
Note how you'll find rules defined for both the "db" hostgroup (of which db0m is a direct member), and rules for the "prod" hostgroup (of which db0m is only indirectly a member through the "prod" child hostgroup "prod-east").
Additionally, you'll notice that the db0m server has 2 rules generated for the 2 app servers (app1 and app2). In case you'll add 5 more app servers, infra will automatically generate a rule for each of them (7 in total).
infra firewall:install <hosts> to install the firewall. Infra will ssh into the host(s) and:
- backup the current iptable rules using
iptables-save. The backup is created in /tmp, and the exact filename will be displayed on the console.
- Copy the firewall rules to the server
iptables-restoreon the rules file and output the results.
If you'd like to build tools that leverage your infrastructure graph in your language of choice (php, node, python, ruby, bash, etc), scripts are your solution.
To create your own custom script:
- Add an executable script in the
scripts/directory of your infrastructure configuration repository
- Make sure to
chmod 755the file, as only executable files are detected by infra
- Let your script execute GraphQL queries using the
infra querycommand (pass in the query over stdin, getting the response json from stdout)
- Use the returned data to apply any logic you see fit: export to common config formats, call a set of APIs, generate documentation, etc.
An SDK is not required, but may make your life as an infra script developer more convenient. Currently available SDKs:
Let us know if you've created an SDKs for other languages, we'll be glad to link to it from this section.
Examples are included in the
example/scripts directory. (examples in more languages welcomed as PRs!)
A set of "core" scripts are included in the
scripts/ directory of this repository. The core scripts are for commonly used applications that will be of use to many infra users. If you've created a reusable script that you'd like to be included in the core scripts, be sure to send a PR!
- Command: Add
infra scriptcommand for listing and executing discovered scripts
- Command: Add
infra servecommand that spins up a GraphQL server
- Feature: Allow in-line resources in order to define resources as part of their container (i.e. DnsRecord as part of Deployment etc)
- Feature: Allow variable expansion in resource configuration (i.e. to access parent container variables)
- Feature: Set resource index value (incremental, from 1) and output in resource listings. Allow to fetch resource by index. This should simplify exploring using the
- Feature: Create minimal PHP SDK for scripts to simplify querying
- Feature: script discovery
- Feature: infra manifest file (for base variables, directory references, bulk script commands, etc)
- Feature: support wildcards in host expansion algorithm (i.e. by using fnmatch)
- Refactor: extract sensu commands into scripts
- Refactor: extract ansible commands into scripts
- Refactor: Loader in external class
- Refactor: SSH or other infra-specific methods in external services (so Infra becomes general resource container)
- Script: Add script to generate HTML documentation from infra
- Script: Add script to fetch host facts
- Script: Add grafana dashboard generator
- Script: Add DNS zone publisher
- Script: Add Rundeck inventory generator
- Feature: Add option to augment loaded resources with labels (i.e. for host facts)
- Resource: BackupRule + script for BackupPC, rsync, or similar
- Resource: CronJob + conf.d/infra file generator
- Resource: DockerEngine, DockerApp. With docker-compose YAML as inline spec
- Resource: Secret
- Resource: DatabaseCluster or MysqlCluster to setup and monitor replication
- Resource: Database or MysqlDatabase
- Resource: Ingress or NginxSite or ApacheVhost for ingress configuration
- Resource: GitRepository
- Resource: Deployment / Ansistrano / Capistrano / DeployerPHP configurations
- Resource: File to manage deployment configuration files
MIT. Please refer to the license file for details.
Check out our other projects at linkorb.com/engineering.
Btw, we're hiring!