Genesis is a BOSH Deployment paradigm as well as a tool that supports that paradigm. Genesis makes it easy to deploy your infrastructure. This is done by providing curated “kits” for common deployments that work out-of-the-box to get you started with just a few inputs from the operator, but with enough flexibility to support any configuration changes needed that are future without making you jump through hoops.
The available kits can be found in Github under the
genesis-community organization. Each has a repository that ends
-genesis-kit which contains the the source code and releases.
While you don’t have to go there to download kits for use in
Genesis, you can use this to determine what kits are available,
how to use each kit, how to what versions each kit has, and the
release notes for each version to determine how the kit has
changed. You may also want to peruse the source code to better
understand how the kit works.
Once you have selected the type of kit you want to use, you can initialize a deployment repository based on that kit. To do so, run the following command:
$ genesis init -k <kit-name>
In this case, you specify the kit name without the
-genesis-kit suffix. This will create a directory named based
on the kit name you specified with a
-deployments suffix on the
end. For example, if you wanted to create a new BOSH deployment
repository, you would run
genesis init -k bosh and it would
This new directory will have a
.genesis configuration directory,
containing the metadata that Genesis needs to do its work, but it
won’t contain any deployments yet. See the next section for how
to create deployments.
This newly created directory will also contain an initialized .git directory, making it a fully functional git repository. Genesis deployment repositories are designed to use Github, but any similarly featured git manager that can be reached by your Concourse workers can be used. This includes on-premise products like Github Enterprise.
Your deployment environments are stored in the root directory of your deployment repository, and are in YAML format, similar to standard BOSH manifests. In fact, they are fragments of BOSH manifests, so anything you could put in a BOSH manifest, you can put in environment files, but more about that later.
To create a new environment, make sure you’re in the root directory of your deployment repository and run the following command (using an environment named myorg-site-demo for example):
$ genesis new myorg-site-demo
Depending on the kit used in this deployment repository, you will be asked a series of questions. These questions should be self-evident or sufficiently explained in the text preceding each question. For further clarification, consult the README.md file in the kit’s repository as discussed in the How To Find Available Genesis Kits section above. If you’re still unsure of how to answer the question, open an issue with the kit to ask for clarification.
Kits may have features that are optional. You will be asked if you want to include the feature during this setup session. If the feature you wish to enable has further parameters, you will be asked those too.
If everything works correctly, a new environment deployment file
myorg-site-demo.yml will be created. It will also create
a file named
myorg.yml if this doesn’t already exist. This is
because Genesis uses a hierarchical file structure for environments
to reduce duplication and facilitate propagation of changes
through pipeline deployments.
Genesis environment files are meant to make your deployments easier, and this included deployment infrastructure and lifecycle management. To this end, Genesis uses hierarchical naming to group and distribute environment information.
Environment files use dashes (
-) to build up a chain of
configuration files that will be merged together to define the
actual environment. This means that in the case of a file named
a-b-c-d.yml, configuration contained in
a-b-c.yml will all be merged underneath
a-b-c-d.yml to fully
define the environment
a-b-c-d. In practice, this hierarchy is
broken down into a top “org” level, a infrastructure level, and
the purpose (i.e. staging or prod). The longest-named environment
file will contain a
params.env property, which indicates that
this file represents an actual environment that can be deployed.
Not all hierarchical files need to be present; Genesis will only
use what is there. In the example above, it would be perfectly
acceptable to have
You can build another deployment environment file of another one. This can be done to create “personal dev” environments that are based on a common deployment (i.e. preprod).
Anything that applies to most deployments of a given deployment type can, and should, be pushed to the top level file. If one or two deployments do not use a given property, they can over-ride it in their specific file where it will be more obvious that they deviate.
Not all deployments are to the same infrastructure. You may have a
BOSH-lite warden QA sandbox environment, vSphere-based development and
preproduction environments, and a bare-metal production environment. In
these situations, you would put infrastructure in the intermediary
myorg-vspherelab.yml). In this way, you keep the
infrastructure properties shared between those that use a common
infrastructure, but without breaking those that don’t.
Where this sharing configuration shines is during pipeline propagation of deployments. By putting the changes at the top levels, you can test the actual changes in sandbox or pre prod environments before pushing them to production, and not getting downtime due to incorrect or partially omitted changes. Similar for infrastructure changes that are shared between test and production environments. This leaves only things that are unique to the environment, such as scaling or IP addresses in the bottom-most environment.
Once a deployment environment has been created, you use Genesis to
deploy it. Genesis
deploy command is a wrapper around BOSH
deploy and support most of its options, such as
--fix. To deploy, run
genesis deploy my-env
my-env is the name of your environment file (with or
The first thing Genesis will do is compile the environment file, any hierarchical files found, and the kit contents into a manifest. It will also at this time pull any secrets out of Vault to be embedded in this manifest. If you are missing any secrets, these will be reported and the deployment will fail.
Once the manifest is compiled, it’s BOSH’s turn to deploy the
manifest to the target BOSH director. By default, it will use the
BOSH director that is named the same as your environment. If your
BOSH director is named differently, you can set
the alias of the desired BOSH director, or preferably, add an
alias for the BOSH directory with the same name as the
environment. (You can have multiple aliases for a single BOSH
Ideally, your environment should just deploy. Kits are designed
to work out of the box for the majority of situations, but you may
encounter one that doesn’t. The first step in debugging a
deployment that doesn’t work is obviously the error messages that
a failed deployment spits out. One of the most common issues that
is outside the control of Genesis is a mismatch between what the
kit needs for resources and what resources are provided in the
cloud-config on the BOSH director. Consult the Kit usage guide
for what resources it uses by name, and how to overwrite them if
Once deployed, Genesis will store a redacted version of the
.genesis.manifests.<my-env-name>.yml. You will
want to commit this to your repository after each deployment so you have
a local copy of the currently deployed manifest for future
reference. When deployments are done by the Genesis-created
Concourse pipelines, this will be automatically done.
In a similar manner to deploying an environment, you can generate
a manifest for an environment without deploying it. Use the
genesis manifest my-env command to do so, where
my-env is the
name of your environment.
By default, this will not be redacted (it will contain all the
secrets). If you’d prefer to have a redacted manifest for
sharing, or if you don’t have access to your vault, you can
generate a redacted manifest with the
This manifest is compiled by machine, using the
All maps are sorted alphabetically, which means you need to pay
careful attention to the structure when searching for arrays of
maps, as often the
name: <id> key is not aligned with the
- indicating the start of the array element.
The kit being used by an environment will most likely be found in
the top hierarchical file for that environment. This is done to
allow upgrading environment through pipelines. The version will be
specified under the
To upgrade a kit to a newer version, you will need to download the
new version and set the environments to use it. The
genesis fetch-kit <kit>[/<version>] command is used to get the kit. If
you want to use the latest version, you can omit the
$ genesis fetch-kit vault/1.5.0 Attempting to retrieve Genesis kit vault (v1.5.0)... Downloaded version 1.5.0 of the vault kit $ genesis fetch-kit vault Attempting to retrieve Genesis kit vault (latest version)... Downloaded version 1.5.1 of the vault kit $ genesis list-kits Kit: vault v1.4.0 v1.5.0 v1.5.1
Once you have the version of the kit you want to use, you need to
update the kit version. This will be found under the
kit.version property, most likely in the top hierarchical
environment file, but could also be located in the deployment
New kits may introduce new properties or secrets. Consult the
release notes, edit the environment files manually and add
missing secrets. You should not have to
regenerate the environment file using the
genesis new ... command,
but you can if you want to.
Ideally, you will commit these changes to your repository and the
pipeline will roll out the changes, testing in sandbox before
progressing through to production. You can however, simply run
genesis deploy ... to deploy these changes.
Kits are designed to pull in parameters it needs from these environment files, but these files can encompass any manifest propertied explicitly. This is sometimes needed to support esoteric features of the BOSH release that are not supported by the kit.
Feel free to edit the environment files as you would a manifest. There is nothing magical about them, but keep in mind the following:
These are YAML files, so they use spaces, not tabs, and follow the formatting rules of YAML.
It is best to be familiar with the kit and the manifests it
generates if you are going to overwrite base functionality of the
kit. You may accidentally end up changing something that uses a
params which will render that parameter
non-operative and confuse future debugging.
With the exception of
genesis new --force, Genesis will not
overwrite an existing environment file, so feel free to add
comments and alter the content without worry that it will be lost.
Files are merged with
spruce which supports powerful
operators to help keep your files modular and
"DRY". Put any configurable items under
params keys, and
(( grab ...)) or
(( concat ...)) operators to use
them. Put common structures in the top level, and different
parameters used by that structure in the deployment environment
files if they aren’t meant to propagate through the pipeline.
Sometimes you will get messages stating you are missing secrets in the Vault:
[Checking generated certificates] ✘ secret/snw/tliebel/lab/shield/certs/ca [CA certificate] ✘ :certificate ✘ :combined ✘ :crl ✘ :key ✘ :serial
This is because genesis verifies that all secrets are present prior to
attempting to compile the manifest. If you would like to run this check
manually, you can run
genesis check-secrets <my-env>.
$ genesis check-secrets snw-tliebel-lab Retrieving secrets for snw/tliebel/lab/shield...ok [Checking generated credentials] ✔ secret/snw/tliebel/lab/shield/agent [ssh] [Checking generated certificates] ✘ secret/snw/tliebel/lab/shield/certs/ca [CA certificate] ✘ :certificate ✘ :combined ✘ :crl ✘ :key ✘ :serial ✔ secret/snw/tliebel/lab/shield/certs/server [certificate] ✔ secret/snw/tliebel/lab/shield/vault/ca [CA certificate] ✔ secret/snw/tliebel/lab/shield/vault/server [certificate]
If you do have missing secrets, possibly due to a kit upgrade, you
genesis add-secrets my-env to add any missing secrets to
my-env deployment environment.
You can change all the secrets that were generated when you
created a new environment, either as a response to accidentally
leaked secrets, or as part of a scheduled cycling of secrets.
This is done with the
genesis rotate-secrets my-env command.
This will recreate all secrets generated for the kit used by the
my-env deployment environment, with the exception of any secret
that the kit marked as
fixed and any self-signed CA
certificates. If you want these rotated as well, specify the
If you would like to check or set the secets provider, you can run
genesis secrets-provider name.
In the secret provider Vault, all Genesis environments store their secrets
under a predictable path. Under the base
secret/ , the environment is
split on dashes (
-) and joined by slashes (
/), then the deployment type
is added to the end. For example, for a environment named
deployment using the
cf-rabbitmq-genesis-kit, the path would be
secret/myorg/site/prod/cf-rabbitmq. Notice that any dashes in
the deployment type are not converted to slashes.
This information is useful if you ever need to get secrets from a
different deployment. You can reference another environment’s
secrets in the Vault by specifying a relative path to the
secret: (( vault meta.vault "/../staging/thing/path/to/secret:key" ))
To rename an deployment that hasn’t been deployed is easy: rename the file. However, there are two caveats:
Ensure that anything you pushed up to the hierarchical chain is
moved over to an equivalent file for the new name. For example,
if you are renaming
our-new-thing.yml, anything in
my.yml that’s applicable should
be moved to
our.yml, and similarly anything in
You have to move the secrets stored under the old
environment’s path to a one for the new environment. To continue
the above example, to move the secrets for a BOSH deployment
You can move secrets via
safe move -r secret/my/original/thing/bosh \ secret/our/new/thing/bosh
If you’ve already deployed an environment, it is HIGHLY recommended to not rename the deployment in the BOSH director. The best method is to backup any data on the existing deployment, delete it, rename the environment file as described above, and deploy it and restore the data from the backup. Shield makes it easy to backup and restore most deployment data.
Genesis does not have a command to delete an existing environment.
Instead, it defers to the BOSH CLI to do this, with the simple
Increasing or decreasing the number of VMS in an environment is one of the most common operations you may encounter. Each kit may have a parameter to increase different VM types for the deployment, but you may want to change the VM count for a type that is not provided for by the kit.
If the kit supports it, changing VM counts is as simple as
params.<param> in the deployment environment file.
For example, the Concourse kit supports setting the counts of the
web and worker instances. Simply change (or add) these parameters
in the yml file:
--- # my-concourse.yml params: num_web_nodes: 3 workers: 10
If there is no first-class support by a kit to change VM counts
for the VM type you want to adjust, you can provide this yourself.
For example, if the Concourse kit didn’t support changing worker
counts, you could add this to the
--- my.yml instance_groups: - name: worker instances: 3
By placing it in the top hierarchical file, you effectively add support for changing worker counts for all deployment environments, as well as specifying a default value for any deployment environment that doesn’t provide an override.
In the these examples, we changed the instance counts, but the same principle can be used to scale up other aspects of deployments, such as CPU, memory, or disk sizes — just keep in mind this may need to be done indirectly by specifying instance or disk types and adding the desired resources in your cloud-config.
Genesis provides a simple way for you to update stemcells for deployments based off of Genesis Kits. The typical process is to upload a new stemcell to the BOSH Director and perform a
genesis deploy on each deployment.
In each kit there is a default for the os flavor and version:
params: stemcell_os: ubuntu-trusty stemcell_version: 3468.latest
To upgrade the stemcell version for a deployment start by uploading a new stemcell to the BOSH Director. Full instuctions for doing this are located in the BOSH Runbook under Uploading a Stemcell. It should be noted that most kits ship pre-compiled BOSH releases and do not allow the major version of OS of the stemcell to be changed. If you are currently using
3468.22 you should upload a newer version in the
3468 series, such as
Once the newer stemcell is uploaded you have two choices, each should be followed by a
params.stemcell_versionas defined by the kit
Overriding the default stemcell version in the Genesis kit is not normally recommended. If you want more explicit controls over the version of the stemcell you are using you can add the following parameters to your environment yml:
params: stemcell_os: ubuntu-trusty stemcell_version: see_options_below
<major>.latest- Grabs the highest minor stemcell for the listed major version, (ie
latest- Grabs the highest major.minor stemcell for the stemcell_os. This is not recommended since kits use pre-compiled releases and this option would allow you to try and use a different major release of stemcell.
3468.2- Hard codes a this specific major.minor stemcell version
Once the new stemcell is uploaded to the BOSH Director and any overrides to the version are provided to the environment yaml file perform a
genesis deploy on the kit deployment.