Blacksmith Services

On-demand and shared services are provided by Blacksmith. Blacksmith is a service broker for Cloud Foundry that uses a dedicated BOSH director to deploy actual VMs in response to service provisioning requests. If a CF user wants a PostgreSQL box all to themselves, Blacksmith will do so.

The magic of Blacksmith is accomplished by way of Forges, small bits of configuration, packaged up as a BOSH release, that can be added into a Blacksmith deployment.

The Blacksmith Genesis Kit packages five forges at the moment:

  1. Redis
  2. RabbitMQ
  3. PostgreSQL
  4. MariaDB
  5. Kubernetes

Refer to their specific project pages for more details information on their features and configuration.

The Blacksmith Kit documentation explains how to configure each of these Forges and their service offerings.

Get information about your Blacksmith

If deployed with Genesis, genesis info my-env will dump most of the relevant information for the deployment. This includes general env and deployment information, URLs and creds for the bosh director and UI, and the catalog of services.

genesis info my-env

================================================================================

BLACKSMITH Deployment for Environment 'my-env'

  Last deployed about a week and a half ago (10:04PM on Oct 25, 2019 UTC)
             by admin
        to BOSH my-env
   based on kit blacksmith/0.5.0
          using Genesis v2.6.16
  with manifest .genesis/manifests/my-env.yml (redacted)

--------------------------------------------------------------------------------

ca certificate
 ...


bosh env
  Using environment 'https://10.128.80.129:25555' as client 'admin'

  Name               my-env-blacksmith
  UUID               e33feb30-0e9d-4461-906e-18239abba00c
  Version            270.2.0 (00000000)
  Director Stemcell  ubuntu-xenial/315.36
  CPI                vsphere_cpi
  Features           compiled_package_cache: disabled
                     config_server: disabled
                     local_dns: enabled
                     power_dns: disabled
                     snapshots: disabled
  User               admin

  Succeeded


blacksmith (internal) bosh director
  bosh url: https://10.128.80.129:25555
  username: admin
  password: ...


blacksmith web management UI
  web url:   http://10.128.80.129:3000
  username:  blacksmith
  password:  ...
  clickable: ...


blacksmith catalog
  Service     Plans          Tags
  =======     =====          ====
  mariadb     standalone     blacksmith
                             dedicated
                             mariadb

  postgresql  small-cluster  blacksmith
              standalone     dedicated
                             postgresql

  rabbitmq    small-cluster  blacksmith
              standalone     dedicated
                             rabbitmq

  redis       standalone     blacksmith
                             dedicated
                             redis


================================================================================

Register a Blacksmith Broker

Before your Cloud Foundry users can start provisioning services via your new Blacksmith, it has to be registered with Cloud Foundry, and enabled in the CF marketplace.

The register Genesis kit addon makes this easy.

$ genesis do my-env register
Running register addon for my-env
authenticating to https://api.system.10.128.80.140.netip.cc as admin...
Setting api endpoint to https://api.system.10.128.80.140.netip.cc...
OK

api endpoint:   https://api.system.10.128.80.140.netip.cc
api version:    2.138.0
Not logged in. Use 'cf login' to log in.
API endpoint: https://api.system.10.128.80.140.netip.cc
Authenticating...
OK

Use 'cf target' to view or set your target org and space.
creating service broker my-env-blacksmith...
Creating service broker my-env-blacksmith as admin...
OK

enabling service access...
 ...

You can verify that the broker is registered, by checking that it is listed in the output of cf service-brokers

For more details, refer to Managing Service Brokers, in the official Cloud Foundry documentation.

This makes all the plans and services owned by the your-env-name-blacksmith broker available to all orgs and spaces in your Cloud Foundry instance. If you need stricter permissions, check the Marketplace Access Control section of the official Cloud Foundry documentation.

You can verify the publication of your Blacksmith broker’s plans by running cf marketplace and checking that you see the plans you defined in your Blacksmith manifests.

Access the Blacksmith Management UI

The Blacksmith Management Web UI provides an overview of what is going on with a single Blacksmith Services Broker.

You can get the UI by using the visit addon

$ genesis do my-env visit
Running visit addon for my-env

The visit addon will perform auth for you. If for any reason, you need to auth manually, all of Blacksmith is protected by HTTP basic authentication.
The username is always blacksmith, and the password can be found in your Vault:

$ safe read secret/my/env/blacksmith/broker:password

The UI lists the current catalog:

Catalog Listing

It also lists all deployed instances:

Services Listing

Finally, the last 8,000 (or so) log lines are buffered and printed to the Log section:

Blacksmith Logs

Provision a Service via cf

Cloud Foundry users can create new services using the standard process, with the cf command-line utility:

$ cf create-service dedicated-redis standalone my-kvstore

This will create a dedicated Redis instance, as a standalone, single-VM deployment. The name of the service (which is arbitrary) will be my-kvstore.

Blacksmith is an asynchronous service broker, so the create-service operation will return immediately, but the service will be marked as in progress, until the Blacksmith BOSH director has finished deploying it. You can check on the progress by issuing:

$ cf service my-kvstore

Once the service is created, it can be bound to an application:

$ cf bind-service key-value-app my-kvstore

The first argument is the application name (from a cf push), and the second argument is the name we gave the service in our cf create-service call.

For more in-depth review of Cloud Foundry services and applications, check the Service Overview in the official Cloud Foundry documentation.

Remove a Service via cf

To remove a service, Cloud Foundry users should use the standard cf command-line utility.

First, get a list of services provisioned in the current space:

$ cf services

If the service you want is bound to any applications, it will need to be unbound before it can be deleted:

$ cf unbind-service app-name service-name

Then, it can be deleted:

$ cf delete-service service-name

Blacksmith is an asynchronous service broker, so the delete-service operation will return immediately, but the service will be marked as in progress, until the Blacksmith BOSH director has finished deploying it. You can check on the progress by issuing:

$ cf service service-name

but be aware that once the operation succeeds, the service information will be purged from the Cloud Foundry database, so cf service will start failing because the service is unknown.

For more in-depth review of Cloud Foundry services and applications, check the Service Overview in the official Cloud Foundry documentation.

Install boss

boss is a command-line utility for interacting with blacksmith. boss can be helpful to see what blacksmith services are running or to troubleshoot provisioning services outside of a Cloud Foundry or Kubernetes cluster. Blacksmith, like all Cloud Foundry service brokers, conforms to OSB. So, you can use other service broker tools to interact with Blacksmith, such as eden. However, boss is designed specifically for Blacksmith which provides it some advantages, such as being able to see all services provisioned by Blacksmith, provisioned via boss or otherwise.

Installation is simple, download the binary for your environment from the most recent release, and it put in your PATH as boss.

Using boss with your Blacksmith

Using boss as a Genesis addon, there is no setup required. Genesis will hand the creds to boss for the Blacksmith deployment. This can be helpful if you are running multiple blacksmiths. You can quickly go between them by allowing genesis to handle the creds for each deployment.

$ genesis do my-env boss [command]
$ genesis do my-dev-env boss [command]
$ genesis do my-prod-env boss [command]

If you want to be able to use boss independently of Genesis, you can do that, too. The easiest way is to export a few environment variables. You can find the info for these using genesis.

$ genesis info my-env

 ...

blacksmith web management UI
  web url:   http://10.128.80.129:3000
  username:  blacksmith
  password:  ...

 ...

$ export BLACKSMITH_URL=http://10.128.80.129:3000
$ export BLACKSMITH_USERNAME=blacksmith
$ export BLACKSMITH_PASSWORD= ...

boss also supports these being passed in as command line arguments.

-U, --url       (required) URL of Blacksmith
                Defaults to $BLACKSMITH_URL

-u, --username  (required) Blacksmith username.
                Defaults to $BLACKSMITH_USERNAME

-p, --password  (required) Blacksmith password.
                Defaults to $BLACKSMITH_PASSWORD

Provisioning Services via boss

boss allows you to bypass Cloud Foundry or Kubernetes and create services directly in Blacksmith. This can be extremely useful in debugging and diagnosing service related issues, especially if Cloud Foundry or Kubenetes seem to be responsible for some strange behavior.

That said, be sure use this tool responsibly. Services provisioned via boss are done so directly with the Blacksmith broker. These services will appear in the Blacksmith Management UI and these services will still count against your service quotas, but will not show up in cf services. This can make services provisioned using this tool easy to forget about, so remember to clean up after yourself when debugging in this manor.

To see what services are available, use boss catalog

$ genesis do my-env boss catalog
Running boss addon for my-env
Service     Plans          Tags
=======     =====          ====
mariadb     standalone     blacksmith
                           dedicated
                           mariadb

postgresql  small-cluster  blacksmith
            standalone     dedicated
                           postgresql

rabbitmq    small-cluster  blacksmith
            standalone     dedicated
                           rabbitmq

redis       standalone     blacksmith
                           dedicated
                           redis

To provision a service, use boss create

$ genesis do my-env boss create redis/standalone
Running boss addon for my-env
redis/standalone instance jolly-minsky created.

Using boss to Interact with Services

To see what services are running, use boss list. This list all services provisioned on the specified Blacksmith, boss provisioned or otherwise.

$ genesis do my-env boss list
Running boss addon for my-env
ID                                    Service  Plan
==                                    =======  ====
85036887-0030-4707-a496-02b51f41a08f  mariadb  standalone
jolly-minsky                          redis    standalone

You can get information about a service using the commands: boss creds, boss manifest, and boss task.

boss creds and boss task can throw a 500 error. creds will not work until the service is fully deployed and task needs the deployment task to have started before it will return properly.

$ genesis do my-env boss task jolly-minsky
Running boss addon for my-env
# jolly-minsky
Task 496 | 15:52:17 | Preparing deployment: Preparing deployment started
Task 496 | 15:52:18 | Preparing deployment: Preparing deployment finished
Task 496 | 15:52:18 | Preparing deployment: Rendering templates started
Task 496 | 15:52:18 | Preparing deployment: Rendering templates finished
Task 496 | 15:52:18 | Preparing package compilation: Finding packages to compile started
Task 496 | 15:52:18 | Preparing package compilation: Finding packages to compile finished
Task 496 | 15:52:18 | Creating missing vms: standalone/31d6bb24-abdb-49d4-b928-514281e23f0e (0) started

$ genesis do my-env boss creds jolly-minsky
Running boss addon for my-env
!!! API 500 Internal Server Error

$ genesis do my-env boss task jolly-minsky
Running boss addon for my-env
# jolly-minsky
Task 496 | 15:52:17 | Preparing deployment: Preparing deployment started
Task 496 | 15:52:18 | Preparing deployment: Preparing deployment finished
Task 496 | 15:52:18 | Preparing deployment: Rendering templates started
Task 496 | 15:52:18 | Preparing deployment: Rendering templates finished
Task 496 | 15:52:18 | Preparing package compilation: Finding packages to compile started
Task 496 | 15:52:18 | Preparing package compilation: Finding packages to compile finished
Task 496 | 15:52:18 | Creating missing vms: standalone/31d6bb24-abdb-49d4-b928-514281e23f0e (0) started
Task 496 | 15:53:48 | Creating missing vms: standalone/31d6bb24-abdb-49d4-b928-514281e23f0e (0) finished
Task 496 | 15:53:48 | Updating instance: standalone/31d6bb24-abdb-49d4-b928-514281e23f0e (0) (canary) started
Task 496 | 15:54:06 | Updating instance: standalone/31d6bb24-abdb-49d4-b928-514281e23f0e (0) (canary) finished

$ genesis do my-env boss creds jolly-minsky
Running boss addon for my-env
# jolly-minsky
host: 10.128.80.195
password: ...
port: 6379

Cleaning Up After boss

This is a reminder that services provisioned with boss will not show up in cf services, but will count against your service quotas. Luckily, cleaning up is easy using boss delete

$ genesis do my-env.yml boss delete jolly-minsky
Running boss addon for my-env
jolly-minsky instance deleted.

Troubleshoot Service Failures

If a service fails to deploy, Cloud Foundry users will not get much detail from Cloud Foundry itself. This is a self-protective measure taken by the Blacksmith system itself, to avoid leaking sensitive implementation details.

As an operator, you have access to the deployment log for the service, if you know the internal UUID/GUID that Cloud Foundry has assigned to the service instance.

This can be found by running:

$ cf service --guid service-name

From the Blacksmith Management Web UI, go to the Services section and look for the entry in the table that corresponds to the service GUID. The deployment log can be reached by the task.log link.

Services Section

The task log is what you would have seen had you deployed this service yourself, manually, via bosh deploy or genesis deploy:

Task 15 | 20:54:31 | Preparing deployment: Preparing deployment started
Task 15 | 20:54:31 | Preparing deployment: Preparing deployment failed
Task 15 | 20:54:31 | ERROR: [50003] Stemcell with Operating System 'ubuntu-trusty' doesn't exist

In this case, the stemcell hasn’t been uploaded. Oops.

Retrieve Service Credentials as an Operator

From the Blacksmith Management Web UI, you can access the full and unredacted deployment manifest by clicking on the manifest.yml link in the Services section.

Services Section

The credentials are stored in the manifest:

director_uuid: f03bca94-cef8-4257-b2f2-df1b527633c1
instance_groups:
- azs:
  - z1
  instances: 1
  jobs:
  - name: standalone
    properties:
      auth:
        password:
byett87B5RfQvcjvaX1VDEy7ZLwBU31Tz1c0zjUKQVjkHUWrLJyubjV24usa962j
    release: redis-forge
  name: standalone
  networks:
  - name: redis-service
  persistent_disk_type: default
  stemcell: default
  vm_type: sm-2cpu-4g
meta:
  params: {}
  password:
byett87B5RfQvcjvaX1VDEy7ZLwBU31Tz1c0zjUKQVjkHUWrLJyubjV24usa962j
  size: sm-2cpu-4g
name: redis-dedicated-cache-493db828-c58a-4253-aa5b-fb12aa5004ba
releases:
- name: redis-forge
  version: latest
stemcells:
- alias: default
  os: ubuntu-trusty
  version: latest
update:
  canaries: 1
  canary_watch_time: 1000-30000
  max_in_flight: 10
  update_watch_time: 1000-30000

This manifest has been mangled by machines, so it’s not the most legible thing in the world.

Retrieve Service Credentials as an End User

Cloud Foundry users should use cf service my-service and inspect the contents of the VCAP_SERVICES environment variable JSON to determine their service credentials.

Target the Blacksmith BOSH Director

Blacksmith deploys all of its service deployments by way of an internal BOSH director, separate from all of your other BOSH directors. This internal director lives on the Blacksmith broker VM itself.

For more advanced troubleshooting, you may need to SSH into a VM on a service deployment. To do that, you first need to target that director.

Run the bosh Genesis addon:

$ genesis do my-env bosh
Running bosh addon for my-env
Logging you in as user 'admin'...
Using environment 'https://10.128.80.129:25555'

Username (): admin
Password ():

Using environment 'https://10.128.80.129:25555' as client 'admin'

Logged in to 'https://10.128.80.129:25555'

Succeeded

From there, you can run all the normal BOSH commands against the director. You can SSH into deployments, check VM health, etc.

All of your tasks will be attributed to the admin user. Blacksmith itself uses the blacksmith user so as not to confuse things.