Google Cloud KMS

This guide will show you how to set up your Teleport Cluster to use the Google Cloud Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your Teleport cluster.

Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Server's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use Google Cloud KMS, all private key material for these CAs will be generated, stored, and used for signing inside of Google Cloud KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave Google Cloud KMS.

If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have already generated software private keys, a CA rotation must be performed. Read on to migrating an existing cluster to learn more.

Prerequisites

• Teleport v17.0.0-dev Enterprise (self-hosted).
• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example:

  $ tsh login --proxy=teleport.example.com --user=email@example.com
  $ tctl status
  # Cluster  teleport.example.com
  # Version  17.0.0-dev
  # CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
• A Google Cloud account.

Compatibility Warning

Teleport Cloud and Teleport Open Source do not currently support HSMs or Key Management Services.

Step 1/5. Create a key ring in GCP

Each Teleport Auth Server will need to be configured to use a GCP key ring which will hold all keys generated and used by that Auth Server. If running a High-Availability Teleport cluster with two or more Auth Servers, every Auth Server can be configured to use the same key ring, or if desired each can be configured to use a unique key ring in a different region (for redundancy or to decrease latency).

It is recommended to create a dedicated key ring for use by Teleport to logically separate it from any other keys in your cloud account. Choose a supported KMS location for the key ring which is geographically near to your Teleport Auth Servers.

You can create a key ring from the Google Cloud Console or from the gcloud CLI tool. Follow this guide or run the following command if you have the gcloud CLI configured:

$ gcloud kms keyrings create "teleport-keyring" --location location

Step 2/5. Create a GCP service account

Teleport needs permissions to create, list, destroy, sign with, and view KMS keys in your key ring. Start by creating the following custom IAM role.

# teleport_kms_role.yaml
title: teleport_kms_role
description: 'Teleport permissions for using KMS keys'
stage: ALPHA
includedPermissions:
- cloudkms.cryptoKeys.create
- cloudkms.cryptoKeys.list
- cloudkms.cryptoKeyVersions.create
- cloudkms.cryptoKeyVersions.destroy
- cloudkms.cryptoKeyVersions.useToSign
- cloudkms.cryptoKeyVersions.viewPublicKey

$ gcloud iam roles create teleport_kms_role \
    --project GCP-Project-ID \
    --file teleport_kms_role.yaml \
    --format yaml

Note the name field in the output which is the fully qualified name for the custom role and must be used in later steps.

$ export IAM_ROLE=<role name output from above>

If you don't already have a GCP service account for your Teleport Auth Server you can create one with the following command, otherwise use your existing service account.

$ gcloud iam service-accounts create teleport-auth-server \
    --description="Service account for Teleport Auth Server" \
    --display-name="Teleport Auth Server" \
    --format=yaml

Note the email field in the output, this must be used as the identifier for the service account.

$ export SERVICE_ACCOUNT=<email output from above command>

Create the IAM policy binding to grant the role to the service account for this keyring.

$ gcloud kms keyrings add-iam-policy-binding teleport-keyring \
    --location location \
    --member "serviceAccount:${SERVICE_ACCOUNT}" \
    --role "${IAM_ROLE}"

warning

Be aware that anyone with access to this service account will be able to create signatures as if they are the Teleport CA. It should be considered highly privileged and access should be restricted as much as possible.

Step 3/5. Provide the service account credentials to your Auth Server

The Teleport Auth Server will use Application Default Credentials to make requests to the GCP KMS service. Provide credentials for the teleport-auth-server service account created in step 2 to the Application Default Credentials of the environment you are running your Teleport Auth Server in. Supported environments include GCE VMs, GKE pods, and others.

See the GCP docs for Application Default Credentials to learn how to provide them for your preferred environment.

Manually checking permissions

To make sure the credentials have been configured correctly, you can run the gcloud CLI tool from your Teleport Auth Server's environment. Some example commands you could use to debug are listed here:

$ gcloud kms keys list --location location --keyring "teleport-keyring"
Listed 0 items.
$ gcloud kms keys create --location location --keyring "teleport-keyring" \
    --purpose asymmetric-signing \
    --default-algorithm rsa-sign-pkcs1-4096-sha512 \
    test-key
$ gcloud kms keys list --location location --keyring "teleport-keyring"
NAME                                                                                   PURPOSE          ALGORITHM                   PROTECTION_LEVEL
projects/my-gcp-account/locations/global/keyRings/teleport-keyring/cryptoKeys/test-key ASYMMETRIC_SIGN  RSA_SIGN_PKCS1_4096_SHA512  SOFTWARE
$ echo hello > /tmp/hello.txt
$ gcloud kms asymmetric-sign --keyring "teleport-keyring" --location location \
    --key "test-key" --version 1 \
    --input-file /tmp/hello.txt --signature-file /tmp/hello.sig
$ gcloud kms keys versions destroy --keyring "teleport-keyring" --location location --key "test-key" 1

Step 4/5. Configure your Auth Server to use KMS keys

CA key parameters are statically configured in the teleport.yaml configuration file of the Teleport Auth Server(s) in your cluster.

Find the fully qualified name of the KMS key ring you created in step 1 in the GCP Console or by running:

$ gcloud kms keyrings list --location location

Choosing a protection level

Supported KMS protection levels are SOFTWARE and HSM. If you choose SOFTWARE, GCP KMS will performs all cryptographic operations in software (Teleport performs no cryptographic operations). If you choose HSM, GCP KMS will perform all cryptographic operations in a Hardware Security Module.

Both protection levels are considered secure by Google and Teleport, you should evaluate your own organization's requirements and security policies when making your decision.

One relevant difference is the usage quotas available to keys of each protection level. At the time of writing, software keys have a project-wide quota of 60k cryptographic operations per minute, while asymmetric HSM keys have a quota of 3k operations per minute. See the KMS docs for updated numbers. If your cluster will have many thousands of hosts or active users, the higher quota software keys have may help to avoid any potential throttling, especially during CA rotations where many new certificates must be signed.

Include the following ca_key_params configuration in the auth_service section.

# /etc/teleport.yaml
auth_service:
  # ...
  ca_key_params:
    gcp_kms:
      keyring: "projects/<your-gcp-project>/locations/<location>/keyRings/<your-teleport-keyring>"
      protection_level: "SOFTWARE"

If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in GCP and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to GCP KMS keys, read on to migrating an existing cluster.

Step 5/5. Make sure everything is working

After starting up your Auth Server with the gcp_kms configuration, you can confirm that Teleport has generated keys in your keyring in the GCP Console or by running

$ gcloud kms keys list --keyring "teleport-keyring" --location location

Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.

tctl status should also show GCP KMS as the storage method for all Certificate Authority keys.

Migrating an existing cluster

If you have an existing Teleport cluster it will have already created software CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.

In order for the Teleport Auth Service to generate new keys in GCP KMS and have them trusted by the rest of the cluster, you will need to rotate all of your Teleport CAs.

teleport start will print a warning during startup if any CA needs to be rotated. Any users allowed the update verb for cert_authority resources in any of their Teleport roles will also see a cluster alert reminding them to rotate the CAs.

CA rotation can be performed manually or semi-automatically, see our admin guide on Certificate rotation. All CAs listed in the output of tctl status must be rotated.