Skip to main content
Version: 14.x

Teleport SSH Nodes

The SSH Node service​

The Teleport Node service is optional. You can use it to replace OpenSSH on your infrastructure. Here is why we recommend Teleport Node service instead of OpenSSH:

  • The node service supports BPF recording of all syscalls, network calls and files accessed during SSH session.
  • It can record terminal sessions.
  • It provides automatic registration, certificate and certificate authority rotation,
  • It can provision OS user and update sudoers files according to teleport roles.
  • You can connect nodes to proxies with outbound persistent tunnels, for your IoT lab or remote infrastructure.

Just like with OpenSSH, the node service provides SSH access to every node with any clients supporting client SSH certificates:

Joining Nodes​

A node candidate becomes a Teleport Node when it joins a cluster and authenticates itself to receive cluster certificate.

All cluster Nodes keep the Auth Server updated on their status with periodic ping messages. They report their IP addresses and the values of their assigned labels. Clients can access the list of all Nodes in their cluster via the Auth Server API or CLI.

Tip

Nodes can register with Auth servers directly, or use proxies to establish the connection to auth servers. The latter is helpful if you have multiple proxies and nodes all over the world.

SSH Host certificate​

Node's identity is represented by SSH host certificate it receives after registering withing the cluster:

Host certificate

This certificate contains information about the node including:

  • The host ID, a generated UUID unique to a node
  • A nodename, which defaults to hostname of the node, but can be configured.
  • The cluster_name, which defaults to the hostname of the auth server, but can be configured
  • The node role (i.e. node,proxy) encoded as a certificate extension
  • The cert Expiry time

A Teleport Cluster is a set of one or more machines whose certificates are signed by the same certificate authority (CA) operating in the Auth Server. A certificate is issued to a node when it joins the cluster for the first time.

Single-Node Clusters are Clusters

Once a Node gets a signed certificate from the Node CA, the Node is considered a member of the cluster, even if that cluster has only one node.

Connecting to Nodes​

Nodes support two modes - a standard mode and a reverse tunnel mode.

In standard mode, nodes act like OpenSSH servers that only accept client SSH certificates. Users can connect to nodes through Proxy as a jump-host or directly:

In reverse tunnel mode, nodes establish reverse tunnels back to proxy. Nodes do not bind to any interface on the host, making sure the connection is only possible through proxies:

tip

You can mix both modes in the same cluster, depending on your use case. For example, you can have several IOT devices joining the cluster via reverse tunnel and a large fleet of servers in the internal network using standard mode.

Cluster state​

Cluster state is stored in a central storage location configured by the Auth Server. Each node (or proxy) is stateless and holds no secrets such as keys or passwords.

The cluster state includes:

  • Node membership information and online/offline status for each node.
  • List of active sessions.
  • List of locally stored users.
  • RBAC configuration (roles and permissions).
  • Dynamic configuration.

SSH Session recording​

By default, nodes submit SSH session traffic to the Auth server for storage. These recorded sessions can be replayed later via tsh play command or in a web browser.

SSH node recording​

Some Teleport users assume that audit and session recording happen by default on the Teleport proxy server. This is not the case in default configuration because a proxy cannot see the encrypted traffic, it is encrypted end-to-end, i.e. from an SSH client to an SSH server/node, see the diagram below:

session-recording-diagram

Proxy recording mode​

In this mode, the proxy terminates (decrypts) the SSH connection using the certificate supplied by the client via SSH agent forwarding and then establishes its own SSH connection to the final destination server, effectively becoming an authorized "man in the middle". This allows the proxy server to forward SSH session data to the auth server to be recorded, as shown below:

recording-proxy

The recording proxy mode, although less secure, was added to allow Teleport users to enable session recording for OpenSSH's servers running sshd, which is helpful when gradually transitioning large server fleets to Teleport.

We consider the "recording proxy mode" to be less secure for two reasons:

  • It grants additional privileges to the Teleport proxy. In the default mode, the proxy stores no secrets and cannot "see" the decrypted data. This makes a proxy less critical to the security of the overall cluster. But if an attacker gains physical access to a proxy Node running in the "recording" mode, they will be able to see the decrypted traffic and client keys stored in the proxy's process memory.
  • Recording proxy mode requires SSH Agent Forwarding. Agent Forwarding is required because without it, a proxy will not be able to establish the 2nd connection to the destination Node.

However, there are advantages of proxy-based session recording too. When sessions are recorded at the Nodes, a root user can add iptables rules to prevent sessions logs from reaching the Auth Service. With sessions recorded at the proxy, users with root privileges on Nodes have no way of disabling the audit.

See the reference to learn how to turn on the recording proxy mode. Note that the recording mode is configured on the Auth Service.

More concepts​