Notice: Our URL has changed! Please update your bookmarks. Dismiss

Configuring OpenShift to use Custom Certificates

Overview

OpenShift creates a lot of certificates by default. Understanding where and how those certificates are used will help to understand how it can be configured to meet your needs.

In this guide we will discuss how and where TLS Certificates are used in OpenShift, how they are created, and how to go about designing a proper certificate management strategy for OpenShift. We will then show you how to configure OpenShift via Ansible to reflect this strategy.

How OpenShift Utilizes Certificates for Internal Communication

OpenShift utilizes TLS Certificates in three primary ways.

  1. To encrypt traffic

  2. To authenticate certain infrastructure components

  3. To authenticate certain users

The table below is a breakdown of the various places in which certificates are used to do one of the things from the list above.

Table 1. OpenShift Certificate Usage
Source (Client) Destination (Server) TLS Type Notes

Nodes

Master API

Infrastructure Component Authenticating via X509 Client Certs

Node Service account have the format system:node:<hostname>

Master

Node (kubelet)

Raw Client Certificate Auth

system:admin User

Master API

Infrastructure Component Authenticating via X509 Client Certs

This is the default cluster admin user. Only accessible as root on a master host, as this is the only user that has read access to the client certificate for the account.

Router

Master API

Service Account API Token

Registry

Master API

Service Account API Token

Master controllers

Master API

Infrastructure Component Authenticating via X509 Client Certs

Multiple service accounts used here

Master

Etcd

Raw Client Certificate Auth

How OpenShift does all read/write to etcd

Etcd

Etcd

Raw Client Certificate Auth

Used to establish cluster membership and do data replication

User

Master API

API Token over HTTPS connection

Primary method which users use to authenticate to OpenShift Console/API

User

Router

HTTPS Connection

Primary method for front-end users to access Applications in OpenShift

User

Kibana (Logging)

API Token over HTTPS connection

Fluentd

ElasticSearch

Raw Client Certificate Auth

ElasticSearch

ElasticSearch

Raw Client Certificate Auth

Kibana

ElasticSearch

Raw Client Certificate Auth

User

Hawkular (Metrics)

API Token over HTTPS connection

User

Registry

API Token over HTTPS connection

User

Registry Console

API Token over HTTPS connection

Registry Console

Registry

Infrastructure Component Authenticating via X509 Client Certs

To make the provisioning process for all of these certificates more palatable, the Ansible Config Playbook that is used to automate the deployment and configuration of OpenShift automatically generates and signs all of these certificates. This is done from an internally signed CA (the openshift-signer). As of Kubernetes 1.7/OpenShift 3.7, this internally signed CA is ingrained into the Master API, and is used to as the mechanism by which new nodes join the cluster. This is convenient, as it makes the creation, signing, and configuring of these certificates essentially invisible to the administrator, and the install is as seamless as possible. However, when building long running, higher environment OpenShift clusters, it is often desired to have certificates signed by a trusted internal or public Certificate Authority presented to users. The Ansible Config Playbook supports the presentation of custom certificates on the edges of the cluster (e.g. Router, Registry, Master API, etc.). The details of this option is covered in the following section.

Designing a Certificate Approach for OpenShift

The suggested approach is to allow OpenShift Internals to use self-signed/generated certificates, and add a custom server certificate for each User-facing/PublicIP/PublicURL endpoint.

Important
In order to do the above, master cluster public hostname and master cluster hostname must be different. If they’re the same, the named certificates will fail and will require a re-install. See the Designing for DNS section of the Install Guide.

The Advantage of this approach is that it can be more flexible, as it allows you to start off relying completely on the self-signed certificates generated by OpenShift, and add-on custom trusted certificates to individual components as needed.

The downside to this approach is that the internal infrastructure certificates remain self-signed, which may be percieved as bad practice by some Security or PKI teams. In reality the risk here is minimal, as the only clients that are trusting these certificates would be other components within the Cluster, and all external users and systems would be presented with custom trusted certificates.

To see how to configure OpenShift to use this strategy, you can skip down to the Configuring OpenShift to use Component-specific Custom Certificates section.

Configuring OpenShift to use Component-specific Custom Certificates

Alternatively, you can allow OpenShift to use its own CA to generate the internal certificates (self-signed) with which all of the internal components of OpenShift will authenticate. Custom certificates may be created and configured to be used individually by User-facing components of openshift. Discussion of each component is below.

Master API Certificate

In order to facilitate trusted connections with external users of OpenShift, a “Named Certificate” can be provisioned which matches the domain name provided in openshift_master_cluster_public_hostname. This certificate must be placed in a directory accessible to Ansible, and added to the Ansible inventory file like so.

openshift_master_named_certificates=[{"certfile": "/path/to/console.ocp-c1.myorg.com.crt", "keyfile": "/path/to/console.ocp-c1.myorg.com.key", "names": ["console.ocp-c1.myorg.com"], "cafile": "/path/to/console.ocp-c1.myorg.com.ca.crt"}]

Default (Wildcard) Router Certificate

OpenShift’s Router may be configured with a default wildcard certificate, which can be used to provide a convenient way for applications deployed to the platform to take advantage of some level of out of the box encryption without being required to bring their own custom certificates to the table. This is generally a recommended practice, at least in a Non-Production scenario to encourage exploration, experimentation, and rapid development.

In order to configure a default wildcard certificate, a certificate must be provisioned that is valid for *.<app domain>, where <app domain> is the value of openshift_master_default_subdomain. Once provisioned, you will need to place your cert, key and ca cert files on your ansible host, and add the following line to your ansible inventory.

openshift_hosted_router_certificate={"certfile": "/path/to/apps.c1-ocp.myorg.com.crt", "keyfile": "/path/to/apps.c1-ocp.myorg.com.key", "cafile": "/path/to/apps.c1-ocp.myorg.com.ca.crt"}

Registry Certificate

OpenShift’s Image Registry is an internal service, whose primary use is to facilitate builds and deployments in OpenShift. Most of the communication with the registry is facilitated by internal components in OpenShift. As such, there should be no need to replace the certificate used by the Registry service itself. However, by default, the Registry will also be exposed by a Route to allow external systems and users the ability to docker login to the registry to do pulls and pushes of images. In order to avoid interacting with an internal, self-signed certificate, a Reencrypt route may be used with a custom certificate on it, which will be presented to external users. To configure this, simply add the following lines pointing to the cert you would like used for the registry route.

openshift_hosted_registry_routehost=registry.apps.c1-ocp.myorg.com
openshift_hosted_registry_routecertificates={"certfile": "/path/to/registry.apps.c1-ocp.myorg.com.crt", "keyfile": "/path/to/registry.apps.c1-ocp.myorg.com.key", "cafile": "/path/to/registry.apps.c1-ocp.myorg.com-ca.crt"}

Other components

For other components like Logging & Metrics, the approach for fronting services with custom certs can be found in Administrator Solutions for Certificate Management.

Run Ansible

Once your Ansible Inventory has been updated with the above, you can re-run the config playbook.

ansible-playbook -i c1-ocp.myorg.com/hosts /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

Additional Resources