This page describes how to manage TLS certificates in a CockroachDB operator deployment.
The CockroachDB operator is in Preview.
TLS configuration options
You can either allow the operator to generate self-signed certificates, provide a custom CA certificate and generate all other certificates, or use your own certificates.
All self-signed certificates
By default, the certificates are created automatically by a self-signer utility, which requires no configuration beyond setting a custom certificate duration if desired. This utility uses cockroach cert to automatically generate self-signed certificates for the nodes and root client which are stored in a secret. You can see these certificates by running kubectl get secrets:
kubectl get secrets
crdb-cockroachdb-ca-secret Opaque 2 23s
crdb-cockroachdb-client-secret kubernetes.io/tls 3 22s
crdb-cockroachdb-node-secret kubernetes.io/tls 3 23s
If you are deploying on OpenShift you must also set cockroachdb.tls.selfSigner.securityContext.enabled to false to mitigate stricter security policies.
Custom CA certificate
If you wish to supply your own CA certificates to the deployed nodes but allow automatic generation of client certificates, create a Kubernetes secret with the custom CA certificate. To perform these steps using the cockroach cert command:
mkdir certs
mkdir my-safe-directory
cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
Set cockroachdb.tls.selfSigner.caProvided to true and specify the secret where the certificate is stored:
cockroachdb:
tls:
enabled: true
selfSigner:
enabled: true
caProvided: true
caSecret: {ca-secret-name}
If you are deploying on OpenShift you must also set cockroachdb.tls.selfSigner.securityContext.enabled to false to mitigate stricter security policies.
All custom certificates
Set up your certificates and load them into your Kubernetes cluster as secrets using the following commands:
mkdir certs
mkdir my-safe-directory
cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key
kubectl create secret generic cockroachdb-root --from-file=certs
secret/cockroachdb-root created
cockroach cert create-node --certs-dir=certs --ca-key=my-safe-directory/ca.key localhost 127.0.0.1 my-release-cockroachdb-public my-release-cockroachdb-public.cockroach-ns my-release-cockroachdb-public.cockroach-ns.svc.cluster.local *.my-release-cockroachdb *.my-release-cockroachdb.cockroach-ns *.my-release-cockroachdb.cockroach-ns.svc.cluster.local
kubectl create secret generic cockroachdb-node --from-file=certs
secret/cockroachdb-node created
The subject alternative names are based on a release called my-release in the cockroach-ns namespace. Make sure they match the services created with the release during Helm install.
To optionally supply certificates with cert-manager, set cockroachdb.tls.certManager.enabled to true, and cockroachdb.tls.certManager.issuer to an IssuerRef (as they appear in certificate resources) pointing to a clusterIssuer or issuer that you have set up in the cluster:
cockroachdb:
tls:
enabled: true
certManager:
enabled: true
caConfigMap: cockroachdb-ca
nodeSecret: cockroachdb-node
clientRootSecret: cockroachdb-root
issuer:
group: cert-manager.io
kind: Issuer
name: cockroachdb-cert-issuer
clientCertDuration: 672h
clientCertExpiryWindow: 48h
nodeCertDuration: 8760h
nodeCertExpiryWindow: 168h
The following Kubernetes application describes an example issuer, including the trust manager to automatically copy the CA certificate from a secret to a ConfigMap:
apiVersion: v1
kind: Secret
metadata:
name: cockroachdb-ca
namespace: cockroach-ns
data:
tls.crt: [BASE64 Encoded ca.crt]
tls.key: [BASE64 Encoded ca.key]
type: kubernetes.io/tls
---
apiVersion: cert-manager.io/v1alpha3
kind: Issuer
metadata:
name: cockroachdb-cert-issuer
namespace: cockroach-ns
spec:
ca:
secretName: cockroachdb-ca
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
name: cockroachdb-ca
spec:
sources:
- secret:
name: cockroachdb-ca
key: tls.crt
target:
configMap:
key: ca.crt
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: cockroachdb
If your certificates are stored in TLS secrets, such as secrets generated by cert-manager, the secret will contain files named: ca.crt, tls.crt, and tls.key. For CockroachDB, rename these files as applicable to match the following naming scheme: ca.crt, node.crt, node.key, client.root.crt, and client.root.key.
Add the following to the values file:
cockroachdb:
tls:
enabled: true
externalCertificates:
enabled: true
certificates:
nodeSecretName: {node_secret_name}
nodeClientSecretName: {client_secret_name}
Replace the following placeholder values:
- {node_secret_name}: The name of the Kubernetes secret that contains the generated client certificate and key.
- {client_secret_name}: The name of the Kubernetes secret that contains the generated node certificate and key.
Example: Generate and sign custom certificates using cockroach cert
The following example uses cockroach cert commands to generate and sign the CockroachDB node and client certificates. To learn more about the supported methods of signing certificates, refer to Authentication.
Create two directories:
mkdir certs my-safe-directoryCreate the CA certificate and key pair:
cockroach cert create-ca \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.keyCreate a client certificate and key pair for the root user:
cockroach cert create-client root \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.keyUpload the client certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the CockroachDB operator:
kubectl create secret generic cockroachdb.client.root \ --from-file=tls.key=certs/client.root.key \ --from-file=tls.crt=certs/client.root.crt \ --from-file=ca.crt=certs/ca.crtsecret/cockroachdb.client.root createdCreate the certificate and key pair for your CockroachDB nodes, specifying the namespace you used when deploying the cluster. This example uses the
cockroach-nsnamespace:cockroach cert create-node localhost \ 127.0.0.1 \ cockroachdb-public \ cockroachdb-public.cockroach-ns \ cockroachdb-public.cockroach-ns.svc.cluster.local \ *.cockroachdb \ *.cockroachdb.cockroach-ns \ *.cockroachdb.cockroach-ns.svc.cluster.local \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.keyUpload the node certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the CockroachDB operator:
kubectl create secret generic cockroachdb.node \ --from-file=tls.key=certs/node.key \ --from-file=tls.crt=certs/node.crt \ --from-file=ca.crt=certs/ca.crtsecret/cockroachdb.node createdCheck that the secrets were created on the cluster:
kubectl get secretsNAME TYPE DATA AGE cockroachdb.client.root Opaque 3 13s cockroachdb.node Opaque 3 3s default-token-6js7b kubernetes.io/service-account-token 3 9hAdd
cockroachdb.tls.externalCertificates.certificates.nodeSecretNameandcockroachdb.tls.externalCertificates.certificates.nodeClientSecretNameto the values file used to deploy the cluster:cockroachdb: tls: enabled: true externalCertificates: enabled: true certificates: nodeSecretName: cockroachdb.node nodeClientSecretName: cockroachdb.client.root
Rotate security certificates
You may need to rotate the node, client, or CA certificates in the following scenarios:
- The node, client, or CA certificates are expiring soon.
- Your organization's compliance policy requires periodic certificate rotation.
- The key (for a node, client, or CA) is compromised.
- You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates.
Certificates generated by the self-signer utility are re-generated by the utility when the expiry time is near. Certificates managed with cert-manager are also automatically renewed. Whether the new certificates are generated automatically or manually, the CockroachDB pods must be restarted in order to pick up the new certificates.
Example: Manually rotate certificates signed with cockroach cert
If you previously created and signed certificates with cockroach cert, follow these steps to manually rotate the certificates using the same CA:
Create a new client certificate and key pair for the root user, overwriting the previous certificate and key:
cockroach cert create-client root \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key \ --overwriteUpload the new client certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the CockroachDB operator:
kubectl create secret generic cockroachdb.client.root.2 \ --from-file=tls.key=certs/client.root.key \ --from-file=tls.crt=certs/client.root.crt \ --from-file=ca.crt=certs/ca.crtsecret/cockroachdb.client.root.2 createdCreate a new certificate and key pair for your CockroachDB nodes, overwriting the previous certificate and key. Specify the namespace you used when deploying the cluster. This example uses the
cockroach-nsnamespace:cockroach cert create-node localhost \ 127.0.0.1 \ cockroachdb-public \ cockroachdb-public.cockroach-ns \ cockroachdb-public.cockroach-ns.svc.cluster.local \ *.cockroachdb \ *.cockroachdb.cockroach-ns \ *.cockroachdb.cockroach-ns.svc.cluster.local \ --certs-dir=certs \ --ca-key=my-safe-directory/ca.key \ --overwriteUpload the new node certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the CockroachDB operator:
kubectl create secret generic cockroachdb.node.2 \ --from-file=tls.key=certs/node.key \ --from-file=tls.crt=certs/node.crt \ --from-file=ca.crt=certs/ca.crtsecret/cockroachdb.node.2 createdAdd
cockroachdb.tls.externalCertificates.certificates.nodeClientSecretNameandcockroachdb.tls.externalCertificates.certificates.nodeSecretNameto the values file used to deploy the cluster:cockroachdb: tls: externalCertificates: enabled: true certificates: nodeClientSecretName: "cockroachdb.client.root.2" nodeSecretName: "cockroachdb.node.2"Check that the secrets were created on the cluster:
kubectl get secretsNAME TYPE DATA AGE cockroachdb.client.root.2 Opaque 3 4s cockroachdb.node.2 Opaque 3 1s default-token-6js7b kubernetes.io/service-account-token 3 9hNote:Remember that
nodeSecretNameandnodeClientSecretNamein the operator configuration must specify these secret names. For details, see the deployment guide.Apply the new settings to the cluster:
helm upgrade --reuse-values $CRDBCLUSTER ./cockroachdb-parent/charts/cockroachdb --values ./cockroachdb-parent/charts/cockroachdb/values.yaml -n $NAMESPACEThe pods will terminate and restart one at a time, using the new certificates. You can observe this process:
kubectl get podsNAME READY STATUS RESTARTS AGE cockroach-operator-655fbf7847-lvz6x 1/1 Running 0 4h29m cockroachdb-0 1/1 Running 0 4h16m cockroachdb-1 1/1 Terminating 0 4h16m cockroachdb-2 1/1 Running 0 43sDelete the existing client secret that is no longer in use:
kubectl delete secret cockroachdb.client.rootsecret "cockroachdb.client.root" deletedDelete the existing node secret that is no longer in use:
kubectl delete secret cockroachdb.nodesecret "cockroachdb.node" deleted
Secure the webhooks
The operator ships with both mutating and validating webhooks. Communication between the Kubernetes API server and the webhook service must be secured with TLS.
By default, the CockroachDB operator searches for the TLS secret cockroach-operator-certs, which contains a CA certificate. If the secret is not found, the operator auto-generates cockroach-operator-certs with a CA certificate for future runs.
The operator then generates a one-time server certificate for the webhook server that is signed with cockroach-operator-certs. Finally, the CA bundle for both mutating and validating webhook configurations is patched with the CA certificate.
You can also use your own certificate authority rather than cockroach-operator-certs. Both the certificate and key files you generate must be PEM-encoded. See the following example.
Example: Using OpenSSL to secure the webhooks
These steps demonstrate how to use the openssl genrsa and openssl req subcommands to secure the webhooks on a running Kubernetes cluster:
Generate a 4096-bit RSA private key:
openssl genrsa -out tls.key 4096Generate an X.509 certificate, valid for 10 years. You will be prompted for the certificate field values.
openssl req -x509 -new -nodes -key tls.key -sha256 -days 3650 -out tls.crtCreate the secret, making sure that you are in the correct namespace:
kubectl create secret tls cockroach-operator-certs --cert=tls.crt --key=tls.keysecret/cockroach-operator-certs createdRemove the certificate and key from your local environment:
rm tls.crt tls.keyRoll the operator deployment to ensure a new server certificate is generated:
kubectl rollout restart deploy/cockroach-operator-managerdeployment.apps/cockroach-operator-manager restarted
