Skip to main content
Uncategorized

How to use Vault PKI Engine for Dynamic TLS Certificates on GKE

By July 15, 2021No Comments

How to use Vault PKI Engine for Dynamic TLS Certificates on GKE

What is Vault?

I’ve talked about Vault in a couple of blogs before. So just to recap:

Vault is a tool for securely accessing secrets.Vault allows users to store, manage and control access to tokens, username password and database credentials. There are many secrets management tools out there but Vault has gained a lot of popularity thanks to it’s flexible API and providing encryption at rest and in flight.

There have been cases of applications leaking credentials through logs, diagnostic output, external logging to a logging stack. Vault can provide dynamic secrets which help provide short-lived ephemeral credentials to applications instead of long live credentials. With dynamic secrets, even if an application writes it to an external system, it is valid for a specific period of time.

I’m not going to get into how to install Vault but going to cover more around the integration with apps running on GKE.

Vault PKI Secrets Engine

Vault enables you to deal with a whole Public Key Infrastructure (PKI) to guarantee secure correspondence among various applications. This enables organizations to effectively setup their own certificate authority (CA), revoke or issue new certs using basic API calls.. and dumping the agonizing procedure of always creating self signed certificates.

In this guide, I am going to clarify how Vault PKI engine functions and how you can utilize it to make your very own Root CA.

Cert-Manager

Cert-manager is a fantastic open-source project which can be used to request dynamic TLS certificates from Vault, Letsencrypt etc. Cert-manager runs as a pod on GKE and grabs TLS names from ingress objects.

Sounds like a lot of moving parts! This guide assumes that you have a GKE cluster already built. Let’s get started:

Install Helm and Tiller

$ brew install kubernetes-helm (for MacOS) $ kubectl create serviceaccount tiller --namespace=kube-system $ kubectl create clusterrolebinding tiller-admin --serviceaccount=kube-system:tiller --clusterrole=cluster-admin $ helm init --service-account=tiller $ helm repo update

Configure NGINX ingress proxy

  • Install
$ helm install stable/nginx-ingress --name quickstart
  • Get the external IP of the Ingress proxy
$ kubectl get svc

Assign DNS for the app

At Arctiq’s production environment we use Terraform to dynamically update DNS for applications for lab and customer testing. I won’t get into too much about how to accomplish that with Terraform. It’s fairly straight forward by following Google’s and Hashicorp’s guides.

Assign the external IP from the previous step to your DNS name.

Deploy the application

  • Create the namespace. Let’s call it weather
$ kubectl create ns weather
  • Create the deployment for weather app
$ cat weather-deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata:   name: weather spec:   replicas: 1   template:     metadata:       labels:         app: weather     spec:       containers:       - image: https://res.cloudinary.com/arctiq/image/upload/q_auto/posts/gcr.io/container-secrets/weather:latest         imagePullPolicy: Always         name: weather         ports:         - containerPort: 8080

$ kubectl apply -f weather-deployment.yaml -n weather

  • Create the weather service
$ cat weather-svc.yaml apiVersion: v1 kind: Service metadata:   name: weather spec:   ports:   - port: 80     targetPort: 8080     protocol: TCP   selector:     app: weather

$ kubectl apply -f weather-deployment.yaml -n weather

  • Create the weather ingress
$ cat weather-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata:   name: weather   annotations:     kubernetes.io/ingress.class: "nginx" spec:   tls:   - hosts:     - gke.weather.weirdscience.labs.arctiq.ca     secretName: weather-tls   rules:   - host: gke.weather.weirdscience.labs.arctiq.ca     http:       paths:       - path: /         backend:           serviceName: weather           servicePort: 80

$ kubectl apply -f weather-ingress.yaml -n weather

  • Test the application

At this stage, open the weather application on the URL and it should give a TLS warning:

<img src="https://res.cloudinary.com/arctiq/image/upload/q_auto/posts/tlswarning.png" %}

Install cert-manager

$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml  $ helm repo update  $ helm install --name cert-manager --namespace cert-manager stable/cert-manager

Configure Vault PKI back end

  • Create policy

I won’t get into how to install Vault. Cert-manager requires issue and sign on the PKI back end:

path "pki/issue/cert-manager" {   capabilities = ["read", "list", "create", "update"] }  path "pki/sign/cert-manager" {   capabilities = ["read", "list", "create", "update"] }
  • Create approle for cert-manager

Save your role-id and secret-id:

$ vault write auth/approle/role/cert-manager-role policies=cert-manager-pki $ vault read auth/approle/role/cert-manager-role/role-id $ vault write -f auth/approle/role/cert-manager-role/secret-id
  • Enable Vault PKI backend to act as a private CA
$ vault secrets enable pki vault secrets tune -default-lease-ttl=2160h -max-lease-ttl=87600h pki $ vault write pki/root/generate/internal common_name=arctiq.ca ttl=87600h $ vault write pki/roles/cert-manager \ allowed_domains=weirdscience.labs.arctiq.ca \ allow_subdomains=true max_ttl=8760h
  • Create kubernetes secret containing secret-id (saved in the previous steps)
cat cert-manager-vault-approle.yaml  apiVersion: v1 kind: Secret type: Opaque metadata:   name: cert-manager-vault-approle   namespace: weather data:   secretId: "<base-64-encoded>" ### your secret-id
  • Create the vault issuer for cert-manager
$ cat issuer.yaml apiVersion: certmanager.k8s.io/v1alpha1 kind: Issuer metadata:   name: vault-issuer spec:   vault:     path: pki/sign/cert-manager     server: https://vault.weirdscience.labs.arctiq.ca:8200     auth:       appRole:         path: approle         roleId: "d37b79df-3d0f-f089-4121-be4c6887b367" ### your role-id         secretRef:           name: cert-manager-vault-approle           key: secretId
  • Create the certificate for cert-manager
$ cat certificate.yaml apiVersion: certmanager.k8s.io/v1alpha1 kind: Certificate metadata:   name: weather spec:   secretName: weather-tls ### should be the same as in ingress manifest   issuerRef:     name: vault-issuer   commonName: gke.weather.weirdscience.labs.arctiq.ca   dnsNames:   - www.gke.weather.weirdscience.labs.arctiq.ca

At this point, test if the weather-tls secret is dynamically created:

<img src="https://res.cloudinary.com/arctiq/image/upload/q_auto/posts/secrets.png" %}

Finally, test the app

Test the app by opening it in a new browser window.

<img src="https://res.cloudinary.com/arctiq/image/upload/q_auto/posts/workingcert.png" %}

Check the weather (securely)!

<img src="https://res.cloudinary.com/arctiq/image/upload/q_auto/posts/temperature.png" %}

And done! I have committed the code for the guide here

This demo was also recorded live at one of Arctiq’s event. A recap can be seen here.

Interested in learning more about this demo? Feel free to contact us by click on the link below

//take the first step