Skip to main content

Deploying a production-grade Consul on GKE using Terraform Helm provider

By July 15, 2021No Comments

Deploying a production-grade Consul on GKE using Terraform Helm provider

Deploying a production-grade Consul on GKE using Terraform Helm provider

In this blog we will show you how to use Terraform Helm provider to deploy Consul on Google Kubernetes Engine using Hashicorp’s official Helm chart. Consul will be installed with TLS and Gossip key encryption. Helm charts are used for packaging applications on Kubernetes. These charts can be installed using Helm tool but for this blog we will use Terraform for provisioning.

I’m assuming a cluster is already deployed in GKE. First we will prepare self-signed TLS certificates and keys for the use of Consul. We are using self-signed certs here to keep things simple, however in production you will likely want to load your own certificates.

#### Generate self-signed TLS certificates for Consul  resource "tls_private_key" "consul-ca" {   algorithm = "RSA"   rsa_bits  = "2048" }  resource "tls_self_signed_cert" "consul-ca" {   key_algorithm   = tls_private_key.consul-ca.algorithm   private_key_pem = tls_private_key.consul-ca.private_key_pem    subject {     common_name  = "consul-ca.local"     organization = "Arctiq"   }    validity_period_hours = 8760   is_ca_certificate     = true    allowed_uses = [     "cert_signing",     "digital_signature",     "key_encipherment",   ] }  # Create the Consul certificates resource "tls_private_key" "consul" {   algorithm = "RSA"   rsa_bits  = "2048" }  # Create the request to sign the cert with our CA resource "tls_cert_request" "consul-req" {   key_algorithm   = tls_private_key.consul.algorithm   private_key_pem = tls_private_key.consul.private_key_pem    dns_names = [     "consul",     "consul.local",     "consul.default.svc.cluster.local",     "server.dc1.consul",   ]    ip_addresses = [     google_compute_address.vault.address,   ]    subject {     common_name  = "consul.local"     organization = "Arctiq"   } }  # Now sign the cert resource "tls_locally_signed_cert" "consul" {   cert_request_pem = tls_cert_request.consul-req.cert_request_pem    ca_key_algorithm   = tls_private_key.consul-ca.algorithm   ca_private_key_pem = tls_private_key.consul-ca.private_key_pem   ca_cert_pem        = tls_self_signed_cert.consul-ca.cert_pem    validity_period_hours = 8760    allowed_uses = [     "cert_signing",     "client_auth",     "digital_signature",     "key_encipherment",     "server_auth",   ] }

Next we will store these certificate and keys as Kubernetes secret and later we will expose them to Consul.

resource "kubernetes_secret" "consul_certs" {   metadata {     name      = "consul-certs"     namespace =   }    data = {     "ca.pem"         = tls_self_signed_cert.consul-ca.cert_pem     "consul.pem"     = tls_locally_signed_cert.consul.cert_pem     "consul-key.pem" = tls_private_key.consul.private_key_pem   }    type = "Opaque" }

Now we will generate a random key for Gossip encryption and will store it as Kubernetes secret as well.

resource "random_id" "consul_encrypt" {   byte_length = 16 }  resource "kubernetes_secret" "consul_gossip_key" {   metadata {     name      = "consul-gossip-key"     namespace =   }    data = {     gossipkey = random_id.consul_encrypt.b64_std   }    type = "Opaque" }

At this stage certificates and keys are ready for TLS and Gossip encryption. Next, we will create a separate service account for the use of Helm and apply cluster-admin role to the new SA, the reason for this is the default service account used by Terraform has some permission issues which makes the Helm chart fail.

provider "kubernetes" {   load_config_file = false   host             = google_container_cluster.default.endpoint    cluster_ca_certificate = base64decode(     google_container_cluster.default.master_auth[0].cluster_ca_certificate,   )   token = data.google_client_config.current.access_token }  resource "kubernetes_namespace" "consul" {   metadata {     name = "consul"   } } resource "kubernetes_service_account" "helm_account" {   depends_on = [     "google_container_cluster.default",   ]   metadata {     name      = var.helm_account_name     namespace = "kube-system"   } }  resource "kubernetes_cluster_role_binding" "helm_role_binding" {   metadata {     name =   }   role_ref {     api_group = ""     kind      = "ClusterRole"     name      = "cluster-admin"   }   subject {     api_group = ""     kind      = "ServiceAccount"     name      =     namespace = "kube-system"   }    provisioner "local-exec" {     command = "sleep 15"   } }  provider "helm" {   install_tiller = true   tiller_image = "${var.helm_version}"   service_account =    kubernetes {     host                   = google_container_cluster.default.endpoint     token                  = data.google_client_config.current.access_token     client_certificate     = "${base64decode(google_container_cluster.default.master_auth.0.client_certificate)}"     client_key             = "${base64decode(google_container_cluster.default.master_auth.0.client_key)}"     cluster_ca_certificate = "${base64decode(google_container_cluster.default.master_auth.0.cluster_ca_certificate)}"   } }

Helm is now all setup, download Consul Helm chart here in an appropriate location. We will now customize Consul Helm chart as per the requirements.

resource "helm_release" "consul" {   name      = "backend"   chart     = "../helm/consul-helm"   namespace =    set {     name  = "global.bootstrapACLs"     value = "true"   }    set {     name  = "server.connect"     value = "true"   }    set {     name  = "server.replicas"     value = var.num_consul_pods   }    set {     name  = "server.bootstrapExpect"     value = var.num_consul_pods   }    set {     name  = "connectInject.enabled"     value = "true"   }    set {     name  = "client.grpc"     value = "true"   }    values = ["${file("consul_values.yaml")}"]    depends_on = [kubernetes_cluster_role_binding.helm_role_binding] }

We have set some of the parameters inside the resource definition, description for each of these parameters is available in the default values.yaml file that comes with the Consul Helm chart. The remaining parameters are defined in a separate file consul_values.yaml.

global:   enablePodSecurityPolicies: true   gossipEncryption:     secretName: "consul-gossip-key"     secretKey: "gossipkey" dns:   enabled: true ui:   enabled: true server:   storage: 20Gi   storageClass: standard   extraVolumes:     - type: secret       name: consul-certs       load: false     - type: secret       name: consul-gossip-key       load: false   extraConfig: |     {         "ca_file": "/consul/userconfig/consul-certs/ca.pem",         "cert_file": "/consul/userconfig/consul-certs/consul.pem",         "key_file": "/consul/userconfig/consul-certs/consul-key.pem",         "ports": {           "https": 8501         },         "performance": {           "raft_multiplier": 1         },         "log_level": "INFO",         "verify_incoming": true,         "verify_outgoing": true,         "verify_server_hostname": true,         "disable_keyring_file": true,         "disable_host_node_id": true     } client:   grpc: true   extraVolumes:     - type: secret       name: consul-certs       load: false     - type: secret       name: consul-gossip-key       load: false   extraConfig: |     {         "ca_file": "/consul/userconfig/consul-certs/ca.pem",         "cert_file": "/consul/userconfig/consul-certs/consul.pem",         "key_file": "/consul/userconfig/consul-certs/consul-key.pem",         "ports": {           "https": 8501         },         "log_level": "INFO",         "verify_incoming": true,         "verify_outgoing": true,         "verify_server_hostname": true,         "disable_keyring_file": true,         "disable_host_node_id": true     }

In the above config we have defined "extraVolumes" which are set to the Kubernetes secrets defined in the start for TLS and Gossip encryption. These volumes are exposed to Consul in the path ‘/consul/userconfig/’.

Now the Terraform code is ready to deploy, use terraform plan command and review the all the planned changes and then issue terraform apply to deploy Consul. After Terraform is finished making all the changes kubectl can be used to review the deployment.

$ kubectl get all -n consul
NAME                                                                  READY   STATUS      RESTARTS   AGE pod/backend-consul-connect-injector-webhook-deployment-f88dfb7zgh7s   1/1     Running     0          2m55s pod/backend-consul-jwpmt                                              1/1     Running     0          2m55s pod/backend-consul-qqwlp                                              1/1     Running     0          2m55s pod/backend-consul-server-0                                           1/1     Running     0          2m55s pod/backend-consul-server-1                                           1/1     Running     0          2m55s pod/backend-consul-server-2                                           1/1     Running     0          2m55s pod/backend-consul-server-acl-init-j7hlj                              0/1     Completed   0          2m55s pod/backend-consul-sfhns                                              1/1     Running     0          2m55s  NAME                                          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                                                   AGE service/backend-consul-connect-injector-svc   ClusterIP   <none>        443/TCP                                                                   2m55s service/backend-consul-dns                    ClusterIP   <none>        53/TCP,53/UDP                                                             2m55s service/backend-consul-server                 ClusterIP   None          <none>        8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP   2m55s service/backend-consul-ui                     ClusterIP   <none>        80/TCP                                                                    2m55s  NAME                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE daemonset.apps/backend-consul   3         3         3       3            3           <none>          2m55s  NAME                                                                 READY   UP-TO-DATE   AVAILABLE   AGE deployment.apps/backend-consul-connect-injector-webhook-deployment   1/1     1            1           2m55s  NAME                                                                           DESIRED   CURRENT   READY   AGE replicaset.apps/backend-consul-connect-injector-webhook-deployment-f88dfb76c   1         1         1       2m56s  NAME                                     READY   AGE statefulset.apps/backend-consul-server   3/3     2m56s  NAME                                       COMPLETIONS   DURATION   AGE job.batch/backend-consul-server-acl-init   1/1           35s        2m56s

Consul members status can also be checked using below mentioned command.

$ kubectl exec backend-consul-server-0 -n consul -- consul members
Node                                  Address          Status  Type    Build  Protocol  DC   Segment backend-consul-server-0        alive   server  1.6.2  2         dc1  <all> backend-consul-server-1        alive   server  1.6.2  2         dc1  <all> backend-consul-server-2       alive   server  1.6.2  2         dc1  <all> gke-vault-default-pool-4f36e068-fkvg   alive   client  1.6.2  2         dc1  <default> gke-vault-default-pool-598f6b87-hl4m   alive   client  1.6.2  2         dc1  <default> gke-vault-default-pool-eb2ea058-sjds   alive   client  1.6.2  2         dc1  <default>

If Helm tool is installed locally then the status of newly deployed Helm chart can also be verified using below mentioned command.

$ helm list
NAME    REVISION    UPDATED                     STATUS      CHART           APP VERSION NAMESPACE backend 1           Thu Jan  9 15:43:04 2020    DEPLOYED    consul-0.12.0               consul   

Helm makes it really easy to deploy and manage applications on Kubernetes as we just observed in the case of Consul deployment. In order to make this deployment production-grade we have made Consul communication secure by enabling end-to-end TLS for RPC and symmetric key encryption for Gossip protocol. This makes it possible to run Consul on untrusted networks and other shared hosting providers. We have also enabled ACLs with default deny to secure the UI, API, CLI, service and agent communications.