End to End TLS for Azure Front Door and Azure Kubernetes Service

Introduction

Whilst exploring options for exposing Azure Kubernetes Service (AKS) container services in public behind Web Application Firewall (WAF), I was able to find many references on how to accomplish end to end TLS encrypted connections between Azure Application Gateway and AKS (specifically Application Gateway Ingress Controller, AGIC), but not with Azure Front Door (AFD).

In this post, I will share how I achieved end to end TLS connectivity between AFD and AKS, including high level design, issue, resolution and optimised Azure cost.

Design

In a nutshell, the below diagram represents the high level overview regarding what I wanted to achieve:

  • Custom frontend domains to expose (they don’t exist, just examples):
  • Backend pools to be the Azure Load Balancer acting as an ingress controller for the AKS cluster:
    • Backend host header to be null so that the request hostname determines this value.
      For instance, the requests in curl will be:
      • curl -vvv -H "Host: web.ssbkang.com" "LB IP Address"
      • curl -vvv -H "Host: api.ssbkang.com" "LB IP Address"
    • HTTPS health probe hitting /healthz with HEAD protocol
  • 1 x routing rule to be (apply URL rewrite and/or caching if required):
    • Accepted frontend protocol: HTTPS only
    • Forwarding protocol: HTTPS only
      Note: Implementing only one routing rule is to minimise the Azure cost by having only one routing rule but can scale out to multiple frontends / backend pools

Issue

I was expecting everything to work smoothly, however, faced the following from the browser:

Our services aren't available right now. 
We're working to restore all services as soon as possible. Please check back soon.

The first impression was that it sounded like the AFD instance was not fully deployed, but when I digged into it further, I managed to determine that the actual issue was related to the TLS certificate at the ingress controller level. I deployed the ingress controller using the official NGINX Helm Chart and by default, it leverages a self-signed certificate with the subject name ingress.local. Hence, when the AFD tried health probing at the backend (/healthz), it must have been returning bad request due to the fact that it does not match the certificate subject name.

Resolution

To resolve the above issue, there were two elements had to be amended; ingress controller and a public DNS entry for it.

I’ve initially deployed the ingress controller as a Helm chart leveraging FluxCD and whilst Googling, I managed to find an extra argument called default-ssl-certificate. This will automatically assign a TLS certificate if tls is missing from ingress manifests.

First of all, I used openssl to extract a crt file and a key file from a pfx certificate:

openssl pkcs12 -in wildcard.pfx -clcerts -nokeys -out tls.crt
openssl pkcs12 -in wildcard.pfx -nocerts -nodes -out tls.key

And then created a secret manifest in my Flux repository:

apiVersion: v1
kind: Secret
metadata:
  name: public-ingress-tls
  namespace: public-ingress
type: tls
data:
  tls.crt: {CRT}
  tls.key: {KEY}

Then I have updated the HelmRelease with the default-ssl-certificate argument as below:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: public-nginx-ingress
  namespace: public-ingress
spec:
  releaseName: public-nginx-ingress
  targetNamespace: public-ingress
  chart:
    repository: https://kubernetes-charts.storage.googleapis.com/
    name: nginx-ingress
    version: 1.39.0
  values:
    controller:
      ingressClass: public-nginx-ingress
      useIngressClassOnly: true
      replicaCount: 3
      nodeSelector.beta.kubernetes.io/os: "linux"
      service:
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "false"
      extraArgs:
        default-ssl-certificate: "public-ingress/public-ingress-tls"
    defaultBackend:
      nodeSelector.beta.kubernetes.io/os: "linux"

Finally, I have added a public DNS for my public ingress controller aks-public-ingress.ssbkang.com and updated the backend accordingly.

After the DNS registration has finished, wallah, everything started working, magic šŸ˜€

Conclusion

In this blog post, I have discussed how to achieve end to end TLS encrypted connectivity between AFD and AKS ingress controller including how to optimise Azure cost at the AFD level.

AFD in fact support TLS terminations, however, as the ingress controller cannot be located in a private VNet i.e. a public instance, it is highly recommended to implement end to end TLS encryption.
Also, another security layer to consider is to update the NSG attached to the AKS subnet to be only accepting HTTP/HTTPS traffics from the AFD instance only (can use the service tag called AzureFrontDoor.Backend). This way, no one will be able to bypass AFD hitting the ingress controller directly.

Hope this helped and if you have any questions or require clarifications, leave a comment šŸ™‚

Azure – Intra Subscription Azure SQL Database Migration Tip

Introduction

First blog for the year! Itā€™s been extremely busy but it means that Iā€™ve got lots of things to share.

In this blog post, I will be sharingĀ efficient way of migrating (copying) a Azure database from one subscription to another.

Letā€™s get started!

Traditional Way

Previous to ARM model, and even ARM was introduced, I believe lots of you would still be using the following method:

  • Make a copy from aĀ production database
  • Export the copied databaseĀ to a blob storage
  • Import the .bacpac to the destination SQL server
  • Delete the database copy
  • Delete the .bacpac file

This method is completely fine, but there are some cons around this:

  • Export & import process takes long time which ends up incurring extra operational cost
  • Possibility of leaving the .bacpac file back in the blob storage (if not an automated way)
  • You need to know the administrator login details for both source and destination SQL servers

New Way

From the new ARM portal, it allows you to migrate a resource to a resource group in different subscription.
Using this functionality, the following approach could be made:

  • Create a temporary SQL server in theĀ resource group where aĀ production SQL server is running
  • Make a copy of the production database to the temporary SQL server
  • Move the temporary SQL server to another subscription
  • Make a copy of the copied database to the destination SQL server
  • Delete the temporary SQL server

What are the benefits?

Pretty much reverse to whatā€™s mentioned above,Ā cons for traditional way.
One of theĀ main benefits is ā€œreduced operational costā€ – it doesnā€™t require a copied database to stand up for export & import processes.

Also, no additional space required in blob storage to hold the .bacpac file.

Hope this helps and feel free to leave a comment for any questions or clarifications šŸ™‚