Connecting to the Cluster
In this second part of the blog post (Part 1 is here), we are now ready to connect to the cluster instance. using the windows terminal I issued the following commands.
“az login” This command will open a new browser, you will need to type in your azure credentials.
|
1 |
az login |
“az account set –subscription <name or id of cluster subscription>” This command allows me to change to the subscription where the aks cluster is.
|
1 |
az account set --subscription <name or id> |
“az aks get-credentials –resource-group rg-aks-lza-radbase-sbx-uks-1 –name aksmbosbxuks1” Next this command connects to the cluster, replace with your cluster name and resource group.
|
1 |
az aks get-credentials --resource-group rg-aks-lza-radbase-sbx-uks-1 --name aksmbosbxuks1 |
“kubelogin convert-kubeconfig -l azurecli” This command basically sets the cluster kubelogin to use Entra ID authentication.
|
1 |
kubelogin convert-kubeconfig -l azurecli |
Enabling Key Vault Access
Next I used the below commands to verify that the containers are running that allow us access to the keyvault.
|
1 |
kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)' |
you should see an output like the below image:
Next I needed to enable the OIDC issuer so that My cluster can communicate using the Managed Identity to retrieve the SSL certificate.
first I issue the command to get the OIDC issuer URL: More information can be read from this MS link: https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer#update-an-aks-cluster-with-oidc-issuer
|
1 |
az aks show --resource-group rg-aks-lza-radbase-sbx-uks-1 --name aksmbosbxuks1 --query "oidcIssuerProfile.issuerUrl" -o tsv |
I then issue the following command to enable oidc on the cluster:
|
1 |
az identity federated-credential create --name aksfederatedidentity --identity-name "mid-aks-dev-uks-1" --resource-group "rg-aks-lza-radbase-sbx-uks-1" --issuer "ISSUER URL FROM PREVIOUS COMMAND HERE" --subject system:serviceaccount:default:workload-identity-sa |
Adding the SSL
You will need to make sure that you have added the Certificate to your Key Vault already. please make sure that you upload the full certificate which contains the root, intermediate and the domain certificate. I have a Blog post on how to do that at this link: Storing an SSL certificate in Azure Key Vault
Once you have done this you can then issue the below commands: This command gets the key vault name and the name of the certificate as it appears in KV.
|
1 2 |
$versionedSecretId= az keyvault certificate show -n Web1AksCertificate --vault-name keylzasecsdevbxuks1 --query "sid" -o tsv $unversionedSecretId=($versionedSecretId -split '/' | Select-Object -First 5) -join '/' |
Next we need to add this certificate to the application gateway this will be for the frontend, the same certificate is also used for the backend.
|
1 2 |
#Add the certificate to AppGw from the key vault az network application-gateway ssl-cert create -n frontend --gateway-name agw-lza-waf-sbx-uks-1 --resource-group rg-agw-lza-waf-sbx-uks-1 --key-vault-secret-id $unversionedSecretId |
Next we need to add the backend root certificate, again for this I just used the certificate that was provided to me by my SSL provider:
|
1 |
az network application-gateway root-cert create --gateway-name agw-lza-waf-sbx-uks-1 --resource-group rg-agw-lza-waf-sbx-uks-1 --name backendroot --cert-file "C:\dev\MySSLCert.crt" |
Deploy the YAML Files
OK we are now ready to deploy the YAML files to our AKS cluster.
Deploy the Azure Application Gateway
I used the following YAML file to deploy the Azure Application gateway.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: aspnetapp annotations: kubernetes.io/ingress.class: azure/application-gateway appgw.ingress.kubernetes.io/ssl-redirect: "true" appgw.ingress.kubernetes.io/backend-protocol: "https" appgw.ingress.kubernetes.io/appgw-ssl-certificate: frontend appgw.ingress.kubernetes.io/backend-hostname: "web1.sbx-aks.databasejoe.com" appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: "backendroot" spec: tls: - secretName: frontend hosts: - web1.sbx-aks.databasejoe.com rules: - host: web1.sbx-aks.databasejoe.com http: paths: - path: / backend: service: name: website-service port: number: 8443 pathType: Exact |
to do this from the terminal, I made sure that I was in the correct directory where my deployment scripts where. I then issues the following command.
|
1 |
kubectl apply -f ./azure-agw.yaml |
after running the above, I then issue the following command:
|
1 |
kubectl get pods -n kube-system |
the above command will get the running pods in the kube-system namespace. In the output that will be shown I am looking for the ingress pod, for me the pod was called:
ingress-appgw-deployment-76cd4d497f-9ls6g
I then issued the command:
|
1 |
kubectl logs ingress-appgw-deployment-76cd4d497f-9ls6g -n kube-sustem |
and I was presented with errors relating to permissions for a managed identity. Now you are likely asking yourself that we have already configured permissions for our managed identity, and you would be correct we did. But this is a different managed identity, this identity gets automatically created within the AKS cluster resource group. This RG is not the one where your instance is housed in but the RG where all the AKS cluster dependencies are created.
In this RG you will fine a Managed Identity that has been created for you, mine was called “ingressapplicationgateway-aksmbosbxuks1” It is this managed identity that we need to assign permissions to. Luckily the errors listed within the logs command, explain the commands to execute.
for me I needed to use the following commands:
|
1 2 3 4 5 6 7 |
az role assignment create --role "Managed Identity Operator" --assignee CLIENT_ID --scope /subscriptions/SUBSCRIPTION_ID/resourceGroups/rg-aks-lza-radbase-sbx-uks-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mid-aks-dev-uks-1 az role assignment create --role Reader --scope /subscriptions/SUBSCRIPTION_ID/resourceGroups/rg-agw-lza-waf-sbx-uks-1 --assignee CLIENT_ID az role assignment create --role Contributor --scope /subscriptions/SUBSCRIPTION_ID/resourceGroups/rg-agw-lza-waf-sbx-uks-1/providers/Microsoft.Network/applicationGateways/agw-lza-waf-sbx-uks-1 --assignee CLIENT_ID az role assignment create --assignee "CLIENT_ID" --scope "/subscriptions/SUBSCRIPTION_ID/resourceGroups/rg-vnw-lza-dev-uks-1/providers/Microsoft.Network/virtualNetworks/vnw-lza-dev-uks-1/subnets/snw-vnw-lza-dev-uks-1-waf-1" --role "Network Contributor" |
I also then issued the following commands:
|
1 2 3 |
kubectl delete -f ./azure-agw.yaml kubectl apply -f ./azure-agw.yaml |
I then re-issued the “kubectl get pods -n kube-system” located the ingress pod and checked the logs, this time I was not seeing any errors.
Deploy Key Vault – Secret provider Class
Next I used the following yaml to deploy the SecretProviderClass This is so that my nginx-ingress container can read the SSL certificate.
again this takes the Key Vault name in my case “keylzasecsdevbxuks1” and the name of the certificate in key vault, which is “Web1AksCertificate ”
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: azure-keylzasecsdevbxuks1 spec: provider: azure secretObjects: - secretName: nginx-cert type: kubernetes.io/tls data: - objectName: Web1AksCertificate key: tls.key - objectName: Web1AksCertificate key: tls.crt parameters: useVMManagedIdentity: "true" userAssignedIdentityID: "CLIENT_ID_FOR_USER_ASSIGNED_ID" keyvaultName: "keylzasecsdevbxuks1" objects: | array: - | objectName: Web1AksCertificate objectType: secret tenantId: "TENANT_ID" |
now from the above, focusing on this section:
|
1 |
objects: | array: - | objectName: Web1AksCertificate objectType: secret |
you can see I have set the “objectType:” to be “Secret”. Now when I was researching how to do this it mentions that you can use “cert” BUT this does not work. Next this section,
|
1 |
- objectName: Web1AksCertificate key: tls.key - objectName: Web1AksCertificate key: tls.crt |
at first I don’t remember setting the tls.crt or tls.key properties in the Key Vault? well this is all done automatically for you.
I then deployed this using the following command:
|
1 |
kubectl apply -f ./kv-secret.yaml |
Deploy Nginx and Sample Web APP
I used the following YAML to deploy the Nginx-Ingress and Same App. I did this as a Deployment. in the same file I am also defining a service and config map.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
apiVersion: apps/v1 kind: Deployment metadata: name: website-deployment spec: selector: matchLabels: app: website replicas: 1 template: metadata: labels: app: website spec: containers: - name: website2 image: "mcr.microsoft.com/dotnet/samples:aspnetapp" ports: - containerPort: 80 protocol: TCP - name: website imagePullPolicy: Always image: nginx:latest ports: - containerPort: 8443 volumeMounts: - mountPath: /etc/nginx/ssl name: secret-volume - mountPath: /etc/nginx/conf.d name: configmap-volume volumes: - name: secret-volume csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "azure-keylzasecsdevbxuks1" - name: configmap-volume configMap: name: website-nginx-cm --- apiVersion: v1 kind: ConfigMap metadata: name: website-nginx-cm data: default.conf: |- server { listen 8086 default_server; listen 8443 ssl; root /usr/share/nginx/html; index index.html; ssl_certificate /etc/nginx/ssl/Web1AksCertificate.crt; ssl_certificate_key /etc/nginx/ssl/Web1AksCertificate.key; location / { proxy_pass http://localhost:8080; } } --- apiVersion: v1 kind: Service metadata: name: website-service spec: selector: app: website ports: - protocol: TCP port: 8443 targetPort: 8443 |
Take your time and have a look over the code. I don’t want to go in depth as to what is happening here as I expect your to know some back Kubernets concepts. But with this section of the code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
- name: website imagePullPolicy: Always image: nginx:latest ports: - containerPort: 8443 volumeMounts: - mountPath: /etc/nginx/ssl name: secret-volume - mountPath: /etc/nginx/conf.d name: configmap-volume volumes: - name: secret-volume csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "azure-keylzasecsdevbxuks1" - name: configmap-volume configMap: name: website-nginx-cm |
But in the above I am defining a volume called “secret-volume” using the “driver: secrets-store.csi.k8s.io” and calling the name of my secretProviderClass that I defined in the previous step.
I am then mounting this volume at “/etc/nginx/ssl ”
I then deployed this yaml using the below command:
|
1 |
kubectl apply -f ./nginx-frontend.yaml |
Finished
and that’s it with all that you should have a working SSL cluster and you can test it with the domain that you have configured.
Troubleshooting
Now you may experience some issues. I know I did when I started this. To troubleshoot you will want to check the logs on the following pods. I issued the following command:
|
1 |
kubectl get pods -n kube-system |
In the output returned look for this container, the id for yours will be different but the first part of the name will be the same:
Then once you have the name issue the following command:
|
1 |
kubectl logs ingress-appgw-deployment-598f99cb7-4zpr8 -n kube-system |
the above command will about any errors. you can also use this command, but the logs for me provided more detail information about the errors.
|
1 |
kubectl describe pod ingress-appgw-deployment-598f99cb7-4zpr8 -n kube-system |
Although I used the commands on the Ingress Azure Application Gateway the same can be used against any other pods, you just need to change the namespace accordingly using the -n flag. for the default namespace you can leave the -n flag off.
If you still need further help, you can reach me at the below:
https://twitter.com/JoeKodeAzure
or Linked in
https://www.linkedin.com/in/joe-moore-773a9a27
