Enable secret encryption on an AKS Edge Essentials cluster

Following Kubernetes security best practices, it's recommended that you encrypt the Kubernetes secret store on AKS Edge Essentials clusters. You can perform this encryption by activating the Key Management Service (KMS) provider for AKS Edge Essentials, which enables encryption at rest for secrets stored in the etcd key-value store. Each secret within the secret store is first encrypted with a unique Data Encryption Key (DEK) generated by the K8s API server. This DEK is temporarily held in memory and never written to disk. The API server then requests that the KMS provider encrypt the DEK using a Key Encryption Key (KEK). The KEK is automatically rotated every 30 days and protected with administrator credentials and is accessible only to administrators.

For more detailed information about using KMS, see the official KMS provider documentation.

This article describes how to activate the KMS provider for AKS Edge Essentials clusters.

Prerequisites

The KMS provider is supported for all AKS Edge Essentials clusters, version 1.10.868.0 and later.

Enable the KMS provider

In your aksedge-config.json file, in the Init section, set Init.KmsPlugin.Enable to true:

"Init": {
 "KmsPlugin": {
     "Enable": true
  }
}

The following output is displayed during deployment, showing that the KMS provider is enabled:

Preparing to install kms-plugin as encryption provider...

For deployment instructions, see Single machine deployment.

Verify that the KMS provider is enabled

To verify that the KMS provider is enabled, run the following command and ensure that the health status of kms-providers is OK:

kubectl get --raw='/readyz?verbose'

[+]ping ok
[+]Log ok
[+]etcd ok
[+]kms-providers ok
[+]poststarthook/start-encryption-provider-config-automatic-reload ok

To create secrets in AKS Edge Essentials clusters, see Managing Secrets using kubectl in the Kubernetes documentation.

If you encounter errors, see the Troubleshooting section.

How to update your secrets after KEK is rotated (optional)

The KEK is automatically updated every 30 days. At that point, the DEK encrypting each secret is securely encrypted with the KEK that was in use when you created the secret. When you next update the secret, the associated DEK is re-encrypted with the current KEK. If your workloads update secrets regularly, no additional action is required. If your secrets change infrequently, you may optionally choose to re‑write the same secret values every 30 days so the associated DEKs re‑encrypted with the latest KEKs. This is recommended as per Kubernetes best practices, not a requirement. Even without this step your secrets remain encrypted at rest. Use the command below if you wish to re-encrypt your secrets with the latest KEK:

kubectl get secrets --all-namespaces -o json | kubectl replace -f - 

Troubleshooting

If there are errors with the KMS provider, follow this procedure:

1. Check that the AKS version is 1.10.868.0 or later. Use the following command to check the current version of AKS Edge Essentials:

   Get-Command -Module AKSEdge | Format-Table Name, Version
   

   If the version is older, upgrade to the latest version. For more information, see Upgrade an AKS cluster.

2. View the readyz API. If the problem persists, verify that the KMS provider is enabled. See the Verify that the KMS provider is enabled section.

   If you receive "[-]" before the kms-providers field, collect diagnostic logs for debugging. For more information, see Get kubelet logs from cluster nodes.

3. If there are still errors, the machine running the AKS Edge Essentials cluster might be paused or turned off for an extended period of time (over 30 days). To get KMS back into a healthy state, you can use the Repair-Kms command to restore any necessary tokens:

   Repair-AksEdgeKms
   

4. If you still encounter errors, contact Microsoft Customer Support and collect logs.

Next steps

• Overview
• Uninstall AKS cluster