Managing Kubernetes Secrets with AWS Secrets Manager and External Secrets Operator

What is External Secret Operator?

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret. Instead of storing sensitive information directly in Kubernetes Secrets, which are base64 encoded but not encrypted, External Secret Operator allows you to manage these secrets externally and inject them into your Kubernetes environment as needed.

Prerequisites:

Before you begin, ensure you have the following:

Kubernetes Cluster: A running Kubernetes cluster where you have administrative access.
AWS Account: An AWS account with the necessary permissions to access AWS Secrets Manager.
AWS Secret Manager: A service to securely store and retrieve secrets, which will be accessed by your Kubernetes environment.
Helm: A package manager for Kubernetes, which will be used to install the External Secret Operator.

Step 1. Install External Secret operator with helm

To install the External Secret Operator in kubernetes cluster using helm follow the below commands.

				
					helm repo add external-secrets https://charts.external-secrets.io

				
					helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
  # --set installCRDs=false

NOTE: If you do not want the CRDs to be automatically upgraded and managed, you must set the installCRDs option to false. (e.g. –set installCRDs=false).

Step 2. Creating a Secret in AWS Secrets Manager

To securely store and manage secrets, follow these steps to create a secret in AWS Secrets Manager:

In the AWS Management Console, go to “Secrets Manager”.
Click “Store a new secret”.
Choose “Other type of secret” if the secret does not fit into predefined types like database credentials or API keys.
In the “Key/value pairs” section, enter the secret key and corresponding value and click on next.

Provide a descriptive name for the secret in the “Secret name” For example, “MySecrets”.
Optionally, add tags for better organization and management and click next.

Configure automatic rotation if required. This is optional and can be skipped if not needed and click next.
Click “Store” to finalize and store the secret in AWS Secrets Manager.

Step 3. Create an AWS IAM Role for Accessing AWS Secrets Manager

To enable a Kubernetes cluster to securely access AWS Secrets Manager, follow these steps to create an IAM role with appropriate permissions:

Create IAM Policy
- Navigate to the IAM (Identity and Access Management) Console
  - Choose “Policies” from the sidebar.
  - Click “Create policy”.
- Define the Policy:
  - Select the “JSON” tab.
  - Paste the following JSON policy document, replacing <secret-manager-ARN> with the ARN of your AWS Secrets Manager secret:
- Review and Create Policy:
  - Click “Review policy”.
  - Provide a name for the policy, e.g., MySecretsManagerAccessPolicy.
  - Optionally, add a description.

				
					{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "<secret-manager-ARN>"
      ]
    }
  ]
}

- Review and Create Policy:
  - Click “Review policy”.
  - Provide a name for the policy, e.g., MySecretsManagerAccessPolicy.
  - Optionally, add a description.

Step 4. Prepare the Service Account Configuration

Use the following YAML configuration to define the Service Account.
Ensure to replace <ARN-of-role> with the ARN of the IAM role you created earlier.

				
					apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: <ARN-of-role>
  name: my-serviceaccount

Save this configuration to a file, for example, serviceaccount.yaml.
Apply the YAML File to Your Kubernetes Cluster.

				
					kubectl apply -f serviceaccount.yaml

Confirm that the Service Account has been created successfully by running :

				
					kubectl get serviceaccount my-serviceaccount

Step 5. Update the AWS IAM Role Trust Policy :

Navigate to the AWS IAM role and select the IAM role that we have created earlier.
Then select the “Trust Relationship” option and click on the Edit trust policy and paste the below policy and replace the following items :
- <account-id> : replace with your account ID.
- <oidc-url> : Replace it with your oidc url. You can find it in EKS cluster information.
- <namespace> : Provide the namespace where the serviceaccount is created.
- <Service-account-name> : Provide the serviceaccount name that we created inside the kubernetes cluster.

				
					{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/<oidc-url>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "<oidc-url>:aud": "sts.amazonaws.com",
                    "<oidc-url>:sub": "system:serviceaccount:<namespace>:<Service-account-name>"
                }
            }
        }
    ]
}

Step 6. Create Cluster Secret Store for the Secret manager

In your Kubernetes cluster create a Cluster secret store that connect with the AWS Secret Manager and Fetch the secrets.
Use the below yaml configuration to create a Cluster Secret Store and replace the below configuration :
- <region> : Specify the region where your AWS Secrets Manager is located.
- <service-account-name> : Name of the service account that was created earlier.
- <namespace-name> : namespace name where service account is created

				
					apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: <region>
      auth:
        jwt:
          serviceAccountRef:
            name: <service-account-name>
            namespace: <namespace-name>

Ensure the placeholders are replaced with actual values before applying the configuration and run the below command.

				
					kubectl apply -f clustersecretstore.yaml

Step 7. Create External Secret operator that store secrets

After creating the cluster secret store, we’ll now proceed to pull an already saved secret from AWS Secrets Manager.
External secret Operator automatically create a Secret in Kubenetes.
To create a External Secret use the below yaml configuration and replace the placeholder :
- <external-secret-name> : Name of the External Secret
- <cluster-secret-store-name>: Name of the cluster secret store that created earlier
- <secret-name> : Name of the secret that you want to create in your kubernetes cluster
- <secret-key> : Name of the Key that store in Secret
- <secrete-manager-name> : Name of the AWS Secret Manager that Created earlier
- <key-from-secret-manager> : Name of the Key that stored in Secret Manager

				
					apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: <external-secret-name>
spec:
  secretStoreRef:
    name: <cluster-secret-store-name>
    kind: ClusterSecretStore
  target:
    name: <secret-name>
  data:
    - secretKey: <secret-key>
      remoteRef:
        key: <secrete-manager-name>
        property: <key-from-secret-manager>
    - secretKey: <secret-key>
      remoteRef:
        key: <secrete-manager-name>
        property: <key-from-secret-manager>

Step 8. In Addition, Automate pod restarts with Reloader

To ensure your application pods automatically restart when these secrets are updated, you can integrate Reloader into your setup. Reloader monitor for changes in Kubernetes Secrets and ConfigMaps and triggers a rolling restart of the pods that use them.
Install Reloader with helm using the following commands:

				
					helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install reloader stakater/reloader --set watchGlobally=true

Next, annotate your Kubernetes deployments to tell Reloader which secret to monitor.

				
					kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template: 
    metadata:

Conclusion

A reliable and safe way to manage Kubernetes secrets is to integrate the External Secret Operator with AWS Secrets Manager. This integration makes it easier to manage secrets and improves security by enabling smooth access to secrets kept in AWS. This integration supports a safe and effective DevOps environment while streamlining operations and lowering the risk of confidential information being revealed.

Managing Kubernetes Secrets with AWS Secrets Manager and External Secrets Operator