Secrets Unveiled: AWS Secrets Manager

In the rapidly evolving realm of cloud-native technologies and container orchestration, securing sensitive data and managing secrets has become an indispensable facet of modern infrastructure. As organizations embrace these advanced technologies, they encounter new challenges: How can they safeguard critical information such as API keys, database passwords, and encryption keys? How do they ensure that sensitive data remains confidential and is easily accessible only to authorized applications?

Enter Secret Managers, the guardians of confidential information in the era of cloud-native computing. In this blog post, we will embark on an exploration of AWS Secrets Manager, uncovering what it is, and why it plays an essential role in contemporary cloud-native operations, particularly in conjunction with Amazon Elastic Kubernetes Service (EKS). We will delve into how seamlessly AWS Secrets Manager can be integrated into the intricate fabric of modern cloud-native ecosystems, ensuring the security and accessibility of your critical secrets within your EKS environment.

AWS Secrets Manager is a managed service that helps you protect and manage sensitive information securely. It is designed to simplify the management of sensitive data, such as database credentials, API keys, and other secrets, within AWS infrastructure and applications. AWS Secrets Manager offers the following key features and benefits:

1. Centralized Secret Storage: AWS Secrets Manager provides a central and secure repository for storing secrets, reducing the need to manage secrets across different AWS services and applications independently.
2. Secure Storage: Secrets are stored in encrypted format using AWS Key Management Service (KMS) keys, ensuring data-at-rest security.
3. Automated Rotation: AWS Secrets Manager can automatically rotate secrets on a schedule, which enhances security by regularly changing sensitive credentials. For example, it can automatically update the password for a database user.
4. Access Control: You can configure fine-grained access control using AWS Identity and Access Management (IAM) policies to restrict who can access and manage secrets.
5. Integration with AWS Services: Secrets stored in AWS Secrets Manager can be easily integrated into various AWS services, such as Amazon RDS (Relational Database Service), Amazon Redshift, and AWS Lambda, simplifying the management of credentials for these services.
6. Programmatic Access: You can programmatically retrieve secrets from AWS Secrets Manager using AWS SDKs or APIs in your applications and scripts. This allows your applications to access secrets securely without hardcoding them.

In simple terms, a Secret Manager is a tool that lets you safely create confidential information, like passwords or API keys, and then easily supply them to your applications, either as environment variables or files, all while maintaining strong security. You can manage these secrets conveniently from the AWS console.

Prerequisites:

• EKS cluster with OIDC
• Helm
• Kubectl & AWS CLI configured

All files related to this blog can be found here.

Now let’s start setting up Secrets Manager.

Before granting service accounts access to AWS resources, we must create essential IAM roles and policies that can be used as annotations in service accounts. Enter IRSA (IAM Roles for Service Accounts), the native AWS EKS solution that enables applications running within EKS pods to securely access AWS APIs by leveraging permissions configured in AWS IAM roles.

Craft a policy granting permissions for GetSecretValue, DescribeSecret, and ListSecrets across all resources, and assign it a meaningful name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
        }
    ]
}

Navigate to the AWS IAM console, proceed to the ‘Policy’ section, and select ‘Create Policy.’ Then, within the JSON tab, paste the provided JSON content

Click on Next and provide a name to policy and click create.

After completing this step, proceed to the next phase, where we’ll create a role that establishes a trust relationship and uses this policy to grant permissions for secrets.

To facilitate the trust relationship, we require a service account. You can create one using kubectl, or alternatively, execute the following YAML configuration to create it. Change namespace according to your requirement. I have used app namespace.

cat > service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: Secret-manager
  namespace: app
EOF
kubectl apply -f service-account.yaml

This command creates a Kubernetes service account named “Secret-manager” within the “app” namespace by applying the provided YAML configuration.

Next Set your AWS account ID to an environment variable with the following command.

account_id=$(aws sts get-caller-identity --query "Account" --output text)

Set your cluster’s OIDC identity provider to an environment variable with the following command. Replace my-cluster with the name of your cluster and also export aws region.

AWS_REGION="$(aws configure list | grep region | tr -s " " | cut -d" " -f3)"

oidc_provider=$(aws eks describe-cluster --name my-cluster --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

Set variables for the namespace and name of the service account. Replace secret-manager with the Kubernetes service account that you want to assume the role. Replace app with the namespace of the service account.

export namespace=app
export service_account=secret-manager

Run the following command to create a trust policy file for the IAM role. If you want to allow all service accounts within a namespace to use the role, then copy the following contents to your device. Replace StringEquals with StringLike and replace $service_account with *. I have created role to allow all service accounts within app namespace to use this role.

cat >trust-relationship.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$account_id:oidc-provider/$oidc_provider"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringLike": {
          "$oidc_provider:aud": "sts.amazonaws.com",
          "$oidc_provider:sub": "system:serviceaccount:$namespace:*"
        }
      }
    }
  ]
}
EOF

If you want just one service account to be allowed to use this role change * to $service_account and Replace StringLike with StringEquals

"$oidc_provider:sub": "system:serviceaccount:$namespace:$service_account"

Create the role. Replace my-role with a name for your IAM role, and my-role-description with a description for your role.

aws iam create-role --role-name my-role --assume-role-policy-document file://trust-relationship.json --description "my-role-description"

Attach an IAM policy to your role. Replace my-role with the name of your IAM role and my-policy with the name of an existing policy that you created previous to this step.

aws iam attach-role-policy --role-name my-role --policy-arn=arn:aws:iam::$account_id:policy/my-policy

Annotate your service account with the Amazon Resource Name (ARN) of the IAM role that you want the service account to assume. Replace my-role with the name of your existing IAM role.

kubectl annotate serviceaccount -n $namespace $service_account eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role

Next, you can describe the service account and verify that it has been annotated.

kubectl describe serviceaccount Secret-manager -n app

Output should be like this.

Name:                secret-manager
Namespace:           app
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/secret-role
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Now when we have created roles and polices its time to install ASCP (AWS Secrets and Configuration Provider).

Install the ASCP:

To install the Secrets Store CSI Driver and ASCP by using Helm, use the following commands. To ensure the repo is pointing to the latest chart, use helm repo update.

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true

helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

Don’t forgot to set syncSecret.enabled=true. If not set secret provider class will not be able to create secret.

After installing you can check kube-system namespace for the pods deployed.

kubectl get pods -n kube-system

Output should be like this.

csi-secrets-store-secrets-store-csi-driver-grg9p                  3/3     Running   
csi-secrets-store-secrets-store-csi-driver-gszfr                  3/3     Running  
secrets-provider-aws-secrets-store-csi-driver-provider-aws5zqb4   1/1     Running  
secrets-provider-aws-secrets-store-csi-driver-provider-aws8jhhr   1/1     Running  

Now, let’s proceed to create a secret in the AWS Secrets Manager console. As an example, I’ll create a secret with the following values:

• Username: abhishek
• Password: mystrongpassword

Within AWS Secrets Manager, select the “Other” secret type, and then proceed to add key-value pairs. For the Key Management Service (KMS), utilize the default KMS provided by AWS.

Provide a name and description for the secret, then proceed by clicking “Next.” For now, we’ll skip the rotation configuration. Afterward, click “Next” and create the secret.

Next, I’m creating a Helm chart that includes an NGINX image and secret configuration.

helm create nginx

This will create a chart named nginx with default configuration. After running this command we’ll get these

• Chart.yaml
• charts
• templates (directory)
• values.yaml

We need to create secrets.yaml in templates folder.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: nginx
  namespace: app
spec:
  provider: "aws"
  secretObjects:
   - secretName: nginx
     type: Opaque
     data:
       - objectName: "username"
         key: "username"
       - objectName: "password"
         key: "password"
  parameters:
    objects: |
      - objectName: arn:aws:secretsmanager:us-west-2:1234567890:secret:nginx-test-secret-baRne8
        jmesPath:
          - path: "username"
            objectAlias: "username"
          - path: "password"
            objectAlias: "password"

By default, the secret is mounted as a file in the /mnt directory, and to use it in a system environment variable format, follow the method described above. Change ARN to yours. Following this, you’ll need to include volume and volume mount configurations in your deployment.yaml file, referencing the secret as needed.

Add Volume mount:

volumeMounts:
  - name: secrets-store-inline
    mountPath: "/mnt/secrets-store"
    readOnly: true

Add Volume:

volumes:
   - name: secrets-store-inline
     csi:
       driver: secrets-store.csi.k8s.io
       readOnly: true
       volumeAttributes:
         secretProviderClass: "nginx" # Use same as secret provider class.

Add env:

env:
  - name: username
    valueFrom:
      secretKeyRef:
        key: username
        name: nginx
  - name: password
    valueFrom:
      secretKeyRef:
        key: password
        name: nginx

here is the complete deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "nginx.fullname" . }}
  labels:
    {{- include "nginx.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "nginx.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "nginx.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "nginx.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
          env:
            - name: username
              valueFrom:
                secretKeyRef:
                  key: username
                  name: nginx
            - name: password
              valueFrom:
                secretKeyRef:
                  key: password
                  name: nginx
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "nginx"

Now edit values.yaml

Change image tag to latest and disable service account and provide name of service account which is annotated.

serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "secret-manager"

Done, with the changes. Now install the app with helm.

helm upgrade nginx --install . -f values.yaml -n app

You should get nginx pod running in app namespace. Verify with

kubectl get pods -n app

Output should be like this

nginx-5b7c5f8b8d-b9dmj           2/2     Running   0          2m52s

To verify the presence of environment variables in the pods, you can execute the following command within the pods:

kubectl exec -it nginx-5b7c5f8b8d-b9dmj bash

env | grep username
env | grep password

Output:

Great! We have successfully integrated secret manager with EKS. In case you update some value for any env, you just need to upgrade the chart to get updated values.
You can download the used helm chart from here.

I am open to further discussions on automation & DevOps. You can follow me on LinkedIn and Medium, I talk about DevOps, Learnings & Leadership.