Karpenter: The Secret Sauce to Drastically Reduce AWS EKS Bills.
In the ever-evolving world of Kubernetes, the efficient scaling of clusters stands as a pivotal pillar of operational success. As organizations strive to optimize resource utilization and minimize costs, the choice of an auto-scaling solution can make all the difference. In this blog post, I’ll share my experience with Karpenter and how you can also use it to achieve better cost savings and increased scalability. You can find all the files related to this blog here.
What is Karpenter?
Karpenter is an intelligent, open-source autoscaling solution that empowers Kubernetes clusters to dynamically adjust resources based on demand, all while keeping a keen eye on cost-efficiency. It takes the hassle out of managing node groups, allowing your cluster to efficiently handle workloads without breaking the bank.
Key Features and Benefits:
- Autoscaling Intelligence: Karpenter employs advanced algorithms to intelligently scale your cluster nodes up or down in response to workload changes, ensuring your applications always have the resources they need, no more and no less. Suppose you have 5 pods pending, each with CPU and memory requirements of 250m and 500MB, respectively. Now, Karpenter will aggregate these resource requirements, calculate the optimal node for scheduling these pods, and dynamically scale the cluster as needed to accommodate them.
- Cost Savings: One of Karpenter’s primary strengths lies in its ability to save costs by efficiently managing resources. It optimizes node usage, eliminating the need for overprovisioning, and reducing cloud infrastructure expenses.
- Cluster Resource Utilization: Karpenter maximizes the utilization of existing cluster resources, minimizing idle capacity and helping to achieve a lean and efficient Kubernetes environment.
- Auto Node Management: Say goodbye to the complexities of managing node groups. Karpenter takes care of node provisioning and decommissioning, simplifying cluster operations.
- Flexibility and Customization: Karpenter is highly customizable, allowing you to tailor its behavior to your specific use case and requirements, ensuring it aligns perfectly with your application needs.
Karpenter vs Cluster Autoscaler:
Cluster Auto Scalers typically operate based on predefined node groups, where you specify a minimum and maximum node count for each group. These groups are often static and inflexible, requiring manual adjustments when workload patterns change. Cluster Auto Scalers often struggle with efficiency due to their reliance on static node groups. They may lead to overprovisioning during peak times and underutilization during off-peak periods, increasing infrastructure costs.
Karpenter, on the other hand, takes a more dynamic approach. It operates independently of node groups, relying on advanced algorithms to analyze real-time cluster demands. It dynamically scales nodes up or down as needed, ensuring resources are allocated optimally without reliance on fixed node group configurations. Karpenter’s unique autoscaling intelligence actively monitors pod requirements and cluster utilization. It intelligently adjusts the number of nodes and resources allocated, ensuring your applications always have the right amount of resources, eliminating overprovisioning, and minimizing idle capacity.
Karpenter installation:
Prerequistes:
1- AWS CLI Installed and configured.
2- VPC, Subnets, Security groups & EKS Cluster running with Atleast 2 nodes.
3- Kubectl
4- Helm
First we need to export some values to variables.
CLUSTER_NAME=<your cluster name>
AWS_PARTITION="aws"
AWS_REGION="$(aws configure list | grep region | tr -s " " | cut -d" " -f3)"
OIDC_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} \
--query "cluster.identity.oidc.issuer" --output text)"
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' \
--output text)Then create IAM roles for Karpenter to use. You can run the same commands after changing the required values.
echo '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}' > node-trust-policy.json
aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
--assume-role-policy-document file://node-trust-policy.jsonThe provided code is creating an AWS IAM role that can be assumed by Amazon Elastic Compute Cloud (EC2) instances running within your AWS environment.
Now attach the required policies to the role
aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
--policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
--policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKS_CNI_Policy
aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
--policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
--policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCoreAttach the IAM role to an EC2 instance profile.
aws iam create-instance-profile \
--instance-profile-name "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"
aws iam add-role-to-instance-profile \
--instance-profile-name "KarpenterNodeInstanceProfile-${CLUSTER_NAME}" \
--role-name "KarpenterNodeRole-${CLUSTER_NAME}"Now we need to create an IAM role that the Karpenter controller will use to provision new instances. The controller will be using IAM Roles for Service Accounts (IRSA) which requires an OIDC endpoint.
cat << EOF > controller-trust-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com",
"${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:karpenter:karpenter"
}
}
}
]
}
EOF
aws iam create-role --role-name KarpenterControllerRole-${CLUSTER_NAME} \
--assume-role-policy-document file://controller-trust-policy.json
cat << EOF > controller-policy.json
{
"Statement": [
{
"Action": [
"ssm:GetParameter",
"ec2:DescribeImages",
"ec2:RunInstances",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypes",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeAvailabilityZones",
"ec2:DeleteLaunchTemplate",
"ec2:CreateTags",
"ec2:CreateLaunchTemplate",
"ec2:CreateFleet",
"ec2:DescribeSpotPriceHistory",
"pricing:GetProducts"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "Karpenter"
},
{
"Action": "ec2:TerminateInstances",
"Condition": {
"StringLike": {
"ec2:ResourceTag/karpenter.sh/provisioner-name": "*"
}
},
"Effect": "Allow",
"Resource": "*",
"Sid": "ConditionalEC2Termination"
},
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}",
"Sid": "PassNodeIAMRole"
},
{
"Effect": "Allow",
"Action": "eks:DescribeCluster",
"Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}",
"Sid": "EKSClusterEndpointLookup"
}
],
"Version": "2012-10-17"
}
EOF
aws iam put-role-policy --role-name KarpenterControllerRole-${CLUSTER_NAME} \
--policy-name KarpenterControllerPolicy-${CLUSTER_NAME} \
--policy-document file://controller-policy.jsonMake sure to change Namespace name if in case you are installing karpenter in another namespace other then karpenter, in my case i have installed in kube-system namespace.
"${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:Namespace:serviceAccount"Now that we have configured access for Karpenter, it’s time to inform Karpenter about the subnets and security groups to be used with the worker nodes it will launch with the help of Tags. Karpenter uses this tags to identify the subnets and security groups.
Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}We can use the below bash snippet to add the tag to the same subnets that we are using with the existing nodegroups or you can manually add this tag to the subnets you want.
for NODEGROUP in $(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \
--query 'nodegroups' --output text); do aws ec2 create-tags \
--tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \
--resources $(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \
--nodegroup-name $NODEGROUP --query 'nodegroup.subnets' --output text )
doneFor security groups use only what suits in your case.
# If your EKS setup is configured to use only Cluster security group, then please execute -
SECURITY_GROUPS=$(aws eks describe-cluster \
--name ${CLUSTER_NAME} --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)# If your setup uses the security groups in the Launch template of a managed node group, then :
SECURITY_GROUPS=$(aws ec2 describe-launch-template-versions \
--launch-template-id ${LAUNCH_TEMPLATE%,*} --versions ${LAUNCH_TEMPLATE#*,} \
--query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \
--output text)After this we need to update aws-auth ConfigMap in the EKS cluster. for this we’ll need kubectl configured.
$ kubectl edit configmap aws-auth -n kube-systemYou will need to add a section to the mapRoles that looks something like this. Replace the ${AWS_PARTITION} variable with the account partition, ${AWS_ACCOUNT_ID} variable with your account ID, and ${CLUSTER_NAME} variable with the cluster name, but do not replace the {{EC2PrivateDNSName}}.
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}
username: system:node:{{EC2PrivateDNSName}}
Installing Karpenter with helm
export KARPENTER_VERSION=v0.30.0 #your required versionWe can now generate a full Karpenter deployment yaml from the helm chart.
helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter \
--set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
--set settings.aws.clusterName=${CLUSTER_NAME} \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \
--set controller.resources.requests.cpu=250m \
--set controller.resources.requests.memory=250m \
--set controller.resources.limits.cpu=1 \
--set controller.resources.limits.memory=1Gi > karpenter.yamlEdit the karpenter.yaml file and find the karpenter deployment affinity rules. Modify the affinity so karpenter will run on one of the existing node group nodes.
The rules should look something like this. Modify the value to match your $NODEGROUP, one node group per line.
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: karpenter.sh/provisioner-name
operator: DoesNotExist
- matchExpressions:
- key: eks.amazonaws.com/nodegroup
operator: In
values:
- Karpenter #Change Value here. Add names of your nodegroups
- Apps
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: "kubernetes.io/hostname"Ready for deployment, Install necessary CRD (Custom Resource Definition)
kubectl create namespace karpenter
kubectl create -f \
https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_provisioners.yaml
kubectl create -f \
https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_awsnodetemplates.yaml
kubectl create -f \
https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_machines.yaml
kubectl apply -f karpenter.yamlNow We should have 2 karpenter pods running in given Namespace.
Now, we need to create a provisioner so that Karpenter can determine the instance types that can be launched. The provisioner is responsible for selecting the appropriate instance types, configuring the nodes, and making them available to the cluster for running workloads.
Provisioners in Karpenter can be customized to align with specific requirements, such as selecting instance types based on workload characteristics, optimizing cost, and ensuring that the cluster’s resource needs are met efficiently.
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
labels:
nodetype: spot
requirements:
- key: karpenter.k8s.aws/instance-category
operator: In
values: [t, m, r]
- key: topology.kubernetes.io/zone
operator: In
values: ["us-west-2a", "us-west-2b", "us-west-2c", "us-west-2d"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["2"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: karpenter.k8s.aws/instance-size
operator: In
values: [nano, micro, small, medium, large]
- key: karpenter.sh/capacity-type
operator: In
values:
- spot
providerRef:
name: default
ttlSecondsAfterEmpty: 10
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: default
spec:
tags:
Name: Karpenter-worker-nodes
amiFamily: AL2
subnetSelector:
karpenter.sh/discovery: Cluster-name
securityGroupSelector:
karpenter.sh/discovery: Cluster-name- Instance Category Requirement:
karpenter.k8s.aws/instance-category: This requirement ensures that Karpenter selects instances with the specified category, which includes "t," "m," and "r." - Availability Zone Requirement:
topology.kubernetes.io/zone: Karpenter will consider instances available in the listed AWS availability zones ("us-west-2a," "us-west-2b," "us-west-2c," and "us-west-2d"). - Instance Generation Requirement:
karpenter.k8s.aws/instance-generation: Instances with a generation greater than "2" are preferred, ensuring that relatively newer instance generations are selected. - Architecture Requirement:
kubernetes.io/arch: Instances with an AMD64 architecture are considered. This requirement ensures that the selected instances are compatible with the AMD64 architecture. - Instance Size Requirement:
karpenter.k8s.aws/instance-size: Instances with sizes ranging from "nano" to "large" are included. This requirement allows flexibility in selecting instances of different sizes based on workload needs. - Capacity Type Requirement:
karpenter.sh/capacity-type: Specifies "spot" as the capacity type. Karpenter will select AWS spot instances, which can be more cost-effective than on-demand instances.
spec.ttlSecondsAfterEmpty: Defines a time-to-live (TTL) duration for worker nodes after they become empty, which is set to 10 seconds. This means that empty nodes will be terminated after 10 seconds.
AWS Node Template Configuration (AWSNodeTemplate resource):
metadata.name: default: Sets the name of the node template.spec.tags.Name: Karpenter-worker-nodes: Assigns a tag to worker nodes with the name "Karpenter-worker-nodes" for identification.spec.amiFamily: AL2: Specifies the Amazon Machine Image (AMI) family to use for the worker nodes, which is Amazon Linux 2 (AL2).spec.subnetSelectorandspec.securityGroupSelector: These selectors are used to associate the worker nodes with specific subnets and security groups within your AWS environment, likely based on a label with the key "karpenter.sh/discovery" and a specific value like the cluster name.
kubectl apply -f provisioner.yamlCheck the logs for Karpenter.
kubectl logs -f -n karpenter -c controller -l app.kubernetes.io/name=karpenterWhen it comes to selecting instance types for your AWS EKS cluster, flexibility is key. Consider your specific workload requirements and the availability of instance types in your region. For stateless applications, I highly recommend leveraging spot instances. This strategic move can yield remarkable cost savings, often up to 70% or more. Remember, the larger the instance you choose, the greater the potential for savings. So, choose wisely, optimize your resources, and watch your AWS bills shrink!
I am open to further discussions on automation & DevOps. You can follow me on LinkedIn and Medium, I talk about DevOps, Learnings & Leadership.
