Kubernetes FinOps on EKS with Karpenter

This is the EKS follow-up to the rightsizing pass. It replaces static node groups and Cluster Autoscaler tuning with Karpenter's just-in-time, bin-packing node provisioning, so each pod gets an instance shape that fits it instead of everything being forced onto a handful of predefined shapes. The cost optimization guide is GKE-focused, but its rightsizing layer applies to any cluster. If you haven't done that work yet, start there.

What Karpenter does differently

Karpenter drops the node-group abstraction entirely. It watches for unschedulable pods, looks at what they need, and picks the cheapest instance types that satisfy those constraints from the full eligible set, then calls the EC2 Fleet API directly. There's no ASG and no node group per shape.

A pending pod that needs 400m CPU and 512Mi memory gets a small instance. A pod that needs 8 vCPU and 32GB gets something else entirely. In the same NodePool, in the same few seconds, Karpenter is solving a bin-packing problem against live EC2 pricing and capacity.

It also makes spot the default instead of a second node group you have to build and babysit. A single NodePool can express "prefer spot, fall back to on-demand," and Karpenter handles the instance-type diversification that makes spot reliable. Teams running Cluster Autoscaler with manually managed spot ASGs often skip that diversification. They pin to two or three instance types and take far more interruptions than necessary.

Installing Karpenter with interruption handling

Karpenter only handles spot interruptions if you give it an SQS queue fed by EventBridge rules and tell it the queue's name. Nothing turns this on implicitly. If settings.interruptionQueue is empty, interruption handling is disabled. The karpenter submodule of terraform-aws-modules/eks creates the queue, the EventBridge rules, the controller's IAM role (via EKS Pod Identity) and the node IAM role in one go:

module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> 21.0"
 
  cluster_name = module.eks.cluster_name
 
  # SQS queue + EventBridge rules for spot interruptions, scheduled
  # maintenance and instance state-change events
  enable_spot_termination = true
 
  # fixed name so the EC2NodeClass below can reference it
  node_iam_role_use_name_prefix = false
  node_iam_role_name            = "karpenter-node-role"
}
 
resource "helm_release" "karpenter" {
  name       = "karpenter"
  namespace  = "kube-system"
  repository = "oci://public.ecr.aws/karpenter"
  chart      = "karpenter"
  version    = "1.14.1" # pin; check the Karpenter compatibility matrix for your EKS version
 
  values = [yamlencode({
    settings = {
      clusterName       = module.eks.cluster_name
      interruptionQueue = module.karpenter.queue_name
    }
  })]
}

The module's Pod Identity association defaults to the karpenter service account in kube-system, which is what the chart creates here. The cluster needs the EKS Pod Identity Agent add-on installed. If you don't use Terraform, Karpenter's getting-started guide ships a CloudFormation template that creates the same queue and rules.

The NodePool and EC2NodeClass

Two CRDs do the work. EC2NodeClass describes the AWS-specific details: AMI, subnets, security groups, IAM role. NodePool describes the scheduling constraints and the disruption behavior.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@latest # pin a specific release (al2023@vYYYYMMDD) in production
  role: "karpenter-node-role"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        deleteOnTermination: true

amiSelectorTerms is required in Karpenter v1, and the alias also implies the AMI family, so there's no separate amiFamily. @latest is convenient for a first run, but every new AMI release then rolls through your fleet via drift on its own schedule. In production, pin a specific version and bump it yourself, through the same Git workflow as the rest of the config.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 336h
      terminationGracePeriod: 24h
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

When a NodePool allows both spot and on-demand, Karpenter prioritizes spot (it doesn't compare the two on price). It falls back to on-demand automatically when no spot offering fits the pod, whether at launch or when replacing an interrupted node. expireAfter: 336h replaces nodes every two weeks even if nothing else triggers it. That caps how long a node can drift from the latest AMI without anyone noticing, and it forces a regular re-bin-packing. terminationGracePeriod: 24h caps how long draining one of those nodes may take. I explain why further down.

Workloads don't need a nodeSelector to land on this pool. A hard nodeSelector: karpenter.sh/capacity-type: spot throws away the on-demand fallback, and when spot is unavailable the pods stay Pending. To prefer spot at the workload level, use a preferred node affinity. Also spread replicas across zones so one spot reclamation wave can't take them all:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 6
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: karpenter.sh/capacity-type
                    operator: In
                    values: ["spot"]
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: checkout-api
      containers:
        - name: checkout-api
          image: registry.internal/checkout-api:1.4.2
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              memory: "1Gi"

The spread constraint's labelSelector only works because the pod template carries app: checkout-api. Without template labels (and the matching spec.selector), the API server rejects the Deployment outright. Resources follow the rest of this series, with CPU requests, no CPU limit, and a memory request and limit.

Consolidation: the part Cluster Autoscaler never did well

Cluster Autoscaler does remove underutilized nodes. When a node's requested resources fall below its utilization threshold and its pods fit elsewhere, it drains the node and shrinks the node group. What it can't change is the shape of the fleet. It only adds or removes nodes of the instance types each node group was built with, so it will never replace three half-empty m5.2xlarge nodes with one right-sized, cheaper instance. Karpenter's consolidation loop does that continuously. It simulates whether running pods could fit onto fewer or cheaper nodes, and if so launches any replacement, then cordons, drains and terminates the excess, and the scheduler re-packs the rest onto a tighter footprint. The plain WhenEmpty policy only removes empty nodes. consolidationPolicy: WhenEmptyOrUnderutilized also targets underutilized ones, so the fleet shrinks back after a traffic spike instead of staying bloated.

Consolidation is disruptive by design. Anything that can't tolerate an unplanned reschedule has to declare it explicitly, on the pod itself. For a batch job, that goes in the pod template of a Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-report
spec:
  backoffLimit: 2
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
    spec:
      restartPolicy: Never
      nodeSelector:
        karpenter.sh/capacity-type: on-demand # spot interruptions ignore do-not-disrupt
      containers:
        - name: report
          image: registry.internal/report-generator:2.1.0
          resources:
            requests:
              cpu: "1"
              memory: "2Gi"
            limits:
              memory: "2Gi"

The hard nodeSelector is intentional here. This job can't survive an interruption, so it shouldn't run on spot at all.

Spot interruption handling, natively

The older pattern, Cluster Autoscaler plus manually managed spot ASGs, relied on aws-node-termination-handler. It runs as a DaemonSet, polls the instance metadata service for the two-minute interruption notice, then cordons and drains the node. It works, but it's one more component to deploy, upgrade and monitor, and it doesn't provision replacement capacity. That only happens once the pods go Pending and Cluster Autoscaler runs its next scan.

With the queue configured above, Karpenter reads the same signals from SQS. That covers the two-minute spot interruption warning, plus scheduled maintenance, instance stopping/terminating and status-check events. On a spot interruption warning it taints and drains the node and starts provisioning the replacement right away, inside the two-minute window.

What this looks like in the FinOps numbers

The rightsizing pass and the provisioning change compound each other. Realistic requests give Karpenter accurate bin-packing inputs, and Karpenter turns them into fewer nodes instead of leaving slack in oversized static groups. If the requests work is already done but the cluster still runs Cluster Autoscaler with two or three static node groups, moving to Karpenter typically takes a further meaningful slice off the compute line, mostly from consolidation and better spot diversification. The size of that slice depends on how badly the static groups fit your workloads. Measure it with Cost Explorer or your CUR data before and after instead of budgeting a number up front.

Adopting this doesn't touch a single application repo. It's a control-plane swap: new autoscaling component, NodePool/EC2NodeClass definitions, PodDisruptionBudgets that mean what they say, and consolidation left running. The real cost shows up afterward, in ownership: someone now maintains NodePool config the way someone used to own ASGs, and someone audits do-not-disrupt before it turns into a hiding place for idle capacity.