Hardening Production AWS EKS Clusters

This hardens a default EKS cluster with three controls: IAM scoped to the ServiceAccount via IRSA instead of a shared, node-wide role, default-deny NetworkPolicies instead of Kubernetes' allow-all default, and OPA Gatekeeper constraints that block bad manifests at admission.

IRSA: scoping IAM to the ServiceAccount, not the node

IAM Roles for Service Accounts lets a Kubernetes ServiceAccount assume an IAM role via OIDC federation: no static credentials, no node-wide permissions. EKS already runs an OIDC provider per cluster: associate it with IAM, then write a trust policy that trusts tokens only for one namespace/ServiceAccount pair.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/ABCDEF1234567890"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-west-1.amazonaws.com/id/ABCDEF1234567890:sub": "system:serviceaccount:payments:invoice-worker",
          "oidc.eks.eu-west-1.amazonaws.com/id/ABCDEF1234567890:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

The sub condition is the entire control: without it, any ServiceAccount in the cluster with an annotation pointing at this role could assume it. Annotate the ServiceAccount with the role ARN, attach a scoped permissions policy (the S3 actions and resource ARNs the workload needs, not s3:*), and remove the corresponding permissions from the node instance profile entirely; that's the step teams skip, and skipping it undoes the whole migration.

Trimming the node role isn't enough on its own. Any pod can still call the instance metadata service (IMDS) at 169.254.169.254 and pick up whatever credentials the node role still has. Require IMDSv2 and set the hop limit to 1 in the node group's launch template. IMDSv2 responses then can't cross the extra network hop into a pod's network namespace, and the node itself (kubelet, the VPC CNI running on the host network) keeps working:

resource "aws_launch_template" "eks_nodes" {
  name_prefix = "eks-nodes-"
 
  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required" # IMDSv2 only
    http_put_response_hop_limit = 1          # pods can't reach IMDS
  }
}

Pods that use IRSA don't need IMDS for credentials. Some add-ons do read the region or VPC ID from it (the AWS Load Balancer Controller, for one), so pass those values explicitly as flags or Helm values before you roll this out.

Default-deny NetworkPolicies, then a scoped allow

NetworkPolicies are additive and namespace-scoped: with none defined, everything is open. The fix is a deny-all policy per namespace, followed by explicit allows for the traffic that's actually legitimate.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-invoice-worker-egress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: invoice-worker
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/16   # VPC CIDR
              - 172.20.0.0/16 # service CIDR (kubernetes.default ClusterIP)
      ports:
        - protocol: TCP
          port: 443

podSelector: {} on the deny-all policy means "every pod in this namespace", which is what makes it default-deny. A policy with a label selector would only cover the pods you remember to label. The second policy then lets invoice-worker resolve DNS through kube-system and make HTTPS calls, which it needs to reach the S3 and STS endpoints for its IRSA-assumed role. Everything else is blocked, including the payments database in another namespace, the marketing namespace and the Kubernetes API server, as long as the HTTPS rule excludes in-cluster and in-VPC addresses.

Be careful with the HTTPS rule. A bare to: [] on port 443 allows any destination on 443, and the Kubernetes API server listens on 443 too, both behind the kubernetes.default ClusterIP and on the private endpoint ENIs inside your VPC. The ipBlock with except keeps public HTTPS open while carving out the VPC CIDR and the service CIDR (replace both with your cluster's values from aws eks describe-cluster --query cluster.kubernetesNetworkConfig). If the workload reaches STS or S3 through interface VPC endpoints, those live inside the VPC range, so add a separate ipBlock allow for the endpoint subnets. If the cluster's API endpoint is public, also restrict its allowed source CIDRs, because an IP-based rule can't tell the public API endpoint apart from any other AWS address.

Network rules are only part of it. A pod that doesn't talk to the Kubernetes API shouldn't carry a token for it either:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: invoice-worker
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/invoice-worker
automountServiceAccountToken: false

IRSA still works with this setting. The EKS pod identity webhook injects its own projected token with audience sts.amazonaws.com, and that token is only valid for STS.

None of these policies do anything until something enforces NetworkPolicy. The AWS VPC CNI can do it natively since v1.14 (keep the add-on on a current version), through a network policy agent that runs alongside aws-node, but it's off by default. Turn it on in the managed add-on's configuration. You don't need Calico for this:

resource "aws_eks_addon" "vpc_cni" {
  cluster_name = aws_eks_cluster.main.name
  addon_name   = "vpc-cni"
 
  configuration_values = jsonencode({
    enableNetworkPolicy = "true"
  })
}

The CLI equivalent is aws eks update-addon --cluster-name <cluster> --addon-name vpc-cni --configuration-values '{"enableNetworkPolicy":"true"}'. Once it's applied, aws-node pods show two containers instead of one. Before that, the API server accepts every policy above and nothing enforces them.

OPA Gatekeeper: blocking bad manifests at admission

Gatekeeper is OPA's constraint framework wired in as a validating admission webhook: a ConstraintTemplate defines the Rego logic, a Constraint applies it with specific parameters.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredresources
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredResources
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredresources
 
        # Find the pod spec, whatever kind of object wraps it
        pod_spec = spec {
          input.review.object.kind == "Pod"
          spec := input.review.object.spec
        }
 
        pod_spec = spec {
          {"Deployment", "StatefulSet", "DaemonSet", "ReplicaSet", "Job"}[input.review.object.kind]
          spec := input.review.object.spec.template.spec
        }
 
        pod_spec = spec {
          input.review.object.kind == "CronJob"
          spec := input.review.object.spec.jobTemplate.spec.template.spec
        }
 
        containers[c] {
          c := pod_spec.containers[_]
        }
 
        containers[c] {
          c := pod_spec.initContainers[_]
        }
 
        violation[{"msg": msg}] {
          c := containers[_]
          not c.resources.requests.cpu
          msg := sprintf("%v: container <%v> is missing a cpu request", [input.review.object.kind, c.name])
        }
 
        violation[{"msg": msg}] {
          c := containers[_]
          not c.resources.requests.memory
          msg := sprintf("%v: container <%v> is missing a memory request", [input.review.object.kind, c.name])
        }
 
        violation[{"msg": msg}] {
          c := containers[_]
          not c.resources.limits.memory
          msg := sprintf("%v: container <%v> is missing a memory limit", [input.review.object.kind, c.name])
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
  name: require-container-resources
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet", "DaemonSet", "ReplicaSet"]
      - apiGroups: ["batch"]
        kinds: ["Job", "CronJob"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]

Two details make this constraint useful. First, it matches workloads as well as Pod. A constraint on Pod alone still blocks the pods, but the Deployment itself is accepted. kubectl apply succeeds, the ReplicaSet controller fails to create pods, and the only trace is a FailedCreate event that nobody watches. Matching Deployments, StatefulSets, DaemonSets, Jobs and CronJobs rejects the manifest at kubectl apply, so the person deploying sees the error. Second, it checks initContainers too, since they get scheduled against the same node resources.

It requires CPU and memory requests on every container plus a memory limit, and leaves out a CPU limit on purpose. CPU requests are what protect you from a noisy neighbor. The scheduler reserves them, and under contention the kernel shares CPU in proportion to them, so a busy pod can't starve the others on the node. A CPU limit adds throttling even when the node has idle cores, which usually hurts latency more than it helps. Memory can't be throttled. Without a limit, one leaking container pushes the whole node into memory pressure and evictions.

The same template pattern extends to disallowing :latest tags, requiring readOnlyRootFilesystem, or blocking hostNetwork: true outside a short allowlist.

Pod Security Standards as the baseline underneath

Gatekeeper constraints are bespoke and can drift from intent, but Pod Security Standards (PSS), enforced via the built-in Pod Security Admission controller, give you a baseline that doesn't depend on anyone having written the right Rego. Labeling a namespace with the restricted profile blocks privileged containers, host namespace sharing, and non-default capabilities without a single custom policy:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted

I treat PSS restricted as the floor. It covers the well-known Kubernetes-level footguns for free. Gatekeeper is where I put the rules specific to the organization, such as required resource requests and memory limits, image provenance and label conventions.

The hardened path, end to end

Each stage narrows what the last one allowed through, and removing any one layer leaves the other two still doing real work. But a cluster with all three is a materially different target than the default-allow, node-role-everywhere cluster most teams start with.

IRSA, NetworkPolicies, and Gatekeeper constraints aren't hard to wire in on their own: clusters ship without them because a deadline doesn't care about blast radius, and nothing forces the conversation until an auditor or an incident does. Retrofitting costs more than building it in from namespace one, but the work is identical, whether you do it on day one or after an incident forces the issue.