Hardening Production AWS EKS Clusters
This hardens a default EKS cluster with three controls: IAM scoped to the ServiceAccount via IRSA instead of a shared, node-wide role, default-deny NetworkPolicies instead of Kubernetes' allow-all default, and OPA Gatekeeper constraints that block bad manifests at admission.
IRSA: scoping IAM to the ServiceAccount, not the node
IAM Roles for Service Accounts lets a Kubernetes ServiceAccount assume an IAM role via OIDC federation: no static credentials, no node-wide permissions. EKS already runs an OIDC provider per cluster: associate it with IAM, then write a trust policy that trusts tokens only for one namespace/ServiceAccount pair.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/ABCDEF1234567890"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-west-1.amazonaws.com/id/ABCDEF1234567890:sub": "system:serviceaccount:payments:invoice-worker",
"oidc.eks.eu-west-1.amazonaws.com/id/ABCDEF1234567890:aud": "sts.amazonaws.com"
}
}
}
]
}The sub condition is the entire control: without it, any ServiceAccount in the cluster with an annotation pointing at this role could assume it. Annotate the ServiceAccount with the role ARN, attach a scoped permissions policy (the S3 actions and resource ARNs the workload needs, not s3:*), and remove the corresponding permissions from the node instance profile entirely; that's the step teams skip, and skipping it undoes the whole migration.
A broad node IAM role is a lateral-movement multiplier in a multi-tenant cluster: it means the blast radius of compromising any pod on that node equals the blast radius of compromising the node itself. Audit for this specifically.
Default-deny NetworkPolicies, then a scoped allow
NetworkPolicies are additive and namespace-scoped: with none defined, everything is open. The fix is a deny-all policy per namespace, followed by explicit allows for the traffic that's actually legitimate.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: payments
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-invoice-worker-egress
namespace: payments
spec:
podSelector:
matchLabels:
app: invoice-worker
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- to: []
ports:
- protocol: TCP
port: 443podSelector: {} on the deny-all policy means "every pod in this namespace": that's what makes it default-deny rather than a policy that only covers pods you remember to label. The second policy then opens exactly two paths for invoice-worker: DNS resolution via kube-system, and HTTPS egress (needed to reach the S3 and STS endpoints for its IRSA-assumed role). Everything else, the payments database in another namespace, the marketing namespace, the Kubernetes API server, stays blocked. This requires a CNI that actually enforces NetworkPolicy: the default amazon-vpc-cni needs extra configuration around AWS_VPC_K8S_CNI_EXTERNALSNAT, and it's commonly paired with Calico.
OPA Gatekeeper: blocking bad manifests at admission
Gatekeeper is OPA's constraint framework wired in as a validating admission webhook: a ConstraintTemplate defines the Rego logic, a Constraint applies it with specific parameters.
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredresources
spec:
crd:
spec:
names:
kind: K8sRequiredResources
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredresources
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.memory
msg := sprintf("container <%v> is missing memory limits", [container.name])
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.cpu
msg := sprintf("container <%v> is missing cpu limits", [container.name])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
name: require-container-limits
spec:
enforcementAction: deny
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
excludedNamespaces: ["kube-system", "gatekeeper-system"]This constraint closes a real incident pattern: a container with no CPU or memory limits lands on a node and starves every co-located pod under load: the noisy-neighbor problem, but security-adjacent instead of cost-driven. The same logic extends to disallowing :latest tags, requiring readOnlyRootFilesystem, or blocking hostNetwork: true outside a short allowlist.
Always deploy a new constraint in dryrun for at least one full deploy cycle. It logs violations without blocking anything, so you find out fourteen existing Deployments would fail the policy before flipping to deny takes down a Friday release. Fix the violators, then switch.
Pod Security Standards as the baseline underneath
Gatekeeper constraints are bespoke and can drift from intent, but Pod Security Standards (PSS), enforced via the built-in Pod Security Admission controller, give you a baseline that doesn't depend on anyone having written the right Rego. Labeling a namespace with the restricted profile blocks privileged containers, host namespace sharing, and non-default capabilities without a single custom policy:
apiVersion: v1
kind: Namespace
metadata:
name: payments
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restrictedI treat PSS restricted as the floor: the well-known Kubernetes-level footguns, covered for free, and Gatekeeper as where the organization-specific rules live: required resource limits, image provenance, label conventions.
The hardened path, end to end
Each stage narrows what the last one allowed through, and removing any one layer leaves the other two still doing real work. But a cluster with all three is a materially different target than the default-allow, node-role-everywhere cluster most teams start with.
IRSA, NetworkPolicies, and Gatekeeper constraints aren't hard to wire in on their own: clusters ship without them because a deadline doesn't care about blast radius, and nothing forces the conversation until an auditor or an incident does. Retrofitting costs more than building it in from namespace one, but the work is identical, whether you do it on day one or after an incident forces the issue.
Want to actually run this in production?
This tutorial covers the concepts and architecture. If you want to implement it in your own infrastructure, or get good enough to own this problem long-term, I offer 1:1 mentoring built around your real environment, not a generic course.
This tutorial
- Core architecture & key concepts
- Illustrative code snippets
- The reasoning behind each decision
1:1 mentoring
- Working sessions on your own environment
- Direct answers to the edge cases you're hitting
- Feedback on your actual implementation
- Ongoing support as you build it out
Related Tutorials
Kubernetes FinOps: Cutting Compute Spend by 40%
How Karpenter's just-in-time, bin-packing node provisioning replaces static node pools and Cluster Autoscaler tuning to cut compute spend further.
Managing CI/CD Secrets Securely with HashiCorp Vault
Getting secrets out of CI/CD env vars and .env files and into HashiCorp Vault, with short-lived credentials issued to the pipeline, not long-lived ones.