Enterprise GitOps Pipelines with ArgoCD & Helm

This sets up a multi-environment GitOps pipeline with ArgoCD and Helm, using an app-of-apps structure and per-environment value overlays, with automated sync policies that continuously reconcile the cluster against Git and self-heal drift instead of just detecting it.

The app-of-apps pattern

At any scale beyond a handful of services, you don't want one Application per microservice; you want a single root Application that itself manages every other Application. One Git commit to the apps repo is enough to onboard, decommission, or re-point an entire environment.

# root-app.yaml, the single object a cluster admin ever touches manually
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-apps
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/gitops-apps.git
    targetRevision: main
    path: apps/production
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Everything under apps/production/ is a plain ArgoCD Application manifest, one per service or one per logical group. The root app watches that directory; add a file, get a new managed app, delete a file, get a clean teardown.

Helm chart structure and per-environment overlays

Under the app-of-apps layer, each service is a Helm chart with a values file per environment rather than per-environment branches or forked repos. Branching your manifests is how you end up with staging and production silently diverging in ways nobody notices until the postmortem.

charts/payments-api/
├── Chart.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── hpa.yaml
├── values.yaml            # shared defaults
├── values-staging.yaml    # overlay
└── values-production.yaml # overlay

# values-production.yaml
replicaCount: 6
 
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    memory: 1Gi
 
autoscaling:
  enabled: true
  minReplicas: 6
  maxReplicas: 24
 
env:
  LOG_LEVEL: warn
  FEATURE_FLAGS_SOURCE: launchdarkly-prod

values.yaml never sets replica counts or resource requests and limits directly. Every environment-sensitive field lives in the overlay, so the diff between staging and production is a diff between two YAML files on the same chart version.

Wiring the Application CRD to Helm with automated sync

Each child Application points at the chart and the environment overlay, and turns on automated sync with prune and selfHeal.

# apps/production/payments-api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/acme/gitops-apps.git
    targetRevision: main
    path: charts/payments-api
    helm:
      valueFiles:
        - values.yaml
        - values-production.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m

prune: true means resources removed from the chart get removed from the cluster on the next sync. Without it, deleted templates leave orphaned objects behind indefinitely. selfHeal: true reverts any manual change to a live object back to what's in Git, automatically, on the next reconciliation pass (by default every three minutes, or immediately if you're watching the resource).

The child points at project: production, and that AppProject has to exist before the Application can sync. It limits which repos this app may pull from and where it may deploy. I keep it in the same directory, so the root app manages it too, with a lower sync wave so it's created first:

# apps/production/project.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  description: Production workloads
  sourceRepos:
    - https://github.com/acme/gitops-apps.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: payments
  # cluster-scoped kinds the apps may create: Namespace for
  # CreateNamespace=true, CRDs for charts that ship them
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition

Every new production namespace has to be added under destinations in a reviewed commit. That way a typo in a child Application's destination fails the sync instead of deploying somewhere unexpected.

Sync waves and hooks for ordering

Helm's template rendering has no concept of "apply this before that" across separate charts. Plenty of real deployments need it anyway: CRDs before the controllers that watch them, or a migration Job that has to finish before the Deployment depending on the new schema comes up. ArgoCD's sync waves solve this with a plain annotation.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresqls.acme.io
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: payments-db-migrate
  annotations:
    argocd.argoproj.io/sync-wave: "0"
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  annotations:
    argocd.argoproj.io/sync-wave: "1"

Lower waves sync first, and ArgoCD waits for each wave's resources to be Healthy before moving to the next. Combined with PreSync hooks for one-shot jobs like migrations, this replaces the fragile "run this script before that Helm release" step.

Drift detection and the manual override workflow

Even with selfHeal on, you need a documented path for legitimate manual intervention. When an incident requires a live patch faster than a PR-and-merge cycle allows, you have to pause auto-sync, and pausing only the child Application doesn't work. The child's syncPolicy is defined in Git and owned by root-apps, which also has selfHeal: true. The root sees your edit as drift and puts automated back within minutes, often while you're still patching. Pause the root first, then the child:

# 1. stop the root app from reverting child Applications
argocd app set root-apps --sync-policy none
# 2. pause the affected app, then patch the live resource
argocd app set payments-api --sync-policy none
# 3. once the fix is merged in Git, check the app matches it
argocd app diff payments-api
argocd app sync payments-api
# 4. re-enable the root: it restores payments-api's syncPolicy from Git
argocd app set root-apps --sync-policy automated --auto-prune --self-heal

Then either revert the manual change or commit it to Git within the same incident window, and re-enable the root only after the app is confirmed in sync. While the root is paused, no Application gets onboarded or removed, so keep that window short. If you'd rather never touch the root during incidents, you can add an ignoreDifferences entry on the root app for /spec/syncPolicy of child Applications. The downside is that the root then also ignores syncPolicy changes you make in Git on purpose.

Outside of an active incident, drift shows up in the ArgoCD UI and API as OutOfSync. It's worth alerting on directly: more than a few minutes of it on a production app, with no incident in progress, means someone bypassed the process.

Someone will still patch a live resource by hand during a bad incident. What changes is what happens next: either the patch lands in Git and reconciliation confirms it, or selfHeal quietly reverts it and you find out in minutes instead of at the next unrelated deploy.