Cut Deployment Lead Time to Under an Hour

Slow deploys are almost never a tooling problem. They're a trust problem: teams add manual gates because they don't trust the pipeline to catch what a human reviewer would catch.

GitOps addresses this by making the pipeline the audit trail. Every production change is a commit in a Git repository that traces back to a reviewed pull request, and the cluster is reconciled to that commit automatically.

Where the lead time goes

Lead time is the time from a merged change to that change running in production. With a weekly change advisory board, the code is usually ready in minutes, and most of the lead time is spent waiting, first for the next CAB slot and then for someone to run the deploy in the release window. A change merged the day after the meeting can easily wait most of a week. Batching also makes each release bigger and riskier, and bigger releases tend to get stricter gates, which pushes teams to batch even more.

The pipeline below removes that waiting. Every merge ships on its own, and the question a human approver used to answer ("does it look healthy in production?") becomes a metric check that runs on every release.

The pattern

• Source of truth: application manifests live in a dedicated Git config repo instead of inside CI scripts.
• Reconciliation: Argo CD watches the repo and converges the cluster to match it, so nobody runs kubectl apply from a laptop. Flux works the same way; I'll use Argo CD here.
• Progressive delivery: the canary steps and their pass/fail criteria are written as config in the repo.
• Rollback: the rollout aborts automatically, and a git revert brings Git back in line with what's running.

Step 1: CI builds the image and bumps the tag

The application repo's release workflow runs on every merge to main. Branch protection has already made the test job pass on the PR, so this workflow only builds and pushes an image tagged with the commit SHA, then writes that tag into the config repo. The review happened on the application PR, and the bump commit points back to it through the SHA, so every deploy can still be traced to a review.

# app repo: .github/workflows/release.yml
name: Release
 
on:
  push:
    branches: [main]
 
permissions:
  contents: read
  packages: write
 
concurrency:
  group: release-payments-api
  cancel-in-progress: false
 
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
 
      - uses: docker/setup-buildx-action@v4
 
      - uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
 
      - uses: docker/build-push-action@v7
        with:
          context: .
          push: true
          tags: ghcr.io/acme/payments-api:${{ github.sha }}
          cache-from: type=registry,ref=ghcr.io/acme/payments-api:buildcache
          cache-to: type=registry,ref=ghcr.io/acme/payments-api:buildcache,mode=max
 
      - name: Check out config repo
        uses: actions/checkout@v7
        with:
          repository: acme/gitops-config
          ssh-key: ${{ secrets.CONFIG_REPO_DEPLOY_KEY }}
          path: gitops-config
 
      - name: Bump image tag
        working-directory: gitops-config/apps/payments-api/overlays/production
        run: |
          kustomize edit set image ghcr.io/acme/payments-api=ghcr.io/acme/payments-api:${{ github.sha }}
          git config user.name "release-bot"
          git config user.email "release-bot@users.noreply.github.com"
          if git diff --quiet; then echo "Tag already deployed"; exit 0; fi
          git commit -am "payments-api: deploy ${{ github.sha }}"
          git pull --rebase
          git push

The deploy key has write access to the config repo and nothing else. Nothing in this workflow can reach the cluster either. CI never holds cluster credentials, because Argo CD pulls the changes from Git. concurrency stops two merges of this service from racing each other, and git pull --rebase covers the case where another service's bot pushed to the same config repo in the meantime. Kustomize is preinstalled on GitHub's Ubuntu runners. If your change control requires a second approval on production changes, open a PR against the config repo instead of pushing. The pipeline stays the same with one review step added. That step costs lead time, so I'd reserve it for the services that need it.

Step 2: the config repo and the Argo CD Application

The config repo holds a base plus a production overlay. After a bump, the overlay's kustomization.yaml looks like this:

# gitops-config: apps/payments-api/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: payments
resources:
  - ../../base
images:
  - name: ghcr.io/acme/payments-api
    newTag: 3f9c2e1a7b4d5c6e8f0a1b2c3d4e5f6a7b8c9d0e

And one Argo CD Application points at that overlay with automated sync:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/gitops-config.git
    targetRevision: main
    path: apps/payments-api/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

By default Argo CD polls Git every few minutes. With a webhook from the config repo, it syncs within seconds of the bump commit. selfHeal reverts manual edits made in the cluster, so what runs keeps matching what's in Git.

Step 3: a canary gated by Prometheus

Instead of a Deployment, the base defines an Argo Rollouts Rollout. It shifts traffic in steps while an analysis runs in the background. If the analysis fails, the rollout aborts and traffic goes back to the stable version on its own.

# gitops-config: apps/payments-api/base/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments-api
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: payments-api
          image: ghcr.io/acme/payments-api  # tag set by the overlay
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 256Mi
  strategy:
    canary:
      canaryService: payments-api-canary
      stableService: payments-api-stable
      analysis:
        templates:
          - templateName: http-success-rate
        startingStep: 1
        args:
          - name: service-name
            value: payments-api-canary
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }

payments-api-canary and payments-api-stable are two ordinary Service objects with the selector app: payments-api. The controller adds a pod-template-hash to each selector, so one selects only canary pods and the other only stable pods, which lets Prometheus measure the canary separately. Without a traffic router (Istio, NGINX, Gateway API), setWeight is approximated with replica counts, so 20% of 5 replicas is one canary pod. If you need precise percentages, add a traffic router.

Here is the analysis template:

# gitops-config: apps/payments-api/base/analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: http-success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      failureLimit: 1
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus-operated.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}", code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

Adjust the metric and label names to match your instrumentation. With no count, a background analysis keeps running until the rollout finishes, and two failed measurements (one more than failureLimit) abort it. If the canary gets no traffic at all, the query returns no usable value and the analysis errors out instead of passing, which is the safer outcome.

How each piece cuts lead time

• Smaller batches: each release contains a single merge, which keeps it small and cheap to roll back. Small releases are what make it reasonable to drop the CAB.
• Automated promotion: the question a human approver used to answer is now a Prometheus query that runs on every release. Adding up the steps, you get CI (a few minutes if you've sped up the build), an Argo CD sync (seconds with a webhook) and 15 minutes of canary pauses. From merge to 100% of traffic that's typically well under an hour, compared with waiting for the next approval slot.
• Fast rollback: when the analysis fails, the rollout aborts by itself and the stable pods are still running. After that, git revert the bump commit so Git matches what's running again, and Argo CD shows the app as healthy on the previous tag.

Teams that adopt this consistently tend to move from weekly deploys with a change advisory board to deploying several times a day. Traceability gets better too, since every production change keeps a commit, a diff, a linked PR and an analysis run attached to it.