SecurityVaultDevSecOpsIntermediate

CI/CD Secrets Management with HashiCorp Vault

Jerry RajaonarivoUpdated Sep 30, 20268 min read

This gets secrets out of CI/CD environment variables and .env files and into HashiCorp Vault. CI authenticates with GitHub's OIDC tokens (or AppRole where OIDC isn't available) and receives short-lived dynamic credentials in place of long-lived static secrets.

Enabling KV v2 and writing your first versioned secret

Vault's KV v2 secrets engine is the starting point: a versioned key-value store where overwriting a secret doesn't destroy history and you can roll back.

bash
# Skip this line on a `vault server -dev` instance: dev mode already mounts
# secret/ as kv-v2, and enabling it again fails with "path is already in use".
vault secrets enable -path=secret kv-v2
 
vault kv put secret/ci/payments-api \
  api_key="sk_live_..." \
  webhook_secret="whsec_..."
 
vault kv get secret/ci/payments-api
vault kv get -version=1 secret/ci/payments-api

That's already better than a CI env var. vault kv metadata get secret/ci/payments-api shows every version and when it was created, deleted or destroyed. It doesn't show who wrote it. For that you need Vault's audit log, which is off until you enable an audit device:

bash
vault audit enable file file_path=/var/log/vault_audit.log

Every request and response is then logged as JSON with the calling identity (entity, auth method, policies) and the path. Secret values are HMAC-hashed rather than written in clear. Enable it before anything else touches Vault, because without an audit trail you can't reconstruct what happened after a leak. Make sure the Vault process can write to that path, and ship the file somewhere durable.

Versioning helps, but this is still a static secret sitting at a path. What matters more is controlling who can read it and how CI proves it's allowed to.

A policy that grants exactly what the pipeline needs

A human logging into Vault uses their own identity through userpass, OIDC or whatever your org runs. A CI pipeline has no identity of that kind, and giving it a personal token (or worse, a shared root token) only moves the credential into a CI secret field. Whatever auth method the pipeline uses, it ends up holding a token bound to a policy like this one:

hcl
# policy: ci-payments-read.hcl
path "secret/data/ci/payments-api" {
  capabilities = ["read"]
}
 
path "secret/metadata/ci/payments-api" {
  capabilities = ["read", "list"]
}
 
# dynamic database credentials, covered further down
path "database/creds/ci-readonly" {
  capabilities = ["read"]
}
bash
vault policy write ci-payments-read ci-payments-read.hcl

GitHub Actions: OIDC instead of a stored Vault credential

GitHub Actions can mint a short-lived OIDC token (a signed JWT) for every job, with claims that say which repository, branch and workflow is running. Vault's JWT auth method verifies that token against GitHub's public keys and returns a Vault token. Nothing is stored in GitHub secrets, so there's no credential to leak or rotate.

bash
vault auth enable jwt
 
vault write auth/jwt/config \
  oidc_discovery_url="https://token.actions.githubusercontent.com" \
  bound_issuer="https://token.actions.githubusercontent.com"
 
vault write auth/jwt/role/ci-payments-pipeline - <<'EOF'
{
  "role_type": "jwt",
  "bound_audiences": ["https://github.com/example-org"],
  "bound_claims": {
    "repository": "example-org/payments-api",
    "ref": "refs/heads/main"
  },
  "user_claim": "repository",
  "token_policies": ["ci-payments-read"],
  "token_ttl": "15m",
  "token_max_ttl": "30m"
}
EOF

bound_claims does the access control here. Only jobs from example-org/payments-api running on main can log in with this role, and a fork, a feature branch or another repo in the org gets rejected. With token_ttl=15m, the resulting Vault token expires shortly after a normal job would finish. Vault needs outbound HTTPS to token.actions.githubusercontent.com to fetch GitHub's signing keys, and the runner needs to reach Vault. In practice that means a self-hosted runner or a Vault endpoint exposed to GitHub's runners.

yaml
# .github/workflows/deploy.yml
name: Deploy payments-api
on:
  push:
    branches: [main]
 
permissions:
  id-token: write   # lets the job request a GitHub OIDC token
  contents: read
 
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
 
      - name: Fetch secret from Vault
        uses: hashicorp/vault-action@v4
        with:
          url: https://vault.internal.example.com:8200
          method: jwt
          role: ci-payments-pipeline
          jwtGithubAudience: https://github.com/example-org
          secrets: |
            secret/data/ci/payments-api api_key | PAYMENTS_API_KEY ;
            secret/data/ci/payments-api webhook_secret | PAYMENTS_WEBHOOK_SECRET
 
      - name: Deploy
        run: ./scripts/deploy.sh
        env:
          PAYMENTS_API_KEY: ${{ env.PAYMENTS_API_KEY }}
          PAYMENTS_WEBHOOK_SECRET: ${{ env.PAYMENTS_WEBHOOK_SECRET }}

jwtGithubAudience must match bound_audiences on the Vault role, otherwise the login fails with an audience error. hashicorp/vault-action handles the login and masks the values in the Actions log. The secret arrives as a step output or env var and is never written to the runner's disk. If you call Vault's HTTP API directly, do the same and pipe the response into an environment variable or a build tool's secret input instead of a temp file you might forget to clean up.

Resist the urge to echo $PAYMENTS_API_KEY even temporarily while debugging a pipeline. Most CI systems only mask known secret values: one fetched fresh from Vault in that same job often isn't registered as a masked string yet, and a failed step's error output can dump environment context verbatim. The safest debug is confirming the variable is non-empty: test -n "$PAYMENTS_API_KEY" && echo "set".

AppRole: for runners that can't do OIDC

Some CI systems can't issue an OIDC token, an older Jenkins or a self-hosted runner fleet for example. For those, use AppRole, which splits authentication into two pieces. The role_id identifies the role, isn't secret, and is safe to put in a pipeline config. The secret_id is secret, short-lived and generated on demand.

bash
vault auth enable approle
 
vault write auth/approle/role/ci-payments-pipeline \
  token_policies="ci-payments-read" \
  token_ttl=15m \
  token_max_ttl=30m \
  secret_id_ttl=10m \
  secret_id_num_uses=1
 
vault read auth/approle/role/ci-payments-pipeline/role-id

With secret_id_num_uses=1 and secret_id_ttl=10m, a secret_id is single-use and expires within ten minutes. That means it can't be stored in a CI secret field, because the first run would consume it and the next one would fail. Instead, a trusted orchestrator (the Jenkins controller, a deploy service, anything that already has its own Vault identity) generates a fresh one per job and hands it over response-wrapped:

bash
# On the orchestrator: its own token needs "update" on
# auth/approle/role/ci-payments-pipeline/secret-id
WRAP_TOKEN=$(vault write -wrap-ttl=60s -field=wrapping_token \
  -f auth/approle/role/ci-payments-pipeline/secret-id)
# ...inject $WRAP_TOKEN into the job, never into its logs
 
# Inside the job:
SECRET_ID=$(vault unwrap -field=secret_id "$WRAP_TOKEN")
VAULT_TOKEN=$(vault write -field=token auth/approle/login \
  role_id="$VAULT_ROLE_ID" secret_id="$SECRET_ID")
export VAULT_TOKEN

A wrapping token can only be unwrapped once. If the job's unwrap fails because the token was already used, assume the secret_id was intercepted in transit. Even then, the attacker gets read access to one Vault path for fifteen minutes, which is far less damage than root access to production.

Never use Vault's initial root token for day-to-day access, CI included. It bypasses every policy you write. Root tokens exist for bootstrapping and emergency break-glass: generated on demand, revoked immediately after, never stored, not even in Vault itself.

Dynamic secrets: stop handing out the password at all

OIDC and AppRole handle authenticating to Vault. Dynamic secrets go further, because the database password itself shouldn't be static either. With the database secrets engine, Vault creates a short-lived database user for each CI run, backed by a real CREATE ROLE on your Postgres instance.

bash
vault secrets enable database
 
vault write database/config/payments-db \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/payments" \
  allowed_roles="ci-readonly" \
  username="vault-admin" \
  password="..."
 
vault write database/roles/ci-readonly \
  db_name=payments-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="15m" \
  max_ttl="30m"
 
vault read database/creds/ci-readonly

That last command is what CI runs instead of reading a stored password; it gets back a username and password that exist only for this lease, scoped to exactly the grants in creation_statements.

These TTLs match the CI token's on purpose. A lease is a child of the token that requested it. When that Vault token expires (15 minutes, 30 at most) or is revoked, Vault revokes every lease it created, whatever the lease's own TTL says. Setting default_ttl="1h" here would promise a duration Vault can't give a CI caller, and it would mislead whoever reads the role later. Size the database role's TTLs to fit within the token that will request them. If a job needs longer, raise token_ttl on the auth role as well as the database TTL.

Leases and revocation: making "expired" actually mean something

Every dynamic secret Vault issues comes with a lease, a tracked object with a TTL that Vault itself manages, so you can see and control it directly, rather than trusting the pipeline to let go of it.

bash
vault list sys/leases/lookup/database/creds/ci-readonly
 
vault lease revoke -prefix database/creds/ci-readonly
 
vault lease renew database/creds/ci-readonly/abcd1234

Renewing a lease extends it up to the role's max_ttl, but never past the life of the token that created it. When a lease's TTL runs out, or its parent token expires, Vault (with the database plugin) drops the corresponding Postgres role automatically, so the credential is deleted outright. If you suspect a leak, vault lease revoke -prefix revokes every credential issued under that path immediately.

Where this leaves you

A single dev-mode Vault instance is enough to try this before rolling it out further (skip the secrets enable -path=secret step there, since it's already mounted). Once it's in place with an audit device enabled, each credential issuance is logged and any lease can be revoked on demand. The credentials also expire before they have time to go stale.

GO FURTHER

Want to actually run this in production?

This tutorial covers the concepts and architecture. If you want to implement it in your own infrastructure, or get good enough to own this problem long-term, I offer 1:1 mentoring built around your real environment, not a generic course.

This tutorial

  • Core architecture & key concepts
  • Illustrative code snippets
  • The reasoning behind each decision

1:1 mentoring

  • Working sessions on your own environment
  • Direct answers to the edge cases you're hitting
  • Feedback on your actual implementation
  • Ongoing support as you build it out
Get in TouchReply within 1 business day.

Related Tutorials