Practical Secrets Management for Cloud CI/CD: Rotation Playbooks & Templates for API Keys, JWTs, and OAuth Tokens

Overview

Secrets include any credential that could grant access to an application, service, account, or data source. Examples include API keys, database passwords, private signing keys, OAuth client secrets, refresh tokens, webhook signing secrets, and deployment credentials. Treating all of them as ordinary environment variables creates avoidable risk: values can appear in logs, persist in build artifacts, remain available to too many jobs, or become impossible to rotate safely.

A workable secrets-management process answers five questions for every credential:

• Where is it stored? Use an approved secrets manager or the CI platform's encrypted secret store rather than source control, issue comments, container images, or local configuration files committed to a repository.
• Who or what can use it? Grant access to a specific workflow, environment, service account, or deployment role. Avoid broad organization-wide exposure.
• When is it available? Inject a secret only into the job or process that needs it, preferably for the shortest practical duration.
• How is it rotated? Document the owner, replacement steps, overlap period, validation test, and revocation step.
• How is recovery handled? Keep a tested emergency procedure that can stop use of a compromised value and restore service without committing a replacement secret to code.

Use this article as a CI secret rotation playbook. Adapt names, permissions, commands, and review requirements to your platform and organization. Commands are examples; verify them against the current documentation for the provider you use.

Checklist by scenario

Scenario 1: GitHub Actions with repository or environment secrets

GitHub Actions can consume encrypted repository, organization, or environment secrets. The safest choice depends on scope. A credential used only by production should generally be associated with a protected production environment rather than exposed to every branch and workflow.

1. Classify the credential and name it consistently, such as PAYMENTS_API_KEY or OAUTH_CLIENT_SECRET. Do not include the actual value in the name.
2. Create the secret in the appropriate repository, organization, or environment scope. Require approvals or branch restrictions for sensitive deployment environments where available.
3. Reference it through the workflow secret context, not by hard-coding it in YAML. For example:

   steps:
     - name: Deploy
       run: ./scripts/deploy.sh
       env:
         API_KEY: ${{ secrets.PAYMENTS_API_KEY }}

4. Prevent accidental disclosure. Do not print the variable, include it in debugging output, or pass it to commands that echo arguments. Be cautious with shell tracing such as set -x.
5. Review pull-request behavior. A workflow triggered by untrusted changes should not receive production secrets unless the execution path and approval model explicitly make that safe.
6. Rotate by creating a replacement value at the provider, updating the GitHub secret, running a deployment or health check, and revoking the old value only after the replacement is confirmed.

A simple rotation record should contain: secret name, service owner, current provider credential ID if available, last rotation date, next review date, test command or endpoint, rollback condition, and revocation confirmation. Never place the secret value in that record.

Scenario 2: GitLab CI with protected variables or Vault

For smaller projects, GitLab CI/CD variables can provide a practical encrypted storage layer. Mark sensitive variables as protected when they should be available only to protected branches or tags, and mask them where the platform supports masking. For larger or multi-project environments, a centralized system such as Vault can provide more consistent access control and auditing.

For native CI variables, use a minimal job scope:

deploy:
  script:
    - ./scripts/deploy.sh
  variables:
    DEPLOY_ENV: production
  rules:
    - if: '$CI_COMMIT_TAG'

Keep the actual secret value in the CI variable settings rather than in .gitlab-ci.yml. For a Vault workflow, prefer short-lived authentication and scoped paths. A generic pattern is:

vault kv get -field=api_key secret/apps/example
# Pass the returned value directly to the deployment process where possible.
# Do not write it to a tracked file or print it in the job log.

Before adopting this pattern, define how the runner authenticates to Vault, what policy it receives, how leases expire, and what happens when Vault is unavailable. The pipeline should fail safely rather than substitute an empty or stale credential without notice.

Scenario 3: AWS or Google Cloud with a managed secrets service

For cloud-native applications, store runtime credentials in a managed service and let workloads authenticate through an identity role or service account. This avoids distributing a long-lived master credential to every pipeline job. Separate deployment-time secrets from application runtime secrets when their owners and access patterns differ.

An AWS-oriented workflow might create or update a secret through the CLI:

aws secretsmanager create-secret \
  --name apps/example/payments-api \
  --secret-string file://secret-payload.json

For an existing secret, the update operation may be used according to the provider's current CLI syntax:

aws secretsmanager put-secret-value \
  --secret-id apps/example/payments-api \
  --secret-string file://secret-payload.json

Keep secret-payload.json outside the repository, restrict its filesystem permissions, and remove the temporary file after use. In a mature workflow, the pipeline identity should be allowed to update only the specific secret paths it owns, while the application identity should have read access only to the values it needs.

On Google Cloud, the equivalent pattern uses Secret Manager and a service account with narrowly scoped permissions. A representative command is:

gcloud secrets versions add example-payments-api \
  --data-file=secret-payload.json

Use versioning to support controlled rollout and rollback. Validate that the new version is available to the intended workload, then disable or destroy the old version according to your retention and incident procedures. Avoid assuming that a successful upload means the application has successfully adopted the new value.

What to double-check

• Secret type: API keys may be revoked and recreated, while JWT signing keys often require a key identifier, public-key distribution, and an overlap period. OAuth access tokens are usually short-lived; refresh tokens and client secrets need different handling.
• Rotation compatibility: Confirm whether the provider supports two active credentials. If it does, create the new value first, deploy it, test it, and revoke the old value second. If it does not, schedule a controlled maintenance window or use a fallback credential path.
• Identity permissions: Check both the CI runner's permissions and the deployed application's permissions. A pipeline that can read every production secret is harder to contain than one limited to a named application.
• Log and artifact hygiene: Review command arguments, test reports, crash dumps, build artifacts, cache directories, and notification integrations. A masked value can still be exposed through transformations, partial output, or an encoded representation.
• Token audience and scope: For OAuth and JWT workflows, validate issuer, audience, expiration, scopes, and signing algorithm expectations. A token decoder can help inspect non-sensitive claims, but never paste live credentials or private tokens into an untrusted online tool. See the JWT decoder and debugging guide for a safer inspection workflow.
• Application reload behavior: Determine whether a service reads the secret at startup, per request, or through a sidecar or mounted file. Rotation is incomplete until the running workload uses the replacement.
• Contract and deployment checks: If a rotated credential changes API configuration, test the request path and schema as part of deployment. The OpenAPI schema validation guide can help keep API contracts aligned.

Common mistakes

Committing a secret temporarily

A value committed and deleted can remain in repository history, forks, caches, or build artifacts. Revoke it immediately, investigate access, remove it from reachable history when appropriate, and replace it through the approved store. History cleanup is not a substitute for revocation.

Rotating storage but not the provider credential

Updating a CI variable does not invalidate an API key. Rotation has four distinct actions: create or issue the replacement, distribute it, verify usage, and revoke the old value. Record each step.

Using one credential everywhere

A shared key across development, staging, production, and multiple services expands the impact of one leak and makes diagnosis difficult. Prefer separate credentials by environment, application, and purpose where the provider supports it.

Testing with real production tokens

Use dedicated test credentials, synthetic data, and restricted scopes. When a test must reach a real service, limit the target, duration, permissions, and cleanup procedure.

Making emergency rotation depend on the normal pipeline

If the pipeline itself may be compromised, its credentials may be unavailable or untrustworthy. Maintain a break-glass procedure using a separately controlled administrator path, with approval, logging, and post-incident review.

When to revisit

Review this checklist before seasonal planning cycles, major release periods, audits, and ownership changes. Revisit it immediately when a workflow, runner, cloud account, identity model, secret provider, or deployment architecture changes. Platform interfaces and CLI commands can change, so verify examples before copying them into automation.

Use a recurring review to confirm:

• Every production secret has an owner, purpose, scope, and documented rotation method.
• Rotation dates and expiration dates are tracked without recording secret values.
• At least one non-production rotation has been rehearsed recently.
• Emergency contacts and break-glass access are current and separately protected.
• Logs, artifacts, caches, and notifications do not expose credentials.
• Old keys, revoked token versions, and unused CI variables have been removed.

For an emergency rotation, follow this order: identify the exposed credential and affected systems; preserve relevant evidence without copying the secret; disable or revoke the old value when safe; issue a replacement; update the approved secret store; deploy or reload dependent services; test authentication and critical paths; monitor for failures or continued use of the old value; then document the timeline and preventive actions. Keep this runbook accessible to the responders who may need it, but do not store it alongside the credentials it describes.

Finally, pair secrets management with the rest of your cloud-native workflow. Use an API testing workflow to verify authentication changes, and review the Dockerfile security checklist to ensure secrets are not copied into images or build layers. A small, tested rotation process is more reliable than a large policy document that nobody can execute during a deployment or incident.