Secrets Management

flowchart TB
  DEV["Developer laptop
.env file"] --> GIT["Accidental git commit"]
  CI["CI variables
long-lived AWS key"] --> LEAK["Log exposure / fork PR"]
  K8S["K8s Secret
base64 in etcd"] --> RBAC["Over-broad RBAC read"]
  GIT --> BOT["Bot harvests in minutes"]
  LEAK --> LATERAL["Lateral movement to prod"]

name: Secret baseline
on: [push]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: gitleaks detect --source . --verbose --redact

secret-audit:
  stage: test
  image: zricethezav/gitleaks:latest
  script:
    - gitleaks detect --source . --verbose --redact

Where secrets leak

Best practices

• Prefer dynamic secrets with TTL under 1 hour for databases and cloud APIs.
• Never pass secrets as CLI arguments—visible in process list and shell history.
• Use environment: protection rules for production credentials.
• Audit secret access monthly; alert on anomalous read patterns.
• Document rotation runbooks with RTO targets; test twice per year.

Anti-patterns

1. Shared root AWS keys in a team 1Password folder.
2. Same staging and production DB password "for simplicity".
3. Mounting K8s Secrets as env vars in 50 microservices—blast radius on one compromise.
4. Disabling fork PR pipelines entirely instead of scoped unprivileged workflows.

Zero-trust secret lifecycle

Treat every secret as time-bounded: creation → scoped use → audit → rotation → revocation. Long-lived credentials violate zero trust because compromise detection depends on luck, not architecture.

name: Vault in CI
on: [push]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: hashicorp/vault-action@v2
        with:
          url: https://vault.example.com
          method: jwt
          role: github-actions-deploy
          secrets: |
            secret/data/prod/db password | DB_PASSWORD
      - run: deploy.sh
        env:
          DB_PASSWORD: ${{ env.DB_PASSWORD }}

deploy:
  stage: deploy
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.example.com
  script:
    - export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=gitlab-deploy jwt=$VAULT_ID_TOKEN)
    - export DB_PASSWORD=$(vault kv get -field=password secret/prod/db)
    - ./deploy.sh
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Best practices

• Prefer dynamic secrets with TTL under 1 hour for databases and cloud APIs.
• Never pass secrets as CLI arguments—visible in process list and shell history.
• Use environment: protection rules for production credentials.
• Audit secret access monthly; alert on anomalous read patterns.
• Document rotation runbooks with RTO targets; test twice per year.

Anti-patterns

1. Shared root AWS keys in a team 1Password folder.
2. Same staging and production DB password "for simplicity".
3. Mounting K8s Secrets as env vars in 50 microservices—blast radius on one compromise.
4. Disabling fork PR pipelines entirely instead of scoped unprivileged workflows.

Vault policy example

# payments-api can read only its DB creds
path "database/creds/payments-role" {
  capabilities = ["read"]
}
path "secret/data/payments/*" {
  capabilities = ["read"]
}
# deny all else implicitly

Dynamic database credentials

Vault's database secrets engine creates per-session SQL users with TTL. When the app disconnects or TTL expires, Vault revokes the user—stolen creds stop working without a global password rotation.

name: OIDC to AWS
on: [push]
permissions:
  id-token: write
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-deploy
          aws-region: us-east-1
      - run: aws secretsmanager get-secret-value --secret-id prod/db --query SecretString

deploy-aws:
  stage: deploy
  id_tokens:
    AWS_TOKEN:
      aud: sts.amazonaws.com
  script:
    - |
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
        $(aws sts assume-role-with-web-identity \
          --role-arn $AWS_ROLE_ARN \
          --web-identity-token $AWS_TOKEN \
          --role-session-name gitlab-ci \
          --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
          --output text))
    - aws secretsmanager get-secret-value --secret-id prod/db

Best practices

• Prefer dynamic secrets with TTL under 1 hour for databases and cloud APIs.
• Never pass secrets as CLI arguments—visible in process list and shell history.
• Use environment: protection rules for production credentials.
• Audit secret access monthly; alert on anomalous read patterns.
• Document rotation runbooks with RTO targets; test twice per year.

Anti-patterns

1. Shared root AWS keys in a team 1Password folder.
2. Same staging and production DB password "for simplicity".
3. Mounting K8s Secrets as env vars in 50 microservices—blast radius on one compromise.
4. Disabling fork PR pipelines entirely instead of scoped unprivileged workflows.

Cloud manager comparison

GCP Workload Identity Federation sketch

# GCP: pool provider maps GitHub OIDC iss/sub to service account
# No JSON key file in CI — ever
attributeMapping:
  google.subject: assertion.sub
  attribute.repository: assertion.repository
serviceAccountImpersonation:
  serviceAccount: [email protected]

External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: ClusterSecretStore
  target:
    name: app-db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/database
        property: password

Sealed Secrets for GitOps

# Sealed Secrets — encrypt Secret manifests for GitOps
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: app-db
spec:
  encryptedData:
    password: AgBx...sealed...blob

name: Validate ExternalSecrets
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-kubectl@v4
      - run: kubectl apply --dry-run=server -f k8s/external-secrets/

validate-secrets:
  stage: test
  image: bitnami/kubectl:latest
  script:
    - kubectl apply --dry-run=server -f k8s/external-secrets/
  rules:
    - changes:
        - k8s/external-secrets/**/*

Best practices

• Prefer dynamic secrets with TTL under 1 hour for databases and cloud APIs.
• Never pass secrets as CLI arguments—visible in process list and shell history.
• Use environment: protection rules for production credentials.
• Audit secret access monthly; alert on anomalous read patterns.
• Document rotation runbooks with RTO targets; test twice per year.

Anti-patterns

1. Shared root AWS keys in a team 1Password folder.
2. Same staging and production DB password "for simplicity".
3. Mounting K8s Secrets as env vars in 50 microservices—blast radius on one compromise.
4. Disabling fork PR pipelines entirely instead of scoped unprivileged workflows.

Encryption at rest

Enable the Kubernetes EncryptionConfiguration API with a KMS provider (AWS KMS, GCP Cloud KMS, Vault transit). Without it, anyone with etcd backup access reads Secrets in plaintext.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - kms:
          name: aws-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}

CSI vs ESO

• External Secrets Operator — syncs to native Secret objects; familiar to apps using envFrom.
• Secrets Store CSI — mounts secrets as volumes; no Secret object in etcd; better for zero-k8s-secret footprint.
• SOPS + GitOps — Mozilla SOPS encrypts YAML in git; Flux/Argo decrypt at apply time with KMS.

name: Safe CI secrets
on: [pull_request]
jobs:
  test:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - run: npm test
        env:
          API_KEY: ${{ secrets.STAGING_API_KEY }}
# Never echo secrets; use env not args to avoid ps leakage

unit-test:
  script:
    - npm test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: never
    - when: on_success
  # MR pipelines from forks skip secrets — use separate workflow

Best practices

• Prefer dynamic secrets with TTL under 1 hour for databases and cloud APIs.
• Never pass secrets as CLI arguments—visible in process list and shell history.
• Use environment: protection rules for production credentials.
• Audit secret access monthly; alert on anomalous read patterns.
• Document rotation runbooks with RTO targets; test twice per year.

Anti-patterns

1. Shared root AWS keys in a team 1Password folder.
2. Same staging and production DB password "for simplicity".
3. Mounting K8s Secrets as env vars in 50 microservices—blast radius on one compromise.
4. Disabling fork PR pipelines entirely instead of scoped unprivileged workflows.

GitHub vs GitLab secret models

name: Secret hygiene
on: [push]
permissions:
  contents: read
  id-token: write
jobs:
  oidc-deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_DEPLOY_ROLE }}
          aws-region: us-east-1
      - run: ./deploy.sh  # no static keys in repo or vars

include:
  - local: .gitlab/ci/secret-scan.yml

production-deploy:
  stage: deploy
  environment:
    name: production
  id_tokens:
    AWS_TOKEN:
      aud: sts.amazonaws.com
  script:
    - ./deploy.sh
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  when: manual

Best practices

• Prefer dynamic secrets with TTL under 1 hour for databases and cloud APIs.
• Never pass secrets as CLI arguments—visible in process list and shell history.
• Use environment: protection rules for production credentials.
• Audit secret access monthly; alert on anomalous read patterns.
• Document rotation runbooks with RTO targets; test twice per year.

Anti-patterns

1. Shared root AWS keys in a team 1Password folder.
2. Same staging and production DB password "for simplicity".
3. Mounting K8s Secrets as env vars in 50 microservices—blast radius on one compromise.
4. Disabling fork PR pipelines entirely instead of scoped unprivileged workflows.

Break-glass rotation runbook

1. Identify compromised secret scope (which systems, which repos).
2. Revoke in source system (Vault lease revoke, AWS key deactivate, DB user drop).
3. Issue new credential with narrower policy if root cause was over-permission.
4. Redeploy all consumers—stale env vars in running pods keep old secret alive.
5. Post-incident: add detection rule (CloudTrail anomaly, failed auth spike).

sequenceDiagram
  participant CI as CI runner
  participant IdP as GitHub/GitLab OIDC
  participant Cloud as AWS/GCP/Azure
  participant SM as Secret manager
  CI->>IdP: Request OIDC token (aud scoped)
  IdP->>CI: JWT (sub, repo, ref)
  CI->>Cloud: AssumeRoleWithWebIdentity
  Cloud->>CI: Session creds (15m–1h)
  CI->>SM: GetSecret / Vault read
  SM->>CI: Short-lived DB password

Audit checklist

• Inventory all secrets touched by CI—spreadsheet is fine, CMDB is better.
• Zero static cloud root keys in CI variables (grep org for AKIA patterns).
• Environment protection: required reviewers on production deploy jobs.
• Secret scanning on .github/ and .gitlab-ci.yml in pre-commit.

flowchart TD
  Q1{"Need secrets in git?"}
  Q1 -->|Yes encrypted| SOPS["SOPS + KMS + GitOps"]
  Q1 -->|No| Q2{"K8s workload?"}
  Q2 -->|Yes| ESO["ESO / CSI Driver"]
  Q2 -->|No| Q3{"CI pipeline?"}
  Q3 -->|Yes| OIDC["OIDC federation\nno static keys"]
  Q3 -->|No| VAULT["Vault / Cloud SM API"]
  ESO --> VAULT
  OIDC --> VAULT

name: Org secret standards
on:
  schedule:
    - cron: '0 6 * * 1'
jobs:
  audit-static-keys:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          ! grep -rE 'AKIA[0-9A-Z]{16}' .github/ || (echo "Static AWS key in workflow" && exit 1)
      - run: gitleaks detect --source . --redact

weekly-secret-audit:
  stage: audit
  script:
    - gitleaks detect --source . --redact
    - echo "Review protected variables and OIDC audiences quarterly"
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

30-day rollout plan

1. Week 1 — Enable gitleaks on all repos; inventory CI variables.
2. Week 2 — Migrate one service to OIDC deploy; delete static AWS key.
3. Week 3 — Deploy ESO; migrate one K8s Secret to ExternalSecret.
4. Week 4 — Run rotation game-day; document MTTD and gaps.

Maturity model