Environment Management

Core principles

• Parity where it matters — staging mirrors prod topology (ingress, DB engine, message broker) even if scaled down
• Immutability across promotion — promote signed image digests, not rebuilt tags
• Least privilege per env — prod credentials never exist in dev CI variables
• Explicit data classification — each environment declares what data classes it may hold
• Cost-aware sprawl control — ephemeral envs auto-destroy; long-lived envs have owners and budgets

Environment matrix

Naming conventions

flowchart LR
  subgraph dev["Development"]
    L[local] --> D[dev]
  end
  subgraph preprod["Pre-production"]
    D --> S[staging]
    S --> U[UAT]
  end
  subgraph live["Live"]
    U --> P[prod]
    P --> DR[dr standby]
  end
  PR[review app] -.->|merge| D
  style P fill:#7c3aed,color:#fff
  style PR fill:#38bdf8,color:#000

Anti-patterns to avoid

Decision checklist

1. How many environments does your compliance framework require? (SOC2 often needs staging + prod separation)
2. Can you afford full prod parity or only critical-path parity?
3. Who owns each long-lived environment? (RACI: platform provisions, app team configures)
4. What is your RTO/RPO for DR—does DR need warm standby or cold restore?
5. Do you need regional environments (EU data residency) or single-region with regional overlays?

Lifecycle

1. PR opened — CI builds artifact, deploys to pr-{number} namespace
2. DNS/TLS — wildcard cert + ingress rule: pr-1842.preview.example.com
3. Dependencies — spin ephemeral Postgres (CloudNativePG) or wire to shared staging services
4. Comment bot — post preview URL on PR for designer/QA access
5. PR updated — redeploy same namespace, rolling update
6. PR merged/closed — teardown namespace, delete DNS, purge secrets

sequenceDiagram
  participant Dev as Developer
  participant CI as CI Pipeline
  participant K8s as Kubernetes
  participant Bot as PR Bot
  Dev->>CI: Push branch / open PR
  CI->>CI: Build + scan image
  CI->>K8s: Helm upgrade --install pr-1842
  K8s-->>CI: Ingress URL ready
  CI->>Bot: Post preview link
  Bot-->>Dev: Comment on PR
  Dev->>K8s: QA on preview URL
  Dev->>CI: Merge PR
  CI->>K8s: helm uninstall pr-1842

GitHub Actions — review app deploy

name: Review App
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
permissions:
  contents: read
  id-token: write
  pull-requests: write
env:
  PREVIEW_NS: pr-${{ github.event.pull_request.number }}
  PREVIEW_HOST: pr-${{ github.event.pull_request.number }}.preview.example.com
jobs:
  deploy:
    if: github.event.action != 'closed'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-preview
          aws-region: us-east-1
      - name: Build and push
        run: |
          docker build -t $ECR_REPO:${{ github.sha }} .
          docker push $ECR_REPO:${{ github.sha }}
      - name: Deploy preview
        run: |
          helm upgrade --install $PREVIEW_NS ./chart \
            --namespace $PREVIEW_NS --create-namespace \
            --set image.tag=${{ github.sha }} \
            --set ingress.host=$PREVIEW_HOST \
            --wait --timeout 5m
      - name: Comment preview URL
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({{
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '🚀 Preview: https://${{ env.PREVIEW_HOST }}'
            }})
  teardown:
    if: github.event.action == 'closed'
    runs-on: ubuntu-latest
    steps:
      - run: helm uninstall $PREVIEW_NS -n $PREVIEW_NS || true
      - run: kubectl delete namespace $PREVIEW_NS --wait=false

GitLab CI — review app with environments

review:deploy:
  stage: deploy
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  environment:
    name: review/$CI_MERGE_REQUEST_IID
    url: https://mr-$CI_MERGE_REQUEST_IID.preview.example.com
    on_stop: review:stop
    auto_stop_in: 3 days
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - helm upgrade --install mr-$CI_MERGE_REQUEST_IID ./chart
        --set image.tag=$CI_COMMIT_SHA
        --namespace mr-$CI_MERGE_REQUEST_IID --create-namespace

review:stop:
  stage: deploy
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual
    - if: $CI_MERGE_REQUEST_EVENT_TYPE == "close"
  environment:
    name: review/$CI_MERGE_REQUEST_IID
    action: stop
  script:
    - helm uninstall mr-$CI_MERGE_REQUEST_IID -n mr-$CI_MERGE_REQUEST_IID
    - kubectl delete namespace mr-$CI_MERGE_REQUEST_IID

Cost and resource controls

Database strategies for previews

Promotion flow

flowchart TD
  A[Build + test + scan] --> B[Push to registry]
  B --> C{Scan pass?}
  C -->|no| X[Block pipeline]
  C -->|yes| D[Sign with Cosign]
  D --> E[Deploy staging]
  E --> F{E2E + perf OK?}
  F -->|no| X
  F -->|yes| G[Manual/auto approval]
  G --> H[Deploy prod same digest]
  H --> I[Deployment marker in APM]

Registry promotion models

GitHub Actions — promote to production

name: Promote to Production
on:
  workflow_dispatch:
    inputs:
      image_digest:
        description: 'sha256 digest verified in staging'
        required: true
      release_notes:
        required: true
jobs:
  verify-staging:
    runs-on: ubuntu-latest
    steps:
      - name: Confirm digest deployed in staging
        run: |
          DEPLOYED=$(kubectl -n staging get deploy api -o jsonpath='{{.spec.template.spec.containers[0].image}}')
          echo "Staging runs: $DEPLOYED"
          [[ "$DEPLOYED" == *"${{ inputs.image_digest }}"* ]] || exit 1
  promote:
    needs: verify-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - name: Update prod manifest
        run: |
          yq -i '.image.digest = "${{ inputs.image_digest }}"' deploy/overlays/prod/values.yaml
          git config user.name "github-actions[bot]"
          git commit -am "promote: ${{ inputs.image_digest }}"
          git push
      - name: Wait for ArgoCD sync
        run: argocd app wait api-prod --health --timeout 600

GitLab — environment-scoped deploy with protected environments

stages: [build, staging, prod]

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - cosign sign --yes $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy:staging:
  stage: staging
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - helm upgrade api ./chart --set image.tag=$CI_COMMIT_SHA -n staging
  needs: [build]

deploy:production:
  stage: prod
  environment:
    name: production
    url: https://api.example.com
  script:
    - helm upgrade api ./chart --set image.tag=$CI_COMMIT_SHA -n production
  needs: [deploy:staging]
  when: manual
  only:
    - main

Approval gates and compliance

Rollback strategy

• Git revert — revert the promotion commit; ArgoCD syncs previous digest (fastest for GitOps)
• Helm rollback — helm rollback api 42 restores previous release revision
• Flag flip — disable feature in LaunchDarkly while digest rollback proceeds
• Database — forward-only migrations require compatibility window; never auto-rollback schema

Configuration layers

Helm values hierarchy

replicaCount: 2
image:
  repository: 123456789012.dkr.ecr.us-east-1.amazonaws.com/api
  tag: ""  # set by CI
  pullPolicy: IfNotPresent
resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi
env:
  LOG_LEVEL: info
  OTEL_EXPORTER_OTLP_ENDPOINT: ""  # overlay per env

replicaCount: 1
env:
  LOG_LEVEL: debug
  OTEL_EXPORTER_OTLP_ENDPOINT: https://otel-staging.internal:4317
  PAYMENT_API_URL: https://sandbox.payments.example.com
ingress:
  host: api-staging.example.com
  tls:
    secretName: api-staging-tls

replicaCount: 6
resources:
  requests:
    cpu: 500m
    memory: 1Gi
env:
  LOG_LEVEL: warn
  OTEL_EXPORTER_OTLP_ENDPOINT: https://otel-prod.internal:4317
  PAYMENT_API_URL: https://api.payments.example.com
ingress:
  host: api.example.com
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
podDisruptionBudget:
  minAvailable: 4

Kustomize overlay structure

deploy/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml    # bases: ../../base + patches
    │   ├── replica-patch.yaml
    │   └── ingress-staging.yaml
    └── production/
        ├── kustomization.yaml
        ├── hpa.yaml
        └── ingress-prod.yaml

External Secrets Operator pattern

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: api-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: prod/api/database-url
    - secretKey: API_KEY
      remoteRef:
        key: prod/api/payment-key
        property: current

Config drift detection

Feature flags vs environment config

• Environment config — connection strings, replica counts, ingress hosts (changes rarely, reviewed in Git)
• Feature flags — user-facing behavior toggles (changes frequently, owned by product, no redeploy)
• Use LaunchDarkly/Unleash/Flagsmith for flags; never conflate with Helm values unless the flag is deploy-gating (e.g. enable canary)

GitOps repository layout

Organize environment config so ArgoCD ApplicationSets can discover overlays without custom glue code.

gitops/
├── apps/
│   ├── api/
│   │   ├── base/                 # shared Deployment, Service, SA
│   │   └── overlays/
│   │       ├── dev/
│   │       ├── staging/
│   │       ├── prod-us-east/
│   │       └── prod-eu-west/
│   └── worker/
├── clusters/
│   ├── staging-eks/              # cluster-specific bootstrap
│   └── prod-eks/
├── policies/                     # Kyverno / OPA bundles
└── environments/
    ├── staging.yaml              # ArgoCD ApplicationSet generator input
    └── production.yaml

Multi-region and data residency

Environment access control (RBAC)

Cost optimization checklist

1. Auto-stop preview environments after 72h idle (GitLab auto_stop_in or CronJob)
2. Scale staging to zero replicas nights/weekends if no scheduled tests
3. Right-size staging nodes—do not run prod-sized instances for 10% traffic
4. Use spot/preemptible nodes for dev and preview tiers
5. Tag all cloud resources with env, team, cost-center for chargeback
6. Review monthly: count orphaned namespaces, unused PVCs, stale preview DNS records

ArgoCD ApplicationSet — preview generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: api-previews
  namespace: argocd
spec:
  generators:
    - pullRequest:
        github:
          owner: org
          repo: api
          labels: [preview]
        requeueAfterSeconds: 60
  template:
    metadata:
      name: 'api-pr-{{number}}'
    spec:
      project: previews
      source:
        repoURL: https://github.com/org/api.git
        targetRevision: '{{head_sha}}'
        path: deploy/overlays/preview
        helm:
          parameters:
            - name: ingress.host
              value: 'pr-{{number}}.preview.example.com'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'pr-{{number}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true