Pipeline Observability & Feedback Loops

Essential CI/CD metrics

DORA metrics from pipeline data

Platform-native analytics

• GitHub Actions Insights — job duration, billable minutes, workflow usage; identify expensive workflows
• GitLab CI Analytics — pipeline duration trends, DORA metrics dashboard (16.0+), failure rate by stage
• Datadog CI Visibility — pipeline spans, flaky test detection, critical path analysis across jobs
• Self-hosted — Prometheus + Grafana with Pushgateway or OpenTelemetry CI receiver

Export metrics after every pipeline run

      - name: Record pipeline metrics
        if: always()
        env:
          JOB_START: ${{ env.JOB_START_EPOCH }}
        run: |
          END=$(date +%s)
          DURATION=$(( END - JOB_START ))
          STATUS=${{ job.status == 'success' && 1 || 0 }}
          cat <<EOF | curl --data-binary @- http://pushgateway.monitoring:9091/metrics/job/ci
          # TYPE ci_build_duration_seconds gauge
          ci_build_duration_seconds{repo="${{ github.repository }}",workflow="${{ github.workflow }}",branch="${{ github.ref_name }}"} $DURATION
          # TYPE ci_build_success gauge
          ci_build_success{repo="${{ github.repository }}"} $STATUS
          EOF

metrics:export:
  stage: .post
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - DURATION=$(( $(date +%s) - CI_PIPELINE_CREATED_AT ))
    - |
      curl -X POST "$METRICS_WEBHOOK" -H "Content-Type: application/json" -d "{
        \"pipeline_id\": \"$CI_PIPELINE_ID\",
        \"duration_seconds\": $DURATION,
        \"status\": \"$CI_PIPELINE_STATUS\",
        \"project\": \"$CI_PROJECT_PATH\",
        \"deployment_frequency\": 1
      }"
  when: always

flowchart LR
  CI["CI platform\nGHA / GitLab"] --> WH["Webhook / OTel exporter"]
  WH --> PG["Prometheus / Datadog"]
  PG --> GR["Grafana dashboards"]
  PG --> AL["Alerts on regression"]
  CI --> DT["Dependency Track\nCVE trends"]

DevEx metrics that map to pipelines

SPACE framework applied to CI

Actionable failure feedback in PRs

      - name: Upload SARIF
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: trivy.sarif

      - name: Summarize scan on failure
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: '🔴 **Container scan failed.** Open the **Security** tab for CRITICAL/HIGH CVEs. Fix base image or upgrade dependency before merge.'
            })

container_scan:
  stage: scan
  script:
    - trivy image --format template --template "@gitlab.tpl" -o gl-container-scanning-report.json $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  artifacts:
    reports:
      container_scanning: gl-container-scanning-report.json
  allow_failure: false
# GitLab renders findings inline on the merge request Security tab

Local / CI parity checklist

1. Pre-commit runs same linters and secret scanners as CI
2. make ci or npm run ci reproduces unit + lint stages in < 2 min
3. Devcontainer or compose stack matches staging service topology
4. CI failures link to runbook section ("fix SAST finding XYZ")
5. Document required env vars in .env.example—never commit secrets

Deployment markers in observability tools

VERSION=$(git rev-parse --short HEAD)
IMAGE_DIGEST="sha256:abc123..."

# Datadog deployment event
curl -X POST "https://api.datadoghq.com/api/v1/events" \
  -H "DD-API-KEY: $DD_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"title\": \"Deploy api $VERSION\",
    \"text\": \"Digest $IMAGE_DIGEST to production\",
    \"tags\": [\"service:api\",\"env:prod\",\"version:$VERSION\"],
    \"alert_type\": \"info\"
  }"

# Sentry release + deploy
export SENTRY_AUTH_TOKEN=$SENTRY_TOKEN
sentry-cli releases new "$VERSION"
sentry-cli releases set-commits "$VERSION" --auto
sentry-cli releases deploys "$VERSION" new -e production

SLO tracking and deploy freezes

When error budget burn rate exceeds 2× normal, pause production deploys until stability returns. Pipeline gate reads burn rate from Prometheus or Datadog before allowing promote job.

      - name: Check error budget
        run: |
          BURN=$(curl -s "$PROM_URL/api/v1/query?query=error_budget_burn_rate" | jq -r '.data.result[0].value[1]')
          if (( $(echo "$BURN > 2.0" | bc -l) )); then
            echo "Error budget burning too fast — deploy blocked"
            exit 1
          fi

deploy:production:
  stage: deploy
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  before_script:
    - ./scripts/check-error-budget.sh || exit 1
  script:
    - argocd app sync api-prod --revision $CI_COMMIT_SHA

Post-deployment smoke tests

Run minimal E2E against production (or prod-like canary) after every deploy—login, checkout, critical API health. Fail the deploy job or trigger rollback workflow if smoke fails within 5 minutes.

flowchart LR
  D[Deploy canary 10%] --> S[Smoke + metrics 15m]
  S --> A{Analysis pass?}
  A -->|yes| P[Promote 50% → 100%]
  A -->|no| R[Rollback workflow]
  R --> I[Incident + postmortem]
  I --> T[New test in CI]

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{status=~"5..",service="api"}[2m]))
            / sum(rate(http_requests_total{service="api"}[2m]))

Deployment notifications

Every prod deploy posts to Slack with version, digest, deployer, dashboard link, and one-click rollback runbook link.

      - name: Notify Slack
        if: success()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "🚀 *api* deployed to prod\n• SHA: ${{ github.sha }}\n• By: ${{ github.actor }}\n• <https://grafana/d/api|Dashboard> | <https://runbooks/rollback|Rollback>"
            }

Incident timeline data sources

PagerDuty / OpsGenie integration

Enrich alerts with deploy context at creation time. PagerDuty Events API v2 accepts custom_details—populate with last deploy SHA and ArgoCD sync status.

{
  "routing_key": "$PD_ROUTING_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "api error rate > 5% (SLO breach)",
    "severity": "critical",
    "source": "prometheus/alertmanager",
    "custom_details": {
      "last_deploy_sha": "abc123f",
      "last_deploy_time": "2026-06-07T02:14:00Z",
      "last_deploy_actor": "deploy-bot",
      "argocd_app": "api-prod",
      "rollback_runbook": "https://runbooks.example.com/api-rollback"
    }
  }
}

Automated rollback workflow

name: Auto Rollback on SLO Breach
on:
  repository_dispatch:
    types: [slo-breach]

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: GitOps rollback (revert manifest)
        run: |
          git revert HEAD --no-edit
          git push origin main
      - name: Notify incident channel
        run: |
          curl -X POST "$SLACK_WEBHOOK" \
            -H "Content-Type: application/json" \
            -d '{"text":"⏪ Auto-rollback: SLO breach on api. Reverted GitOps manifest."}'
      - name: Annotate PagerDuty incident
        run: |
          curl -X POST "https://api.pagerduty.com/incidents/$INCIDENT_ID/notes" \
            -H "Authorization: Token token=$PD_TOKEN" \
            -H "Content-Type: application/json" \
            -d '{"note":{"content":"Automated GitOps rollback executed"}}'

rollback:production:
  stage: deploy
  when: manual
  rules:
    - if: $ROLLBACK == "true"
  script:
    - argocd app rollback api-prod
    - curl -X POST "$SLACK_WEBHOOK" -d '{"text":"Rollback triggered for api-prod"}'
  environment:
    name: production
    action: stop

On-call runbook (pipeline-aware)

Postmortem → pipeline improvements

• Missing test — add integration or E2E test that would have caught the failure; block merge if removed
• Missing deploy marker — automate Datadog/Sentry annotation in deploy job
• Slow rollback — practice GitOps revert in game day; measure MTTR monthly
• Config drift — ArgoCD diff alerts; Conftest policy in CI for manifest changes
• Security regression — new Semgrep rule or policy gate from incident root cause

Chaos engineering in the delivery path

Schedule chaos experiments in staging after promote, not only in production fire drills. LitmusChaos or Chaos Mesh validates that rollback and circuit breakers work before you need them at 3 AM.

• Inject pod kill during canary analysis—does Argo Rollouts abort?
• Simulate registry outage during deploy—does pipeline fail cleanly without partial state?
• Game day: disable primary runner pool—does queue alerting fire before devs notice?