CI Pipeline Design

A CI pipeline is the assembly line of software delivery—every stage exists to catch a class of defect before it reaches production. This chapter maps pipeline anatomy, platform-specific YAML for GitHub Actions and GitLab CI, performance optimization, reusable templates, and Jenkins migration—with threat models, security gates, and trade-offs at each layer.

developer devops security DORA 2023 SLSA L2 OIDC

Pipeline Anatomy

Every mature CI pipeline decomposes into predictable stages. Each stage answers a question: Is this code safe to merge? Does it build? Does it behave? Is the artifact trustworthy? Can we deploy it? Security is not a single stage—it is woven through every gate.

The canonical stage model

Think of the pipeline as a factory line. A defect caught at lint costs seconds to fix; the same defect caught in production costs hours of incident response, customer trust, and regulatory scrutiny. DevSecOps shifts expensive checks left while keeping deploy-time verification for what only runtime can prove.

Stage Purpose Security touchpoint Typical failure mode
Trigger Event → workflow (push, PR, tag, schedule) Branch protection, signed commits, required checks Direct push to main bypasses all gates
Lint & format Style, static analysis, secret scan in diff Gitleaks, Semgrep, ESLint security plugins Secrets committed; SQL injection patterns undetected
Build Compile, bundle, produce OCI image Reproducible builds, pinned base images, SBOM gen Dependency confusion; malicious base image layer
Test Unit, integration, contract, smoke Test data sanitization; no prod creds in CI Flaky tests → developers skip CI → vulns ship
Scan SAST, SCA, container, IaC misconfiguration SARIF upload, policy thresholds, license compliance Scan runs but never blocks; alert fatigue
Publish Push artifact to immutable registry Cosign sign, SLSA provenance, promotion rules Unsigned image deployed; tag mutability exploited
Deploy Promote to environment via GitOps or CD OIDC short-lived creds, admission policy verify Long-lived AWS keys in CI variables leaked

Threat → solution mapping

Attackers target the pipeline because it has write access to production. The threat model below maps common attack vectors to the stage where you stop them—and the control that actually works.

Threat Attack vector Pipeline control Stage
Credential theft Exfiltrate GITHUB_TOKEN or cloud keys from logs/env OIDC federation, minimal permissions:, masked variables Trigger / Deploy
Poisoned pipeline Malicious PR modifies workflow to exfil secrets Require approval for workflow changes; pull_request_target ban Trigger
Dependency confusion Typosquat package resolves before internal mirror Lockfiles, private registry proxy, SCA with block policy Build / Scan
Artifact substitution Replace signed image with unsigned copy at deploy Cosign verify in admission; digest-pinned manifests Publish / Deploy
Runner compromise Shared runner retains state between jobs Ephemeral runners, isolated namespaces, no root All
flowchart LR
  T[Trigger\npush / PR] --> L[Lint & secrets]
  L --> B[Build]
  B --> TE[Test]
  TE --> S[Scan\nSAST SCA container]
  S --> P[Publish\nsign + SBOM]
  P --> D[Deploy\nOIDC + policy]
  subgraph gates["Security gates"]
    L
    S
    P
  end
  D --> F[Feedback loop\nDORA metrics]

Reference pipeline YAML

The dual-platform blocks below show a minimal but secure pipeline skeleton. Toggle platform in the toolbar— preference persists via localStorage.

.github/workflows/ci.yml
name: CI Pipeline
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write
  security-events: write

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Secret scan
        uses: gitleaks/gitleaks-action@v2
      - name: Lint
        run: npm ci && npm run lint

  build-test:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci && npm test -- --coverage
      - run: docker build -t app:${{ github.sha }} .

  security-scan:
    needs: build-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: SAST
        run: semgrep scan --config=auto --sarif -o semgrep.sarif
      - name: Container scan
        run: trivy image --exit-code 1 --severity CRITICAL,HIGH app:${{ github.sha }}
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: semgrep.sarif
.gitlab-ci.yml
stages: [lint, build, test, scan, publish]

variables:
  DOCKER_DRIVER: overlay2

lint:
  stage: lint
  script:
    - gitleaks detect --source . --verbose
    - npm ci && npm run lint

build:
  stage: build
  script:
    - npm ci
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
  artifacts:
    expire_in: 1 day

unit-test:
  stage: test
  script:
    - npm test -- --coverage
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

security-scan:
  stage: scan
  script:
    - semgrep scan --config=auto --sarif -o gl-sast-report.sarif
    - trivy image --exit-code 1 --severity CRITICAL,HIGH $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  artifacts:
    reports:
      sast: gl-sast-report.sarif

publish:
  stage: publish
  script:
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
terminal — inspect pipeline runs
$ gh run list --workflow=ci.yml --limit 5
→ STATUS  CONCLUSION  WORKFLOW  BRANCH  EVENT
$ gh run view 12345678 --log-failed
$ glab ci list --per-page 10
$ glab ci trace --job-id 987654
→ Failed step logs surface scan exit codes and missing artifacts
🔒 Security

Default permissions: on GitHub is overly broad for fork PRs. Set explicit read-only scopes at workflow level; grant write only to jobs that upload SARIF or assume cloud roles via OIDC. Never use pull_request_target to run untrusted code from the PR branch.

⚠️ Pitfall

Running security scans only on main pushes means every merged PR skipped the gate. Scans must run on pull requests before merge; main-branch scans catch drift and supply-chain changes between merge and deploy.

⚖️ Trade-off

Blocking vs warning gates: blocking every HIGH CVE freezes delivery; warning-only gates train developers to ignore alerts. Mature teams block CRITICAL + exploitable in prod path, warn on everything else, and track mean-time-to-remediate as a DORA-adjacent metric.

GitHub Actions

GitHub Actions embeds CI/CD in the repository: workflows are YAML in .github/workflows/, runners are ephemeral VMs or self-hosted agents, and the platform provides native OIDC to AWS, Azure, and GCP without long-lived keys.

Core primitives

Concept What it is Security note
Workflow Top-level YAML triggered by events Workflow changes in PRs need CODEOWNERS + required review
Job Parallel unit of work; own runner Isolate privileged jobs (signing) from untrusted build jobs
Step Sequential command or action invocation Pin actions to SHA, not floating @v4 tags
Action Reusable step bundle (composite or Docker) Third-party actions run with your GITHUB_TOKEN
Environment Named deploy target with protection rules Required reviewers + wait timer for production
OIDC JWT exchanged for cloud IAM role Trust policy scoped to repo + environment + branch

Threat: poisoned pipeline via PR

An attacker opens a PR that modifies .github/workflows/ci.yml to exfiltrate secrets on the next run. If workflows from fork PRs run with write token access, the game is over.

Solution:

  • Require CODEOWNERS approval for .github/**
  • Run untrusted fork workflows with read-only permissions:
  • Use pull_request (not pull_request_target) for building PR code
  • Enable "Require approval for first-time contributors" in repo settings
.github/workflows/deploy.yml — OIDC to AWS
name: Deploy to EKS
on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-oidc-deploy
          aws-region: us-east-1

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push
        run: |
          docker build -t $ECR_REGISTRY/app:${{ github.sha }} .
          docker push $ECR_REGISTRY/app:${{ github.sha }}

      - name: Update GitOps manifest
        run: |
          yq -i '.spec.template.spec.containers[0].image = "'$ECR_REGISTRY'/app:${{ github.sha }}"' k8s/deployment.yaml
          git config user.name "github-actions[bot]"
          git commit -am "deploy: ${{ github.sha }}"
          git push

Job orchestration: needs, matrix, concurrency

needs: creates a DAG—security scans wait for build artifacts. strategy.matrix fans out across OS/runtime versions. concurrency cancels superseded runs on the same branch, saving minutes and reducing attack window.

.github/workflows/matrix.yml
jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest]
        node: [18, 20, 22]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: npm
      - run: npm ci && npm test

  scan:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aquasecurity/trivy-action@master
        with:
          scan-type: fs
          severity: CRITICAL,HIGH
          exit-code: 1
flowchart TB
  L[lint] --> B[build]
  B --> T[test matrix\n3 node × 2 OS]
  T --> S[SAST + SCA]
  T --> C[container scan]
  S --> P[publish + sign]
  C --> P
  P --> D[deploy\nenvironment: production]
🔬 Under the Hood

GitHub-hosted runners are fresh VMs per job—disk is wiped after completion. Self-hosted runners retain filesystem state unless you use ephemeral auto-scaled runners. A compromised build can poison the next job's cache or leave backdoors in Docker layer cache.

💡 Pro Tip

Pin third-party actions to full commit SHA: uses: actions/checkout@b4ffde65f46336ab88eb53be708477a393ebcf08. Dependabot can bump SHA pins weekly. Tag-based pins (@v4) are mutable and have been used in supply-chain attacks.

🎯 Interview Tip

When asked "How do you secure GitHub Actions?": mention explicit permissions, OIDC over static keys, environment protection rules, SHA-pinned actions, fork PR restrictions, and separating build (untrusted code) from signing (trusted runner pool).

📦 Real World

GitHub publishes reusable workflow templates for Node, Python, and container builds with native CodeQL and dependency review on PRs. Enterprises use GitHub Enterprise Server with self-hosted runners in isolated VPCs and policy to block public marketplace actions.

GitLab CI

GitLab CI is declarative YAML in .gitlab-ci.yml at repo root. Stages run in order; jobs within a stage run in parallel. Runners poll GitLab for work—shell, Docker, or Kubernetes executor—with integrated container registry and security scanning built into the platform.

GitLab primitives vs GitHub Actions

GitLab GitHub Actions equivalent Notes
stages Implicit via needs: GitLab enforces global stage ordering
rules / only / except on: filters rules: is expressive; prefer over legacy only:
needs needs: DAG within/across stages; can skip stages
artifacts actions/upload-artifact Native SARIF, JUnit, coverage report types
cache actions/cache Key-based; watch for cache poisoning across branches
CI/CD variables Secrets + vars Protected + masked flags; scope to environment

Threat: runner credential exposure

Shared shell executors run jobs as the same OS user. A malicious job can read CI_JOB_TOKEN from /proc of concurrent jobs or scrape runner registration tokens from disk.

Solution:

  • Use Docker or Kubernetes executor with isolated containers per job
  • Disable shell executor on shared infrastructure
  • Scope CI_JOB_TOKEN via CI/CD job token allowlist (GitLab 16+)
  • Tag runners: tags: [trusted-signing] for privileged work only
.gitlab-ci.yml — MR pipeline with rules
stages: [validate, test, scan, publish]

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

validate:
  stage: validate
  script:
    - pip install pre-commit
    - pre-commit run --all-files
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

unit-test:
  stage: test
  image: node:20-alpine
  cache:
    key: ${CI_COMMIT_REF_SLUG}-npm
    paths: [node_modules/]
  script:
    - npm ci
    - npm test
  artifacts:
    reports:
      junit: junit.xml

sast:
  stage: scan
  script:
    - semgrep scan --config=auto --sarif -o gl-sast-report.sarif
  artifacts:
    reports:
      sast: gl-sast-report.sarif
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

container-scan:
  stage: scan
  image: docker:24
  services: [docker:24-dind]
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - trivy image --exit-code 1 $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  needs: [unit-test]

publish:
  stage: publish
  script:
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Merge request pipelines vs branch pipelines

MR pipelines run against the merge result ref—catching integration issues before merge. Branch pipelines on main publish artifacts and trigger deploy. Use workflow:rules to prevent duplicate pipelines on MR open + push.

flowchart TB
  subgraph mr["Merge Request pipeline"]
    V1[validate] --> T1[test]
    T1 --> S1[scan]
    S1 --> X1[no publish]
  end
  subgraph main["Main branch pipeline"]
    V2[validate] --> T2[test]
    T2 --> S2[scan]
    S2 --> P2[publish to registry]
    P2 --> D2[trigger deploy]
  end
terminal — GitLab pipeline inspection
$ glab ci lint
→ Validates .gitlab-ci.yml syntax before push
$ glab ci view --web
$ gitlab-runner verify
→ Confirms runner connectivity and executor type
$ curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab/api/v4/projects/1/packages?package_type=generic"
🔒 Security

Mark production variables as Protected (only runs on protected branches) and Masked (hidden in logs). Combine with environment: production and deployment approvals. Never pass secrets through docker build --build-arg—they persist in image history.

🏗 IaC

GitLab Kubernetes executor installs runners as pods in your cluster. Helm chart values control resource limits, pod security standards, and node selectors. Treat runner namespace as privileged infrastructure—network policies should block lateral movement to prod workloads.

⚖️ Trade-off

DinD (Docker-in-Docker) simplifies image builds but requires privileged containers—a security risk. Alternatives: Kaniko (rootless), Buildah, or cloud-native build services. Trade build complexity for smaller blast radius.

Pipeline Optimization

Slow pipelines erode DevSecOps adoption—developers bypass checks, merge without green CI, or disable scans. Optimization is a security investment: fast feedback loops keep gates enabled without sacrificing DORA lead time.

Where time goes

Bottleneck Typical % of pipeline Fix
Dependency install 25–40% Lockfile cache, remote cache (Bazel/Gradle), pre-baked runner image
Docker build 20–35% BuildKit cache mounts, multi-stage slim images, kaniko layer cache
Security scans 15–30% Incremental scan (diff only on PR), parallel jobs, cache vulnerability DB
Test suite 20–50% Test splitting, fail-fast, move E2E to nightly
Queue / cold start 5–15% Self-hosted pool, larger runner SKU, concurrency cancel

Threat: cache poisoning

Aggressive caching improves speed but introduces risk: a malicious PR poisons the shared cache with compromised dependencies or build artifacts that subsequent pipelines consume on trusted branches.

Solution:

  • Scope cache keys to lockfile hash + base branch, not just branch name
  • Invalidate cache on security scanner or dependency policy changes
  • Separate cache namespaces for fork PRs vs internal branches
  • Verify artifact checksums after cache restore
.github/workflows/optimized.yml — caching
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
          cache-dependency-path: package-lock.json

      - name: Cache Docker layers
        uses: actions/cache@v4
        with:
          path: /tmp/.buildx-cache
          key: buildx-${{ hashFiles('Dockerfile', 'package-lock.json') }}
          restore-keys: buildx-

      - uses: docker/setup-buildx-action@v3
      - name: Build with cache
        uses: docker/build-push-action@v5
        with:
          context: .
          push: false
          tags: app:${{ github.sha }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max

      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache
.gitlab-ci.yml — cache + needs DAG
.npm-cache: &npm-cache
  cache:
    key:
      files: [package-lock.json]
    paths: [node_modules/]
    policy: pull-push

fast-lint:
  stage: validate
  <<: *npm-cache
  script: [npm ci, npm run lint]
  cache:
    policy: pull

unit-test:
  stage: test
  needs: [fast-lint]
  parallel: 4
  script:
    - npm ci
    - npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

incremental-sast:
  stage: scan
  needs: [unit-test]
  script:
    - git fetch origin $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
    - semgrep scan --baseline-commit=origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Parallelization patterns

flowchart LR
  L[lint\n30s] --> B[build\n2m]
  B --> T1[unit tests\nshard 1]
  B --> T2[unit tests\nshard 2]
  B --> T3[unit tests\nshard 3]
  T1 --> M[merge results]
  T2 --> M
  T3 --> M
  M --> S1[SAST\nparallel]
  M --> S2[SCA\nparallel]
  M --> S3[container\nparallel]
  S1 --> G[gate]
  S2 --> G
  S3 --> G
💡 Pro Tip

Track p50/p95 pipeline duration per stage in your observability stack. When p95 spikes, developers notice before you do—and they start merging with admin override. Alert on stage regression like production SLOs.

⚖️ Trade-off

fail-fast: true saves minutes but hides all failures at once—developers fix one, re-run, fix another. fail-fast: false surfaces the full failure set per run (better DX) at higher compute cost. Use false on main; true on feature branches if budget-constrained.

📦 Real World

Shopify reports cutting CI time 50%+ via test sharding and remote caching. Google uses hermetic Bazel builds—cache hits across teams because inputs are content-addressed, not branch-named.

Reusable Templates

Platform teams scale DevSecOps by encoding the golden path once—security gates, OIDC patterns, scan thresholds— and letting product teams inherit via reusable workflows (GitHub) or include: templates (GitLab). Drift becomes a pull request to the platform repo, not fifty divergent copies.

Why templates matter for security

Without templates, each team invents its own pipeline. Security reviews cannot scale to 200 repos. Central templates enforce: minimum scan set, OIDC-only deploy, signed artifacts, and approved action versions. Product teams override only what is unique—runtime version, test command, image name.

Pattern GitHub Actions GitLab CI
Central pipeline definition Reusable workflow in org/.github include: project: 'platform/ci-templates'
Parameterized inputs workflow_call: inputs: spec:inputs (CI/CD components)
Shared steps Composite actions YAML anchors &anchor / *anchor
Version pinning @ref on reusable workflow call ref: 2.4.1 on include
Compliance evidence Org rulesets require workflow file presence Compliance pipeline in GitLab Ultimate
org/.github/workflows/node-ci.yml — reusable
name: Node CI Template
on:
  workflow_call:
    inputs:
      node-version:
        required: false
        type: string
        default: "20"
      run-e2e:
        required: false
        type: boolean
        default: false
    secrets:
      SONAR_TOKEN:
        required: false

permissions:
  contents: read
  security-events: write
  id-token: write

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
          cache: npm
      - run: npm ci && npm test
      - uses: gitleaks/gitleaks-action@v2
      - uses: aquasecurity/trivy-action@master
        with:
          scan-type: fs
          exit-code: 1
          severity: CRITICAL,HIGH
app-repo/.github/workflows/ci.yml — consumer
name: CI
on:
  pull_request:
  push:
    branches: [main]

jobs:
  call-platform-template:
    uses: my-org/.github/.github/workflows/[email protected]
    with:
      node-version: "22"
      run-e2e: false
    secrets: inherit
.gitlab-ci.yml — include template
include:
  - project: platform/ci-templates
    ref: 2.4.1
    file: /templates/node-service.yml

variables:
  SERVICE_NAME: payments-api
  NODE_VERSION: "22"

# Override only what differs from template
unit-test:
  script:
    - npm run test:payments
platform/ci-templates/templates/node-service.yml
stages: [validate, test, scan, publish]

.base-node:
  image: node:${NODE_VERSION}-alpine
  before_script:
    - npm ci

validate:
  extends: .base-node
  stage: validate
  script:
    - gitleaks detect --source .
    - npm run lint

unit-test:
  extends: .base-node
  stage: test
  script: [npm test]

sast:
  stage: scan
  script:
    - semgrep scan --config=auto --sarif -o gl-sast-report.sarif
  artifacts:
    reports:
      sast: gl-sast-report.sarif
flowchart TB
  P[Platform team\norg/.github or ci-templates] --> T[Golden path template\nscans + OIDC + sign]
  T --> A[App repo A]
  T --> B[App repo B]
  T --> C[App repo C]
  A --> PR1[PR checks]
  B --> PR2[PR checks]
  C --> PR3[PR checks]
  PR1 --> M[Merge to main]
  M --> D[Standardized deploy]
🔒 Security

Pin template versions (@v2.4.1, not @main). A compromised template repo becomes a supply-chain nuke—require signed tags, CODEOWNERS on template changes, and staged rollout (canary repos before org-wide bump).

🎯 Interview Tip

Describe the golden path pattern: platform owns secure defaults, product teams consume via template, exceptions require architecture review. Mention versioning, override boundaries, and how you prevent teams from deleting security jobs in their local override.

⚠️ Pitfall

Templates that are too rigid drive shadow pipelines—teams create .github/workflows/nightly-hack.yml outside the golden path. Provide escape hatches (documented override jobs) and audit for workflow files not using the template.

Jenkins & Migration

Jenkins remains entrenched in enterprises—thousands of Groovy Jenkinsfiles, plugin ecosystems, and on-prem agents. Modernization means extracting patterns into cloud-native CI while preserving audit trails. This section covers secure Jenkins operation and a pragmatic migration path to GitHub Actions or GitLab CI.

Jenkins architecture recap

Component Role Security risk
Controller Schedules builds, serves UI, stores config RCE via vulnerable plugins; credential vault on controller
Agent Executes pipeline steps Shared workspace leaks artifacts between jobs
Plugins Extend functionality (~2000 available) Unpatched CVEs; supply chain via update center
Credentials Encrypted store on controller Over-broad folder scope; plaintext in Groovy if mishandled
Shared Library Reusable Groovy functions Library repo compromise = all pipelines compromised

Threat: Jenkins plugin RCE

Jenkins plugins run with controller-level privileges. Historical CVEs (e.g., Script Security bypasses, SnakeYAML deserialization) allow unauthenticated or low-privilege users to execute arbitrary code on the controller.

Solution:

  • Minimize plugin count; prefer declarative pipeline built-ins
  • Enable automatic security updates; audit plugin CVEs weekly
  • Disable script approval for untrusted users; use Job DSL from trusted repo only
  • Run agents on ephemeral VMs/containers—not permanent pets with accumulated secrets
  • Integrate credentials via Vault plugin, not static Jenkins credential store
Jenkinsfile — declarative pipeline with security stages
pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
          - name: node
            image: node:20-alpine
            command: ['sleep']
            args: ['infinity']
            resources:
              limits:
                memory: 2Gi
      '''
    }
  }
  options {
    buildDiscarder(logRotator(numToKeepStr: '30'))
    disableConcurrentBuilds()
    timeout(time: 30, unit: 'MINUTES')
  }
  environment {
    DOCKER_REGISTRY = 'registry.example.com'
  }
  stages {
    stage('Checkout') {
      steps { checkout scm }
    }
    stage('Secret Scan') {
      steps {
        container('node') {
          sh 'gitleaks detect --source . --verbose'
        }
      }
    }
    stage('Test') {
      steps {
        container('node') {
          sh 'npm ci && npm test'
        }
      }
    }
    stage('SAST') {
      steps {
        container('node') {
          sh 'semgrep scan --config=auto --sarif -o semgrep.sarif'
        }
      }
    }
    stage('Build & Scan Image') {
      steps {
        script {
          docker.build("${DOCKER_REGISTRY}/app:${env.GIT_COMMIT}")
          sh "trivy image --exit-code 1 ${DOCKER_REGISTRY}/app:${env.GIT_COMMIT}"
        }
      }
    }
    stage('Sign & Push') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'cosign-key', variable: 'COSIGN_KEY')]) {
          sh 'cosign sign --key env://COSIGN_KEY ${DOCKER_REGISTRY}/app:${env.GIT_COMMIT}'
          sh 'docker push ${DOCKER_REGISTRY}/app:${env.GIT_COMMIT}'
        }
      }
    }
  }
  post {
    always {
      archiveArtifacts artifacts: 'semgrep.sarif', allowEmptyArchive: true
      junit '**/junit.xml'
    }
    failure {
      slackSend channel: '#builds', message: "Failed: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
    }
  }
}

Migration strategy: strangler fig

Do not big-bang migrate 500 Jenkins jobs. Use the strangler pattern: new repos on GitHub Actions/GitLab CI, migrate high-churn services first, keep Jenkins as read-only archive for compliance until decommissioned.

Phase Action Duration
1. Inventory Catalog jobs, plugins, credentials, shared libraries; map owners 2–4 weeks
2. Template Build golden path reusable workflow / GitLab template matching Jenkins stages 2–3 weeks
3. Pilot Migrate 3–5 low-risk services; validate scan parity and deploy hooks 4 weeks
4. Scale Team-by-team migration with pairing; freeze new Jenkins jobs 3–12 months
5. Decom Export build history; shut down agents; retain audit logs in SIEM 2–4 weeks
flowchart LR
  subgraph legacy["Legacy Jenkins"]
    J1[Job A]
    J2[Job B]
    J3[Job C]
  end
  subgraph modern["Cloud CI"]
    G1[GitHub Actions]
    L1[GitLab CI]
  end
  J1 -.->|migrated| G1
  J2 -.->|migrated| L1
  J3 -->|still on Jenkins| J3
  G1 --> R[Unified registry]
  L1 --> R
  J3 --> R
Migration mapping — Jenkins → GitHub Actions
# Jenkins: stage('Test') { steps { sh 'npm test' } }
# Actions equivalent:
- name: Test
  run: npm ci && npm test

# Jenkins: when { branch 'main' }
# Actions equivalent:
if: github.ref == 'refs/heads/main'

# Jenkins: withCredentials([...])
# Actions equivalent: secrets.MY_SECRET or OIDC (preferred)

# Jenkins: shared library vars.buildApp()
# Actions equivalent:
uses: my-org/.github/.github/workflows/build.yml@v1
Migration mapping — Jenkins → GitLab CI
# Jenkins: post { always { junit '**/junit.xml' } }
# GitLab equivalent:
artifacts:
  reports:
    junit: junit.xml

# Jenkins: parallel { stage('A')... stage('B')... }
# GitLab equivalent:
parallel:
  job-a:
    script: [npm run test:unit]
  job-b:
    script: [npm run test:integration]

# Jenkins: buildDiscarder
# GitLab: artifacts:expire_in + project CI/CD settings retention
terminal — Jenkins inventory and export
$ jenkins-cli list-jobs -r | wc -l
→ 487 jobs across 12 folders
$ jenkins-plugin-cli --list | grep -i vulnerable
$ curl -s $JENKINS_URL/job/payments-api/config.xml | xmllint --format -
→ Extract SCM URL, triggers, and credential IDs for migration spreadsheet
$ gh workflow run ci.yml --ref migrate/payments-api
🔬 Under the Hood

Jenkins stores job config as XML on the controller filesystem. Shared libraries clone at runtime from a Git ref— same supply-chain risk as reusable workflows. Migration tools like jenkinsfile-runner or custom Groovy→YAML transpilers help but rarely achieve 100% parity; plan manual validation per service.

📦 Real World

Banks often keep Jenkins for mainframe or legacy .NET Framework builds while cloud-native services run on GitHub Actions with OIDC to AWS. Hybrid is acceptable for years—what matters is no new secrets in Jenkins and a dated decommission roadmap executives have signed.

⚖️ Trade-off

Lift-and-shift Jenkins to K8s (operator + ephemeral agents) buys time but retains Groovy debt and plugin CVE surface. Full migration to SaaS CI is higher upfront cost but eliminates controller patching forever. Most enterprises land in hybrid for 18–36 months.