The Jenkinsfile: Pipelines in Depth

Jenkins is the machine; the Jenkinsfile is how you tell it what to do. The previous page gave you the controller/agent model — where builds run. Pipelines & Stages gave you the universal structure — stages with gates between them. This page joins the two: how Jenkins specifically expresses that structure as code in your repository, what the language actually is, and the security model that makes running other people’s pipeline code survivable. By the end you should be able to read any Jenkinsfile and know exactly which knob does what.

Pipeline-as-code: the build lives in the repo

Jenkins introduced the split between freestyle jobs (defined by clicking around the web UI) and Pipeline jobs (defined by a Jenkinsfile committed to the repo). The whole reason to prefer the latter is that the build process becomes an artifact like any other:

• Versioned — the pipeline is in git, so it has history, blame, and tags. The build that produced release v2.3.0 is recoverable because its Jenkinsfile is in that tag.
• Reviewed — a change to how you build goes through the same pull request and review as a change to what you build. “Who weakened the test gate?” has an answer.
• Reproducible — rebuild Jenkins from scratch and the jobs come back, because their definitions were never in Jenkins; they were in the repos Jenkins reads.

That is the principle. Everything below is the mechanism.

Declarative vs scripted: two languages, one engine

Jenkins Pipeline comes in two syntaxes, and the difference matters.

	Declarative	Scripted
Top-level block	pipeline { ... }	node { ... }
Shape	a fixed, validated structure	free-form Groovy
Power	constrained to the directives	the full Groovy language
Errors	caught early by a schema	found at runtime
Best for	~95% of real pipelines	the rare case you truly need code

Declarative is the modern default. You fill in a known set of sections — agent, stages, steps, post — and Jenkins validates the structure before running it, so a typo’d directive fails fast with a clear message rather than halfway through a deploy. It reads like configuration, which is exactly what most pipelines are.

Scripted is the original syntax: a Groovy program inside a node block. It gives you the entire language — loops, functions, arbitrary control flow — which is occasionally necessary and frequently dangerous, because now your build logic is a program someone has to reason about.

The guidance is blunt: use declarative. When you hit something the directives genuinely can’t express, declarative gives you an escape hatch — a script block that drops into raw Groovy for just that island:

stage('Compute build args') {
  steps {
    script {                                  // escape into scripted Groovy, locally
      def parts = env.BRANCH_NAME.split('/')
      env.SHORT_NAME = parts[-1].take(20)
    }
    sh "echo building ${env.SHORT_NAME}"
  }
}

Keep these islands small. The moment your script blocks dominate the file, you’ve written a scripted pipeline wearing a declarative costume, and you’ve given up the validation that was the point.

Anatomy of a declarative pipeline

Here is a real, end-to-end Jenkinsfile: build the app, test it, build a container image, and push it — tagged with the immutable git commit. Read it once, then we’ll name every directive.

pipeline {
  agent { label 'linux && docker' }           // run on an agent carrying BOTH labels

  options {
    timeout(time: 30, unit: 'MINUTES')         // kill a hung build instead of wedging an executor
    retry(2)                                   // up to 2 attempts total (one automatic retry)
    buildDiscarder(logRotator(numToKeepStr: '20'))  // keep only the last 20 builds' logs/artifacts
    disableConcurrentBuilds()                  // don't run two builds of this branch at once
  }

  parameters {
    booleanParam(name: 'PUSH_IMAGE', defaultValue: true, description: 'Push the built image?')
    string(name: 'IMAGE_REPO', defaultValue: 'ghcr.io/acme/app', description: 'Target image repo')
  }

  environment {
    IMAGE_TAG = "${env.GIT_COMMIT.take(7)}"    // immutable tag = short git SHA
    // credentials() binds a username/password cred into _USR and _PSW vars, masked in the log
    REGISTRY  = credentials('ghcr-credentials')
  }

  stages {
    stage('Build') {
      steps {
        sh 'npm ci'
        sh 'npm run build'
      }
    }

    stage('Test') {
      parallel {                               // these two run at the same time
        stage('unit') { steps { sh 'npm run test:unit' } }
        stage('lint') { steps { sh 'npm run lint' } }
      }
    }

    stage('Build image') {
      steps {
        sh "docker build -t ${params.IMAGE_REPO}:${IMAGE_TAG} ."
      }
    }

    stage('Push image') {
      when {                                   // conditional stage: skip unless both are true
        allOf {
          branch 'main'
          expression { params.PUSH_IMAGE }
        }
      }
      steps {
        // never echo the password; pipe it into stdin and let Jenkins mask it in the console
        sh 'echo "$REGISTRY_PSW" | docker login ghcr.io -u "$REGISTRY_USR" --password-stdin'
        sh "docker push ${params.IMAGE_REPO}:${IMAGE_TAG}"
      }
    }
  }

  post {
    success { echo "Pushed ${params.IMAGE_REPO}:${IMAGE_TAG}" }
    failure { echo 'Build failed — open the stage view and read the red stage.' }
    always  { junit 'reports/**/*.xml' }       // publish test results pass or fail
    cleanup { sh 'docker logout ghcr.io || true' }
  }
}

Every directive, in plain terms:

• agent — where the pipeline runs. agent { label 'linux && docker' } demands an agent whose labels satisfy the expression. You can also set agent per-stage so different stages run on different machines.
• options — guardrails for the whole run. timeout bounds wall-clock time (a hung build no longer holds an executor forever); retry re-attempts on failure; buildDiscarder garbage-collects old build history so the controller’s disk doesn’t fill; disableConcurrentBuilds serializes a branch.
• parameters — typed inputs a human or trigger supplies at launch, available as params.NAME. This is how one pipeline serves “build only” and “build and push” without two files.
• environment — variables for every step. The standout is the credentials() helper: it pulls a secret out of the Jenkins credentials store and binds it into the environment masked — a username/password credential becomes REGISTRY_USR and REGISTRY_PSW, and Jenkins redacts those values if they ever appear in the log.
• stages / stage / steps — the pipeline structure itself: stages holds the ordered list, each stage is a named phase shown in the UI, and steps are the actual commands (sh, junit, etc.).
• when — a conditional gate on a stage. The Push image stage runs only on main and only when the parameter is set, so feature branches build and test but never publish.
• post — what to do after, keyed by outcome: always, success, failure, unstable, and cleanup (which runs dead last, even after always). This is where notifications, test-report publishing, and teardown live, so they happen whether the build went green or red.

parallel and matrix: spend wall-clock once

Two directives turn a serial pipeline into a wide one. parallel (used above) runs sibling stages at the same time. matrix is parallel’s heavy machinery: declare axes, and Jenkins runs the same stages once per combination of axis values — the standard pattern for “test on every OS × every runtime version.”

stage('Cross-test') {
  matrix {
    axes {
      axis { name 'NODE_VERSION'; values '18', '20' }
      axis { name 'OS';           values 'linux', 'windows' }
    }
    stages {
      stage('test') {
        steps { sh 'npm test' }                // runs once per cell: 2 × 2 = 4 cells
      }
    }
  }
}

By the numbers — what parallel actually buys

A matrix of 2 Node versions × 3 operating systems = 6 cells, each taking ~6 minutes:

serial   : 6 cells × 6 min            = 36 min wall-clock
parallel : max(6 min over 6 executors) ≈ 6 min wall-clock
speedup  : 36 / 6                      = 6×

The catch is in the denominator: the 6× assumes six free executors. Parallelism converts wall-clock time into executor demand — you get the answer six times faster only if your agents have the capacity to run six things at once. If you have two executors, six cells run three waves deep and the speedup is ~2×, not 6×. This is the everyday tension in a busy Jenkins: the pipeline is as parallel as you wrote it, but only as fast as the executor pool allows.

Agents as ephemeral environments

The most important modern use of agent isn’t picking a static machine — it’s declaring a disposable, reproducible environment per build. Two forms:

// One container image for the whole pipeline — clean toolchain, no host pollution.
agent { docker { image 'node:20' } }

// A whole pod spec — pick exactly the containers (and sidecars) this build needs.
agent {
  kubernetes {
    yaml '''
      apiVersion: v1
      kind: Pod
      spec:
        containers:
        - name: node
          image: node:20
          command: ['sleep']
          args: ['infinity']
        - name: docker
          image: docker:25-dind          # docker-in-docker sidecar for image builds
    '''
  }
}

Both express the same idea the containers part hammered: the build runs in a fresh, declared image that is destroyed afterward. There’s no “works because the last build left node_modules lying around” — the environment is part of the Jenkinsfile, versioned with it, identical on every run. With the Kubernetes form, agents are pods that exist only for the duration of one build: elastic scale, zero state bleed, and the agent definition lives next to the pipeline that needs it.

Shared libraries: DRY across teams

Once you have twenty repos, you do not want twenty copies of the same 80-line Jenkinsfile. A shared library is a separate git repo of reusable pipeline code that any Jenkinsfile can load:

@Library('platform-ci') _                      // load the org's shared library (the _ is required)

pipeline {
  agent any
  stages {
    stage('build') {
      steps {
        buildAndPush repo: 'ghcr.io/acme/app'  // a custom step defined in the library
      }
    }
  }
}

A shared library has a fixed layout:

• vars/ — files like vars/buildAndPush.groovy that each define a def call(...) and become a custom step you can call by filename (buildAndPush). This is how a platform team ships “the blessed way to build and push” as a one-liner.
• src/ — regular Groovy classes in a package hierarchy, for richer logic the vars/ steps lean on.

The payoff is exactly the DRY payoff everywhere else: fix the build logic once, in the library, and every consuming repo gets the fix on its next run — instead of opening twenty pull requests.

Multibranch pipelines: one Jenkinsfile, every branch

A multibranch pipeline points Jenkins at a repository (or whole GitHub/GitLab org) and tells it: scan for branches and pull requests, and for each one that contains a Jenkinsfile, create and run a job using that branch’s Jenkinsfile.

   repo: acme/app
     ├── main            → job: app/main        (runs main's Jenkinsfile)
     ├── feature/login   → job: app/feature%2Flogin
     └── PR #214         → job: app/PR-214       (runs the PR branch's Jenkinsfile)
   branch deleted        → its job is auto-removed

This is what makes “the pipeline is in the repo” fully self-service: open a branch, it gets built; open a PR, it gets tested; merge and delete, its job disappears. Nobody configures a job by hand. It also means a branch can change its own pipeline — which is powerful, and, for pull requests from forks, exactly the danger the sandbox below exists to contain.

Secrets in a pipeline: bind, mask, never echo

The worked example used credentials() in the environment block. The other binding form is withCredentials, which scopes a secret to a specific block of steps:

withCredentials([usernamePassword(
    credentialsId: 'ghcr-credentials',
    usernameVariable: 'USER',
    passwordVariable: 'PASS')]) {
  sh 'echo "$PASS" | docker login ghcr.io -u "$USER" --password-stdin'
}

Both approaches pull secrets from Jenkins’s credentials store (not from the repo) and both mask the values in the console log — if a secret’s literal text would appear in output, Jenkins replaces it with ****. The rules that make this actually safe:

• Never echo a secret or pass it as a visible command-line argument (process listings leak it). Pipe it into stdin, as above.
• Scope tightly — withCredentials exposes the secret only inside its block, not for the whole run.
• Reference by ID, never inline. The pipeline names a credential; the value lives in Jenkins.

Masking is a safety net, not a license: it catches the value in stdout, but a set -x shell, a crash dump, or a secret written into a file can still leak. Treat the binding as “the secret exists here, briefly, for these steps” and design the steps so it never needs to be printed.

Under the hood — declarative is Groovy, and that’s why the sandbox exists

Declarative pipeline isn’t a separate engine. It’s a thin Groovy DSL that compiles down onto the scripted Pipeline engine — the same one node { } uses. That engine runs your Jenkinsfile through a Groovy interpreter using CPS (continuation-passing style) so a build can survive a controller restart mid-run.

Here is the part most people miss and the reason for everything below: the Groovy in a Jenkinsfile executes on the controller, inside the controller’s JVM. Only the sh/bat steps run out on the agent. So an unrestricted Jenkinsfile could call straight into Jenkins internals — read the credentials store, run arbitrary code on the box that holds every secret — which is catastrophic given that a multibranch setup will happily run a Jenkinsfile authored by whoever opened a pull request.

That is why Jenkins runs untrusted Pipeline code in the Groovy script-security sandbox. The sandbox intercepts every method call and field access and checks it against an allowlist of safe operations. Anything not allowed is blocked — and queued in “In-process Script Approval,” where a Jenkins administrator must explicitly approve that specific method signature before it can ever run. Trusted code (a shared library loaded from a trusted source, or a script an admin runs) is allowed outside the sandbox. The model is the whole least-privilege story in miniature: untrusted code gets a tiny allowlisted vocabulary; widening it is a deliberate, logged, admin-only act.

It actually happened — when the sandbox leaked

The sandbox is load-bearing, and it has, repeatedly, been bypassed. The Script Security and Pipeline Groovy plugins have a long history of sandbox-escape vulnerabilities — most famously the 2019 family (CVE-2019-1003000, -1003001, -1003002), sandbox bypasses that allowed otherwise-restricted Groovy to execute unsandboxed and, chained together, gave remote code execution on the controller — in some configurations triggerable by a low-privileged user who could merely submit pipeline script. Several were demonstrated publicly by the researcher Orange Tsai.

The lesson is structural, not about any one CVE id. A Jenkinsfile is arbitrary Groovy that runs next to your credentials, and “we sandbox it” is a defense that has historically failed. So: keep Jenkins and its plugins patched, be extremely conservative about approving scripts (each approval permanently widens the allowlist), and think hard before letting fork pull requests run their own Jenkinsfile against a controller that holds anything valuable. Untrusted build code and crown-jewel credentials should not share a JVM if you can avoid it.

The thread

What manual, error-prone step does this remove — and how does it make production safer? A Jenkinsfile takes “the way we build this service” out of one engineer’s head (or out of click-ops buried in the Jenkins UI) and turns it into versioned, reviewed, reproducible code: the toolchain is pinned to a container image, the secrets come from a store and are masked, the gates (when, post, the stage order) are explicit, and a multibranch setup applies the same process to every branch and PR with nobody configuring anything. The honest cost is the other half of the thread — because that code is Groovy running on the controller, you inherit a real security boundary (the sandbox, script approval, the discipline around untrusted PRs). You remove human inconsistency from the build by accepting responsibility for the trust model of the code that now does it. Next, we hand the artifact this pipeline produced to a different machine entirely: Jenkins to Argo CD, where CI stops reaching into production.

The architect’s lens

Five questions for pipeline-as-code, before you write your next Jenkinsfile:

• Why does it exist? To take “how we build this service” out of click-ops in the Jenkins UI (or one engineer’s head) and turn it into versioned, reviewed, reproducible code that lives in the repo.
• What problem does it solve? Unreviewable, unrecoverable builds: a Jenkinsfile gives history and blame, pins the toolchain to a declared (often ephemeral) image, masks credentials() from a store, and via multibranch applies the same process to every branch and PR.
• What are the trade-offs? Declarative buys schema-validation-before-run but constrains you to its directives (the script {} escape hatch erodes that if it grows); and because the Groovy executes on the controller, you inherit the sandbox, script approval, and the fork-PR trust problem.
• When should I avoid it? When hosted CI’s YAML model (GitHub Actions/GitLab) covers your needs — you may not want to own a Groovy-on-the-controller security boundary at all.
• What breaks if I remove it? Builds revert to freestyle click-ops: invisible to review, lost if the server dies, and inconsistent from one branch to the next.

Check your understanding

1. Why is a Jenkinsfile in the repo strictly better than a freestyle job for a real service? Give three concrete properties it gains.
2. Declarative and scripted both run on the same engine. What does declarative give you that scripted doesn’t, and when is the script { } escape hatch appropriate?
3. In the worked pipeline, the Push image stage has a when { allOf { branch 'main'; expression { ... } } } block. What two conditions must hold for it to run, and why would you gate a push that way?
4. What does the credentials() helper do, and name two rules that keep a bound secret from leaking despite masking.
5. A matrix has 2 runtime versions × 3 OSes and each cell takes 6 minutes. What’s the ideal wall-clock time, what real-world resource caps that speedup, and by how much if only two executors are free?
6. The Groovy in a Jenkinsfile runs on the controller, not the agent. Explain why that fact makes the script-security sandbox necessary, and what “In-process Script Approval” is for.

Show answers

1. It is versioned (history, blame, recoverable per release tag), reviewed (build changes go through the same PR/review as code changes), and reproducible (rebuild Jenkins from scratch and the jobs come back, because the definitions live in the repos, not inside Jenkins). Freestyle config lives in the UI: invisible to review and lost if the server dies.
2. Declarative is a fixed, schema-validated structure, so structural mistakes are caught before the run and the file reads like configuration. Scripted gives the full Groovy language (loops, functions, arbitrary control flow) at the cost of being a program someone must reason about. Use the script { } block only for a small island of genuine logic the directives can’t express — keep it tiny so you don’t lose validation.
3. It runs only when the branch is main and the PUSH_IMAGE parameter is true. You gate a push like that so feature branches and PRs still build and test (cheap, safe) but never publish an image — publishing is reserved for the integration branch, and even then is opt-out-able.
4. credentials('id') pulls a secret from the Jenkins credentials store and binds it into the environment masked (a username/password becomes _USR and _PSW, redacted in logs). Rules (any two): never echo it or pass it as a visible CLI arg — pipe via stdin; scope it tightly (prefer withCredentials); reference by ID, never paste the value into the pipeline.
5. Ideal wall-clock is max over the cells ≈ 6 minutes (vs 36 serial → 6× speedup), but only if there are six free executors — parallelism trades wall-clock for executor demand. With two executors, six cells run in three waves of two, so wall-clock ≈ 18 minutes and the speedup is ~2×, not 6×.
6. Because the Jenkinsfile’s Groovy executes inside the controller’s JVM (only sh/bat run on the agent), unrestricted pipeline code could read the credentials store and run arbitrary code on the box that holds every secret — and multibranch will run a Jenkinsfile written by whoever opened a PR. The sandbox intercepts every method call/field access against an allowlist; anything outside it is blocked and queued in In-process Script Approval, where an admin must explicitly (and permanently) approve that method signature before it can run.