Jenkins: Architecture & the Controller/Agent Model

Continuous Integration is the practice; Jenkins is the oldest and still one of the most common machines that runs it. Before you can write a pipeline or wire Jenkins to Argo CD, you need the mental model of what Jenkins actually is — a server that sits there, watches your repositories, and turns “someone pushed code” into “the code was built, tested, and packaged” with no human pressing a button. This page is that model: the controller, the agents, and the execution loop between them.

What Jenkins is (and what it is not)

Jenkins is an open-source automation server written in Java — a long-running process you host yourself. It started life as Hudson (created by Kohsuke Kawaguchi at Sun, ~2005) and was renamed Jenkins in 2011 after a governance split with Oracle. Its job is to automate the steps between a commit and a deployable artifact: check out code, compile, run tests, build an image, publish it.

The thing to internalize early: Jenkins is infrastructure you own and operate. Unlike hosted CI (GitHub Actions, GitLab CI), there is a server — with a version, plugins, credentials, disk, and a patch cadence — that you are responsible for. That ownership is Jenkins’s great strength (it can do literally anything, on any hardware, in any network) and its great cost (you maintain it).

The recurring question, applied to Jenkins

What manual, error-prone step does this remove? Without a CI server, “build and test before merge” is a thing humans remember to do — inconsistently, on their own laptops, with their own tool versions. Jenkins makes the build a mechanical, identical, every-single-time event triggered by a push. The cost it charges in return is a server to keep alive and secure.

The controller and the agents

Jenkins runs as two roles. (You will still see the old names master and slave in tutorials; the project renamed slave to agent with Jenkins 2.0 in 2016 and master to controller in 2020 — same concepts.)

                 ┌───────────────────────────────┐
   git push ───► │           CONTROLLER          │   orchestration only:
                 │  • web UI & REST API          │   - watches repos / receives webhooks
                 │  • job scheduling + build queue│   - stores job config & history
                 │  • plugin management          │   - decides WHICH agent runs WHAT
                 │  • credentials store          │   - should run ZERO builds itself
                 └───────────────┬───────────────┘
                                 │ dispatches builds to…
              ┌──────────────────┼──────────────────┐
              ▼                  ▼                  ▼
        ┌───────────┐     ┌───────────┐      ┌───────────┐
        │  AGENT A  │     │  AGENT B  │      │  AGENT C  │   execution:
        │ label:    │     │ label:    │      │ ephemeral │   - check out code
        │  linux    │     │  windows  │      │  k8s pod  │   - run the pipeline steps
        │ 2 executors│    │ 1 executor│      │ 1 build   │   - stream logs back
        └───────────┘     └───────────┘      └───────────┘

• Controller — the brain. It serves the UI, holds configuration and build history, manages plugins and credentials, and schedules work. A production controller should run no builds of its own: a build step runs arbitrary code, and you never want arbitrary code executing on the box that holds every credential.
• Agents — the hands. Each agent is a machine (or container) that connects to the controller and actually executes pipeline steps. Agents carry labels (linux, gpu, docker) so a pipeline can demand the right environment.
• Executors — the slots. Each agent has N executors = N builds it can run at once. Two executors on one agent means two pipelines run concurrently on that machine.

Under the hood — how a build finds an agent

When a build is triggered, the controller puts it in the build queue with a label expression (agent { label 'linux && docker' }). The scheduler then waits for an available executor on an agent whose labels satisfy the expression, assigns the build, and the agent streams its console output back over the connection. If no matching executor is free, the build sits in the queue — which is exactly why a clogged queue is the classic Jenkins bottleneck, and why “add more agents/executors” is the classic fix. Agents connect either outbound from the controller over SSH, or inbound from the agent via the JNLP/WebSocket protocol (useful when the agent is behind a firewall).

Freestyle jobs vs Pipeline-as-code

Two ways to define what a job does:

	Freestyle (legacy)	Pipeline (modern)
Defined in	the web UI, click by click	a Jenkinsfile committed to your repo
Versioned?	no — config lives in Jenkins	yes — reviewed in PRs, diffable
Survives a Jenkins rebuild?	only via config backup	yes — it’s in git
Expressiveness	limited, plugin-driven	full stages, conditionals, parallelism

Freestyle jobs are click-ops: powerful for a one-off, but the definition lives inside Jenkins, so it’s invisible to code review and lost if the server dies. The modern answer is Pipeline-as-code — a Jenkinsfile in the repo, so the build process is versioned, reviewed, and reproducible exactly like the code it builds. Treat freestyle as legacy.

Plugins: the superpower and the tax

Jenkins’s core is small; almost everything real is a plugin. Git integration, Docker, Kubernetes agents, the entire Pipeline system, credentials bindings, Slack notifications — all plugins. There are on the order of ~1,800 of them.

Under the hood — the plugin double-edged sword

Plugins are why Jenkins can integrate with anything after 20 years — and why Jenkins maintenance is real work. Plugins have transitive dependencies and version constraints; upgrading one can break another; an abandoned plugin can pin you to an old Jenkins core; and each plugin is attack surface. “Plugin sprawl” is the single most common cause of a fragile, scary-to-upgrade Jenkins. The discipline is: install the fewest plugins that do the job, keep them and the core current, and prefer pipeline-native steps over plugin-specific UI features.

Scaling agents: static vs ephemeral

• Static agents — long-lived machines/VMs you maintain. Simple, but state bleeds between builds (leftover files, cached deps, “works because the last build left something behind”).
• Ephemeral agents — the Kubernetes plugin (or Docker) spins up a fresh pod per build and destroys it after. Every build starts from a clean, declared image: no state bleed, elastic scaling, and the agent definition lives in the Jenkinsfile. This is the modern default for Jenkins on k8s.

It actually happened — CVE-2024-23897 (January 2024)

A flaw in the Jenkins CLI argument parser expanded an @-prefixed argument into the contents of a file on the controller. On a Jenkins reachable by an attacker, this allowed unauthenticated reading of arbitrary files — including the secrets and keys needed to decrypt the credentials store, which several public exploits chained into remote code execution. It was rated critical and patched in 2.442 / LTS 2.426.3. The lesson is the whole reason a production controller runs no builds and hoards no unnecessary plugins: the controller is the crown jewels. Exposed, unpatched Jenkins servers have been a favorite target for years (cryptomining campaigns have hijacked thousands of them). Keep it patched, keep it off the public internet, and give it the least it needs.

Where Jenkins fits versus hosted CI

	Jenkins	GitHub Actions / GitLab CI
Hosting	you run the server	hosted by the provider (SaaS)
Config	Jenkinsfile (Groovy DSL)	YAML in the repo
Flexibility	near-unlimited (any HW, any network, plugins)	high, but within the platform’s model
Maintenance	yours (upgrades, plugins, agents, security)	mostly the provider’s
Best when	complex/air-gapped/bespoke needs, existing investment	you want CI without operating a server

Neither is “better.” Jenkins wins when you need total control, on-prem/air-gapped builds, or you already have deep Jenkins investment; hosted CI wins when you’d rather not operate a build server at all. Knowing why you’d pick each is the senior-level point.

The thread

What manual, error-prone step does this remove — and how does it make production safer? Jenkins turns “build and test before this ships” from a human ritual into a server-enforced, identical-every-time event, and the controller/agent split makes that safe to scale: the controller holds the credentials and the schedule, the agents run the untrusted build code, and ephemeral agents make every build start from a clean, declared environment. The price — a server you must patch and prune — is the recurring theme of self-hosted automation: you remove human error from the build by accepting operational responsibility for the machine that does it. Next, the Jenkinsfile turns that machine’s work into versioned code.

The architect’s lens

Five questions to decide whether to run Jenkins at all:

• Why does it exist? Because “build and test before merge” is otherwise a human ritual done inconsistently, on laptops, with mismatched tool versions — Jenkins is a long-running server that makes it mechanical and identical on every push.
• What problem does it solve? It automates commit→artifact, and the controller/agent split makes that safe to scale: the controller holds credentials and scheduling, agents run the untrusted build code (ideally ephemeral pods with no state bleed).
• What are the trade-offs? Unlike hosted CI, you own the server — its version, its ~1,800 plugins, its patch cadence — so plugin sprawl and an unpatched controller are real attack surface (CVE-2024-23897 let attackers read controller files, chained to RCE).
• When should I avoid it? When you’d rather not operate a build server: GitHub Actions / GitLab CI hand maintenance to the provider and cover most needs that aren’t air-gapped or deeply bespoke.
• What breaks if I remove it? Without a CI server the build reverts to “whoever remembers, on their own machine, with their own tool versions” — inconsistent and impossible to enforce.

Check your understanding

1. Why should a production Jenkins controller run no builds of its own? What does it hold that makes this important?
2. Define controller, agent, and executor, and explain what an agent label is for.
3. A build is stuck “in the queue” and never starts. Using the dispatch model, give two plausible causes and the corresponding fix.
4. What does Pipeline-as-code (a Jenkinsfile) give you that a freestyle job does not?
5. Plugins are called Jenkins’s “superpower and its tax.” Give one concrete benefit and two concrete risks of plugin sprawl.
6. CVE-2024-23897 let an attacker read arbitrary files on the controller. Why is reading files on the controller specifically so dangerous, and what two operational habits limit the blast radius?

Show answers

1. A build step runs arbitrary code. The controller holds the credentials store, all job configuration, and build history — the crown jewels. Running builds there means untrusted build code executes next to every secret, so production controllers offload all execution to agents.
2. Controller = the orchestrator (UI, scheduling, config, credentials, plugins). Agent = a machine/container that actually executes pipeline steps. Executor = one concurrent build slot on an agent (N executors = N simultaneous builds). A label tags an agent’s capabilities (linux, docker, gpu) so a pipeline can require the right environment.
3. The build’s label expression matches no available executor: (a) all matching agents’ executors are busy → add executors/agents or reduce concurrency; (b) no agent has the required label (or the agent is offline) → bring up/label an appropriate agent. (Either way: the scheduler is waiting for a free executor on a label-matching agent.)
4. The build definition lives in the repo, versioned and code-reviewed, so it’s diffable, survives a Jenkins rebuild, and is reproducible — versus freestyle config that lives inside Jenkins, invisible to review and lost if the server dies.
5. Benefit: integration with almost anything (Git, Docker, k8s, Slack…) via reusable plugins. Risks (any two): transitive dependency/version conflicts that break on upgrade; an abandoned plugin pinning you to old, vulnerable core; expanded security attack surface; harder, scarier upgrades.
6. The controller holds the secrets and the master key that decrypts the credentials store, so an arbitrary file read can leak credentials and escalate to RCE/lateral movement. Habits that limit it: keep the controller patched and keep it off the public internet (plus run no builds on it and minimize plugins).