Jenkins → Argo CD: Push-CI, Pull-CD

You now have both halves: Jenkins builds and tests code, and Argo CD reconciles a cluster to git. The obvious move is to wire them together — “Jenkins deploys with Argo CD.” But the way you wire them is the entire lesson of this page, because the naive wiring recreates exactly the manual, dangerous step GitOps was invented to remove. The right architecture splits the work along a trust boundary: Jenkins pushes artifacts from outside the cluster; Argo CD pulls desired state from inside it. Get that split right and CI never touches production directly.

CI and CD are different jobs

It’s tempting to treat “ship it” as one pipeline, but CI and CD answer different questions, run in different places, and need different permissions:

	CI (Jenkins)	CD (Argo CD)
Question	”Is this commit good and packaged?"	"Does the cluster match git?”
Runs	outside the cluster, on build agents	inside the cluster, as a controller
Trigger	a code push	a change to the config repo
Needs	source access, a registry push token	the cluster’s own API (in-cluster)
Must not hold	cluster-admin credentials	your source-signing keys

The boundary between them is a security boundary. The single most important design decision in this whole chapter is which side holds the cluster credentials — and the answer is neither Jenkins nor any external box.

The push-based anti-pattern

Here’s the wiring almost everyone reaches for first. Add a deploy stage to the Jenkinsfile:

stage('Deploy') {
  steps {
    sh 'kubectl apply -f k8s/'          // or: helm upgrade --install app ./chart
  }
}

It works in the demo. It is also the thing to avoid, for four compounding reasons:

• CI must hold cluster-admin credentials. For Jenkins to kubectl apply, an external, internet-adjacent build server now stores a kubeconfig that can do anything to production. The crown jewels sit in the box that runs untrusted pull-request code (recall the Jenkinsfile sandbox problem).
• The cluster trusts an outside machine. A push model means the cluster accepts mutations initiated from outside. Compromise the CI server and you have compromised production, directly.
• Drift is invisible. Once Jenkins fires kubectl apply and exits, nothing watches the cluster. A hand-edit, a crashed controller, a half-applied change — nobody notices, because the deployer is a fire-and-forget script, not a reconciliation loop.
• There’s no git audit of what’s running. The pipeline log says “applied,” but the live state isn’t a reviewed artifact. “What exactly is in prod right now, and who approved it?” has no clean answer.

This is push-based CD: CI reaches into the cluster. Everything below is about turning that arrow around.

Pull-based GitOps: turn the arrow around

Argo CD runs inside the cluster as a controller. Instead of an outside box pushing changes in, Argo CD pulls desired state from git and reconciles the cluster to match — using the cluster’s own API from within. The consequence is the headline: cluster credentials never leave the cluster. There is no kubeconfig on the build server, because the build server never talks to the cluster at all. Jenkins’s job ends at “the artifact and the desired state are published”; Argo CD’s job is “make the cluster equal git, forever.”

The handshake

So how does a green build become a running deployment without Jenkins ever touching the cluster? Through git and a registry — two systems both sides already trust. The flow:

  ┌───────────────┐   push    ┌────────────────────┐
  │   APP REPO    │  + hook   │       JENKINS      │   ── CI: PUSH side ──
  │  source code  ├──────────►│  build • test •    │   (outside the cluster)
  └───────────────┘           │  build image       │
                              └─────────┬──────────┘
                       1. docker push   │
                          registry/app:<git-sha>
                                        ▼
                              ┌────────────────────┐
                              │      REGISTRY      │  immutable image, tagged by commit SHA
                              └─────────┬──────────┘
                       2. commit: bump image tag
                          in values.yaml (PR or push)
                                        ▼
  ┌───────────────┐  git push ┌────────────────────┐
  │  CONFIG REPO  │◄──────────┤   (still Jenkins)  │  writes the new tag into the desired state
  │ Helm/manifests│           └────────────────────┘
  └───────┬───────┘
  ╌╌╌╌╌╌╌╌│╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ trust boundary ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
  ┌───────▼─────────────────────────────────────────────────┐
  │  CLUSTER                                                 │   ── CD: PULL side ──
  │   ┌────────────┐  3. detect git change (poll / webhook)  │   (inside the cluster)
  │   │  ARGO CD   │──── pull desired = config repo          │
  │   │ controller │──── sync ─► cluster == git              │
  │   └────────────┘     cluster creds never leave here      │
  └─────────────────────────────────────────────────────────┘

Walk the three numbered hops:

1. Build, test, push the image — tagged with the git SHA. Jenkins produces the artifact and tags it with the immutable commit hash (registry/app:1a2b3c4), exactly as the pipelines page tagged with github.sha. The tag is content-traceable: that image came from that commit, and nothing else will ever reuse the tag.
2. Bump the tag in the config repo. Jenkins makes a tiny git change — set image.tag: 1a2b3c4 in a Helm values.yaml (or patch a manifest) — by committing, or by opening a pull request if you want a human gate before prod.
3. Argo CD detects the git change and syncs. The controller sees the config repo move, pulls the new desired state, and reconciles the cluster (the ~3-minute poll, or near-instant with a webhook). The deploy is that commit landing. Nobody ran kubectl.

If you’d rather not have CI write to git at all, Argo CD Image Updater is the alternative: it runs in the cluster, watches the registry for new tags matching a pattern, and writes the bump back to git itself — so even step 2 happens from inside the trust boundary.

App repo vs config repo: and why they’re separate

Notice there were two repos: the app repo (source code) and the config repo (the Kubernetes manifests / Helm values that Argo CD watches). That separation is deliberate, and the sharpest reason is mechanical:

  one-repo trap:
    CI builds app  ─► CI commits new image tag to SAME repo
        ▲                                    │
        └───────── that commit re-triggers CI ◄┘   ← infinite loop

If the image tag lived in the same repo as the source, Jenkins’s tag-bump commit would itself be a push to that repo — which re-triggers CI, which bumps the tag again, forever. Splitting source from config breaks the loop: a commit to the config repo isn’t watched by CI, only by Argo CD. (If you must share a repo, the cruder guards are scoping CI to source paths or tagging the bot commit [skip ci] — but a separate config repo is the clean answer, and it also lets the two have different reviewers and permissions.)

Under the hood — the tag bump, the loop, and write-back

The “deploy” is astonishingly small: a one-line diff to a YAML file.

apps/app/values.yaml

image:
  repository: ghcr.io/acme/app
  tag: "0f9e8d7"
  tag: "1a2b3c4"

That diff is the release. It’s reviewable, attributable in git blame, and revertible. Three mechanics make it work cleanly:

• The bump is a commit, so it inherits git’s whole machine — review, history, protected branches, required approvals before the prod path.
• The loop is avoided by repo separation (above): the config repo isn’t a CI trigger, so the bot commit doesn’t cascade.
• Image Updater’s write-back offers two modes: it can update the live Argo CD Application directly via the API, or — the GitOps-pure choice — commit the new tag back to the config repo in git, keeping git the single source of truth even for automated bumps. Either way the cluster credentials stay inside the cluster.

Why the split is safer

Line up what the pull model buys, and it’s the book’s whole safety thesis in one architecture:

• Least privilege. No cluster-admin kubeconfig in CI. Compromising the build server no longer means compromising production — Jenkins can push an image and open a PR, nothing more. (See secrets management.)
• Git is the single source of truth and the audit log. What’s supposed to be running is a reviewed git state; git log on the config repo is the deploy history, with authors and approvals.
• Drift auto-heals. Argo CD’s self-heal reverts out-of-band cluster edits back to git — the fire-and-forget blindness of kubectl apply is gone.
• Rollback is git revert. No frantic hand-rebuild; revert the bad bump and the cluster reconciles to the previous, known-good state.
• Environments are directories or branches in the config repo. Promotion dev→staging→prod is a change to the config repo — copy the tested image tag from the staging path to the prod path. The same image that passed staging is what reaches prod; you promote a reference, you don’t rebuild.

By the numbers — rollback under pressure

The payoff shows up most starkly when something is on fire. Compare recovering from a bad release:

push model (manual):
  find the last-good image tag         ~3 min   (dig through CI logs)
  re-run a deploy job / kubectl apply   ~5 min
  authenticate, hope creds still valid  ~2 min
  verify, discover a missed config      ~5 min
  total, under incident stress         ≈ 15 min  — and unaudited

pull model (GitOps):
  git revert <bad-bump-commit> ; push   ~1 min
  Argo CD reconciles (or `argocd app sync`) ~1–3 min
  total                                 ≈ 2–4 min — reviewed + attributed

The numbers are illustrative, but the structural difference is not: a push-model rollback is a human reconstructing a known-good state from memory under stress, while a pull-model rollback is git restoring a state it never lost. The blast radius shrinks the same way — a leaked CI token can push a bad image, but it can’t reach into the cluster, because there’s nothing in CI that can.

It actually happened — why you don’t want cluster creds in CI

The reason “least privilege in CI” is not abstract: CI environment variables get exfiltrated. In the April 2021 Codecov incident, an attacker modified Codecov’s Bash Uploader script so that, when it ran inside customers’ CI pipelines, it shipped the pipeline’s environment variables — tokens, keys, and credentials — to an attacker-controlled server. Any secret sitting in CI was potentially swept up.

Now replay that with a push-based deploy: the exfiltrated env would have included a cluster-admin kubeconfig, handing the attacker production directly. With pull-based GitOps there is no such credential in CI to steal — the cluster creds never left the cluster. This is precisely why the trust boundary on this page matters; the full story lives in supply-chain attacks and secrets management, and it is the strongest argument for keeping CD on the pull side.

The thread

What manual, error-prone step does this remove — and how does it make production safer? The push model still had a human (or a CI script standing in for one) reaching into the cluster with admin credentials to kubectl apply or helm upgrade — fire-and-forget, unaudited, with the keys to production sitting on an internet-adjacent build box. Splitting CI from CD removes that step entirely: Jenkins’s reach ends at publishing an immutable image and a one-line git bump, and Argo CD — inside the cluster — pulls that reviewed desired state and converges to it on its own. The credential that could wreck production never leaves the cluster, the deploy history is git log, drift self-heals, rollback is git revert, and promotion is moving a tested image reference between directories. You remove the manual deploy and shrink the blast radius of a CI compromise in the same move. That’s the synthesis of this whole part: CI earns the artifact; GitOps decides, from inside, when the cluster is allowed to change.

The architect’s lens

The five questions, applied to the push-CI / pull-CD split itself:

• Why does it exist? Because CI and CD are different jobs with different trust boundaries — the split exists so CI never holds cluster-admin credentials on an internet-adjacent box that also runs untrusted PR code.
• What problem does it solve? The kubectl apply-in-the-pipeline anti-pattern: it hands off via a git-SHA image tag and a separate config repo, so Jenkins only pushes an artifact and a one-line tag bump, while Argo CD pulls and reconciles from inside the cluster.
• What are the trade-offs? You run two repos to break the CI re-trigger loop, and the tag-bump handoff adds a poll delay (~3 min) unless webhooked — in exchange for ~2–4 min git revert rollbacks instead of ~15 min reconstructing a known-good state under stress.
• When should I avoid it? A throwaway demo or a non-Kubernetes target may not justify the config-repo machinery — but the moment real cluster credentials are in play, the split is the whole point.
• What breaks if I remove it? Cluster creds move back into CI, so a leaked token (à la Codecov 2021) hands an attacker production directly, drift goes unwatched, and the deploy audit trail disappears.

Check your understanding

1. CI and CD are described as “different jobs with different trust boundaries.” For each, say where it runs and which credential it must not hold.
2. List the four problems with putting kubectl apply (or helm upgrade) directly in the Jenkins pipeline.
3. Walk the handshake from green build to running deploy. Why is the image tagged with the git SHA specifically, and at what point does anything touch the cluster?
4. Why are the app repo and the config repo kept separate? Describe the failure that happens if the image tag lives in the same repo as the source.
5. What is Argo CD Image Updater, and how does its git write-back mode keep git the single source of truth while still removing CI’s need to commit?
6. A leaked CI token is bad either way. Explain, with the push vs pull split, why it’s far less catastrophic under pull-based GitOps — and connect it to the Codecov 2021 incident.

Show answers

1. CI (Jenkins) runs outside the cluster on build agents and must not hold cluster-admin credentials. CD (Argo CD) runs inside the cluster as a controller (using the cluster’s own API) and must not hold your source-signing keys. The boundary between them is a security boundary, and ideally neither external box holds the cluster creds at all.
2. (1) CI must store cluster-admin credentials on an internet-adjacent box that also runs untrusted PR code; (2) the cluster now trusts mutations initiated from outside, so a CI compromise is a prod compromise; (3) drift is invisible — apply is fire-and-forget, nothing reconciles afterward; (4) there’s no git audit of what’s actually running — the live state isn’t a reviewed artifact.
3. Jenkins builds, tests, and pushes the image tagged with the immutable git SHA to the registry; then it bumps that tag in the config repo (commit or PR); then Argo CD detects the git change, pulls the new desired state, and syncs the cluster. The SHA is used because it’s immutable and content-traceable — that image came from exactly that commit and the tag is never reused. Nothing touches the cluster until Argo CD reconciles from inside it.
4. Separation prevents an infinite CI loop: if the image tag lived in the source repo, Jenkins’s tag-bump commit would push to that repo and re-trigger CI, which bumps again, forever. A commit to a separate config repo isn’t watched by CI (only by Argo CD), breaking the loop — and it also allows different reviewers/permissions. (Crude fallbacks if sharing a repo: path-scoped triggers or a [skip ci] tag.)
5. Image Updater runs in the cluster and watches the registry for new tags matching a pattern. In git write-back mode it commits the new tag back to the config repo, so the bump is still a reviewed, attributable git change (git stays the source of truth) — but CI no longer needs to write to git, and the whole bump happens from inside the cluster’s trust boundary.
6. Under push, CI holds a cluster-admin kubeconfig, so an exfiltrated CI secret hands an attacker production directly. Under pull, that credential doesn’t exist in CI — the cluster creds never leave the cluster — so the same leak can at most push a bad image, not reach the cluster. The April 2021 Codecov Bash-Uploader compromise exfiltrated CI environment variables (tokens/keys/credentials) from customer pipelines; with push-based CD that haul could have included the kube creds, which is exactly the risk pull-based GitOps designs away.