GitOps principles¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the OpenGitOps principles and the Argo CD documentation; nothing was run.
In one line¶
GitOps keeps the desired state of a system declaratively in Git, and an agent inside the environment pulls it and continuously reconciles reality to match; a deploy is a commit, a rollback is a revert, history is the audit log, and the pipeline no longer needs credentials to the cluster.
The plain idea¶
In a classic pipeline, the last step pushes: CI runs kubectl apply or helm upgrade against the cluster, so CI holds cluster credentials, and nothing notices if someone changes the cluster by hand afterwards.
GitOps turns it around. A controller running in the cluster watches a Git repository and applies what it finds, again and again. Git becomes the one place where the desired state lives. Changing production means changing Git, through the same review as code. If someone edits the cluster directly, the controller sees the difference and (if allowed) puts it back.
How it works¶
The four principles (OpenGitOps)¶
| Principle | Meaning |
|---|---|
| Declarative | The desired state is expressed as data (manifests, charts, overlays), not as steps |
| Versioned and immutable | It is stored with full history and immutable versions: a Git repository |
| Pulled automatically | Agents fetch the desired state themselves |
| Continuously reconciled | Agents keep comparing actual with desired state and act on differences |
This is the Kubernetes reconcile loop (Architecture and the reconcile loop) applied one level up: the desired state lives in Git instead of only in the API server.
Push vs pull¶
flowchart TD
subgraph PUSH["Push-based CI/CD"]
A1["CI builds the image"] --> A2["CI runs helm upgrade<br/>with cluster credentials"]
A2 --> A3["Cluster: drift goes unnoticed"]
end
subgraph PULL["Pull-based GitOps"]
B1["CI builds the image and commits<br/>the new tag to the config repo"] --> B2["Controller in the cluster<br/>pulls the repo"]
B2 --> B3["Applies and keeps reconciling;<br/>drift is detected or reverted"]
end
| Push | Pull (GitOps) | |
|---|---|---|
| Who changes the cluster | The CI system | A controller inside the cluster |
| Cluster credentials in CI | Yes | No; CI only writes to Git |
| Drift after deploy | Unnoticed | Detected, optionally reverted |
| Audit trail | CI logs | Git history and reviews |
| Rollback | Re-run an old pipeline | Revert a commit |
Repository layouts¶
- Application repository vs config repository: Argo CD's own best practices recommend keeping manifests in a separate repository from application source code. Config changes (a replica count) do not trigger application builds; access to production config can be limited; history shows deploys cleanly.
- A folder per environment, with Kustomize overlays or Helm values per environment (Kustomize, and Helm vs Kustomize):
config-repo/
├── apps/sample-app/
│ ├── base/
│ └── overlays/
│ ├── dev/ image tag 1.5.0-rc1
│ ├── staging/ image tag 1.4.2
│ └── prod/ image tag 1.4.2
└── platform/ ingress controller, cert-manager, monitoring
Branches per environment are possible but make promotions hard to review (merges between long-lived branches drift); folders are the common recommendation (Branching and release strategies).
What GitOps does not solve¶
- Secrets: they must not sit in Git in plain text. Store references (External Secrets) or encrypted values (Sealed Secrets, SOPS) (Secrets from AWS into pods).
- Ordering: "migrate the database, then deploy" needs sync waves and hooks (Sync, hooks and waves).
- Database and data changes: a revert restores manifests, not data (Deployment strategies).
- Building and testing: CI still builds, tests and scans; GitOps starts where CI ends.
- Infrastructure outside the cluster: Terraform still owns VPCs and databases, unless you use controllers that manage cloud resources from Kubernetes.
A concrete example¶
sample-app moves from push to pull:
- CI builds and pushes image
1.4.2, then opens a pull request on the config repository changingoverlays/stagingto1.4.2. - Merging it is the staging deploy; Argo CD in the cluster syncs it.
- Promotion to production is a second pull request changing
overlays/prod, approved by the service owners (Promotion and rollback). - CI's AWS role loses its cluster access entry; only Argo CD can change the cluster.
- An engineer's manual
kubectl scaleduring an incident shows up as OutOfSync, and is reverted by self-heal unless the change is committed.
Why not just...?¶
- ...keep
helm upgradein CI, since it works? It needs cluster-admin-like credentials in CI, and the cluster can drift from what any repository says. GitOps removes both problems. - ...store manifests in the application repository? It works for small teams, but every config change looks like a code change, triggers builds, and mixes deploy history with code history. Separate repositories scale better.
- ...let GitOps manage everything, including databases? Stateful changes need ordering, backups and sometimes manual steps. Manage the database's manifests with GitOps, but plan data changes separately.
Commands to know¶
| Command | What it does |
|---|---|
git log --oneline -- apps/sample-app/overlays/prod |
Deploy history for production. |
git revert <commit> |
A rollback, as a reviewed change. |
argocd app diff sample-app-prod |
Difference between Git and the cluster. |
argocd app get sample-app-prod |
Sync and health status (Argo CD architecture and Applications). |
kubectl diff -k apps/sample-app/overlays/prod |
The same comparison with plain kubectl. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Manual hotfix disappears minutes later | Reconciliation | Self-heal setting | Self-heal reverted a change not in Git; commit the fix |
| Change merged, nothing deployed | Sync | Application status, repository access | Auto-sync off, wrong path or branch, controller cannot read the repository |
| Secret visible in Git history | Secrets | Repository contents | Plain Secret committed; rotate it and use references or encryption |
| Environments drift apart | Layout | Diff between overlays | Changes applied to one environment's folder only |
| Database migration ran after the new pods started | Ordering | Hooks and waves | No PreSync hook or wave ordering |
Interview questions¶
1. What is GitOps?
Answer
- Desired state declared in Git, versioned and immutable.
- An agent in the environment pulls it and continuously reconciles reality to it.
- Deploys are commits; rollbacks are reverts; Git history is the audit log.
2. Compare push-based CI/CD with pull-based GitOps.
Answer
- Push: CI applies changes with cluster credentials; drift goes unnoticed.
- Pull: an in-cluster controller applies from Git; CI only writes to Git.
- Pull removes cluster credentials from CI and detects drift.
3. Why separate the config repository from the application repository?
Answer
- Config changes do not trigger builds, and deploy history stays clean.
- Different access rules: more people write code than change production config.
- Argo CD's own best practices recommend it.
4. What does GitOps not solve by itself?
Answer
- Secrets (need references or encryption), ordering (hooks, waves), data migrations.
- Building and testing (still CI's job).
- Infrastructure outside the cluster (Terraform or cloud controllers).
5. An engineer scaled a Deployment by hand during an incident; ten minutes later it went back. Why, and what should they have done?
Answer
- Auto-sync with self-heal reverted the change, because Git still said otherwise.
- Commit the change to Git (fast path for emergencies), or pause auto-sync for that app explicitly and record it.
- Afterwards, make sure Git reflects the intended state.
6. How would you structure a config repository for three environments?
Answer
- A base plus one overlay or values file per environment, in folders on one main branch.
- Promotion is a pull request changing the next environment's folder.
- Platform components in their own folders or repository, with separate ownership.
Try it¶
- Draw the push and pull diagrams from memory, and mark where credentials live in each.
- Using the Kustomize example (Kustomize, and Helm vs Kustomize), write the one-line change that promotes
1.4.2from staging to prod. - For a service you know, list everything that would still happen outside Git if you moved it to GitOps.
Zoom out¶
GitOps is CI/CD where the last step is a Git commit and a controller does the deploy (CI/CD concepts). Argo CD is the most common controller on EKS; the following pages cover how it works, how sync is ordered, how to manage many applications, how deletion works, and how to promote and roll back (Argo CD architecture and Applications).