In this guide — jump to a section
Follow a Three-Line Change Into Production
Imagine Alex fixes an order-validation bug. Three lines change. He commits it as a83f29c and pushes the branch.
git add .
git commit -m "Add order validation"
git push origin feature/order-validationAlex has finished the code change. The pipeline now needs to test, package, and deploy it safely.
Before commit a83f29c is allowed anywhere near production, the system has to answer a series of uncomfortable questions. Did the tests pass? Is the dependency tree safe? Which container image came from this commit? Who is allowed to promote it? Which identity can touch the cluster? And after Kubernetes accepts the manifest, can a customer actually place an order?
A Google Cloud CI/CD pipeline should record the source change, test results, artifact, approval, and rollout outcome. Those records let the team review a release and recover if it causes a problem.
a83f29c—from GitHub to GKE. Each stage exists because it answers a question the previous stage cannot. This is an architecture walkthrough, not a complete deployable starter repository. Basic familiarity with Git, containers, and Kubernetes will help; the YAML examples illustrate individual stages.The Components of the CI/CD Pipeline
The path is simple enough to draw. GitHub owns the source change. Cloud Build validates and packages it. Artifact Registry stores the image we trust. Cloud Deploy moves that image through environments. GKE runs it. Observability tells us whether the release deserves to stay there.
Commit / PR
Build • Test • Scan
Immutable image
Release • Promote
Run workload
Prove it works
The important part is not the number of Google Cloud products in the diagram. It is the separation of responsibilities. Source control should not secretly be your deployment engine. Your build identity should not become your runtime identity. Production should not depend on somebody remembering the right kubectl command.
A healthy platform lets you answer six questions quickly: which source change produced the release, which build validated it, which exact image is being deployed, who promoted it, what version is actually running, and whether the application became healthier or worse after the change.
Start with the repository: what exactly changed?
Before Cloud Build gets involved, the repository has to make one thing clear: what is the deployable unit?
order-service/
├── src/
├── tests/
├── Dockerfile
├── cloudbuild.yaml
├── skaffold.yaml
├── clouddeploy.yaml
├── kubernetes/
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── hpa.yaml
│ └── pdb.yaml
└── README.mdThere is no universal repository layout. But there should be very little ambiguity about what gets built, how it is tested, and how it is deployed. If a monorepo contains several independently deployable services, treat them as separate delivery units even if they share one Git repository.
Not every Git event deserves the same pipeline
Commit a83f29c first appears on a feature branch. That event should give Alex fast feedback. A pull request needs stronger evidence. A merge to main can produce a release artifact. Production promotion should use that already-built artifact—not rebuild it again.
feature/*
└─ build + unit test + fast security checks
pull request
└─ full validation + review status
main
└─ build release artifact + publish
release / promotion
└─ deploy the already-built artifact
production
└─ approval + rollout + verificationThe question is not “Which trigger should I configure?” The better question is: what has to be true before this change is allowed to move on?
What should never be in Git
- service-account JSON keys
- database passwords or API tokens
- production kubeconfig files
- environment-specific secrets copied into YAML
- artifacts the pipeline can reproduce
The repository should describe the application and its desired deployment state. It should not quietly become a credential vault.
Validate the Commit Before Building a Release
Cloud Build receives commit a83f29c and runs the checks required before publishing a release image.
A useful CI pipeline usually moves from cheap checks to expensive ones. Fail fast. There is little value in spending minutes building and scanning a container when a unit test already proves the change is broken.
- Restore dependencies. Make dependency resolution reproducible.
- Run unit tests. A failed test should stop the build.
- Run integration or contract checks. Add them where the service boundary justifies the cost.
- Build the container. Only after the source has passed the earlier checks.
- Push to Artifact Registry. Publish something traceable to this build.
- Run vulnerability and policy checks. Treat findings as delivery data, not an afterthought.
- Create a Cloud Deploy release. Hand CD an immutable artifact instead of asking it to rebuild.
steps:
- name: 'gcr.io/cloud-builders/docker'
args:
- build
- -t
- us-central1-docker.pkg.dev/$PROJECT_ID/apps/order-service:$SHORT_SHA
- .
- name: 'gcr.io/cloud-builders/docker'
args:
- push
- us-central1-docker.pkg.dev/$PROJECT_ID/apps/order-service:$SHORT_SHA
images:
- us-central1-docker.pkg.dev/$PROJECT_ID/apps/order-service:$SHORT_SHAThe image you tested should be the image you deploy
The tests pass. Commit a83f29c is now a container image. This is where traceability becomes operationally useful.
order-service:latest # easy to type, hard to trust
order-service:a83f29c # traceable to a commit
order-service@sha256:... # immutable content identityA commit-based tag is already much better than latest when somebody is debugging an incident at 3 a.m. For deployment, a digest is stronger because it identifies the image content itself.
Use the same artifact throughout promotion: build once, then promote the same artifact.
Build
↓
order-service@sha256:abc...
↓
DEV
↓
QA
↓
PRODIf Dev, QA, and Production each rebuild from source, you now have three artifacts that came from similar code. Promotion should increase confidence in one artifact, not manufacture a new one at every stage.
Artifact Registry is also where build and deployment responsibilities cleanly meet. Once CI has produced and published a trusted image, CD can ask a much simpler question: is this exact artifact allowed to move forward?
Every stage needs its own identity
Identify the service account or workload identity used at each pipeline stage.
That omission matters. Builds, deployments, and workloads all act as identities with permissions. Reusing one powerful account everywhere may be convenient, but it turns one compromise into everybody’s problem.
Developer identity
≠
Build identity
≠
Deployment identity
≠
Runtime identityUse a dedicated build identity with only the permissions the build needs. Give the deployment identity deployment permissions. Give the GKE workload only the API access the application itself needs.
For GKE workloads that need Google Cloud APIs, Workload Identity Federation for GKE avoids placing long-lived service-account keys inside pods.
Secrets do not belong in cloudbuild.yaml
A delivery pipeline needs configuration. That does not mean it needs plaintext secrets.
Keep non-sensitive configuration in versioned manifests or overlays where appropriate. Keep secrets in a secrets system such as Secret Manager and grant access through IAM and workload identity.
Repository
├─ deployment.yaml ✓
├─ service.yaml ✓
├─ non-sensitive config ✓
└─ production-password ✗
Secret Manager
└─ production-password ✓ controlled accessPromote the release. Do not rebuild it.
CI answered one question: “Can we produce a trustworthy artifact?” Continuous delivery answers another: “How does that artifact move through environments without becoming a manual adventure?”
Our image from commit a83f29c already exists. From this point forward, the goal is to move that same release through Dev, QA, and Production while preserving its identity and history.
DEV
↓ verification
QA
↓ approval
PRODCloud Deploy models that progression as releases and rollouts across targets. For GKE, Skaffold is part of the rendering and deployment flow; your manifests still describe the application, while Cloud Deploy manages the release lifecycle around them.
Why kubectl apply from a laptop is not a release strategy
There is nothing wrong with kubectl apply. It is useful for learning Kubernetes and excellent for troubleshooting.
For production releases, use a controlled deployment process that records the cluster context, manifest, identity, and approval.
Git
↓
CI
↓
Immutable artifact
↓
Release
↓
Approval
↓
Deployment system
↓
ProductionA controlled delivery path leaves behind a durable record of the source, artifact, identity, approval, rollout, and result. Humans should still be able to investigate production. They just should not be the hidden automation layer.
Getting into GKE is not the finish line
The image has finally reached the cluster. But getting a container into Kubernetes is not the same as designing a workload that belongs in production.
For a typical stateless service, start with a few basics: resource requests and limits, readiness and liveness behaviour, a deliberate rolling-update strategy, and a scaling policy that reflects actual demand. Depending on the workload, a PodDisruptionBudget may also belong in the availability design.
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service
spec:
replicas: 3
selector:
matchLabels:
app: order-service
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: order-service
spec:
containers:
- name: order-service
image: order-service
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1"
memory: "512Mi"
readinessProbe:
httpGet:
path: /ready
port: 8080
livenessProbe:
httpGet:
path: /health
port: 8080The Deployment selector and Pod labels must match. Before using this example, replace image: order-service with your actual image reference or have Skaffold substitute the built image. The application must provide /ready and /health on port 8080; adapt the probes to the application. See the Kubernetes Deployment documentation for the full resource structure. The important distinction is between alive and ready. A process can be running while still being unable to serve useful traffic. Readiness gives the platform a way to keep that instance out of service until it is actually prepared.
The pipeline is green. Can customers use the app?
Kubernetes accepted the manifest. The pods became Ready. The deployment command returned success.
None of those facts proves that checkout works, that the service can reach its database, that an API contract still holds, or that latency has not doubled.
Cloud Deploy supports post-deployment verification. That gives the release process a clean place for smoke tests, integration checks, and other evidence that should exist before promotion continues.
Deploy
↓
Pods become Ready
↓
Smoke / integration verification
↓
PASS ───────────→ promote
FAIL ───────────→ stop / investigate / rollbackNow production gets a vote. Watch logs, latency, saturation, restart rates, dependency failures, and—most importantly—the user journey that actually matters to the business.
- Logs: are new errors appearing after this release?
- Metrics: did latency, saturation, restart rate, or dependency failures move?
- Service indicators: are users still completing the operation that matters?
- Alerts: will somebody be told when the release causes real damage?
- Release context: can the team correlate a change in behaviour with the exact deployment?
Prepare and Test the Rollback Process
Retaining previous release artifacts gives the team a known version to restore.
Suppose the release based on commit a83f29c reaches production and the error rate climbs within minutes. If the previous release is still a known artifact with a known deployment history, recovery is a controlled operation instead of a reconstruction exercise.
new release deployed
↓
error rate rises
↓
alert + verification evidence
↓
rollback
↓
last known-good release restoredCloud Deploy can roll back by creating a rollout from a previously successful release. The important idea is not the button or command. It is that rollback was designed into the release model before the incident happened.
Rolling back is not the same as fixing the bug. It is buying back stability so the team can fix the bug without production pressure distorting every decision.
More tools do not make the pipeline mature
Once the basic path works, teams naturally ask about canaries, policy enforcement, provenance, GitOps, private build pools, multi-cluster delivery, and stronger supply-chain controls.
Those may all be useful. They are not substitutes for the fundamentals.
- A canary is not useful if you cannot identify the artifact being canaried.
- GitOps does not compensate for secrets committed to Git.
- A vulnerability scanner does not help if nobody owns the findings.
- An approval button is security theatre if the approver has no evidence to review.
- More stages do not make a pipeline mature. Better decisions do.
Start with traceability, identity, artifact integrity, promotion, verification, rollback, and observability. Add sophistication when it solves a problem you actually have.
One commit, all the way to a running release
Put the journey together and commit a83f29c now has a complete history:
Developer
↓
Git commit a83f29c
↓
GitHub
↓
Pull request
↓
Cloud Build
├─ compile
├─ unit tests
├─ integration tests
├─ security checks
└─ container build
↓
Artifact Registry
immutable image
↓
Cloud Deploy
release + rollout
↓
DEV verification
↓
QA
↓
production approval
↓
GKE
↓
health + observability
↓
keep / rollback / improveNotice what is missing from the middle of that diagram: a developer manually carrying an image from one environment to another. Human judgement still matters—especially around review and production approval—but the mechanics should be repeatable.
Automate the repeatable steps and retain human review for changes and production approvals.
Check the Pipeline Before Using It for Production
A production-grade CI/CD pipeline is not defined by how impressive the architecture diagram looks.
It is defined by how confidently the team can answer a small set of uncomfortable questions.
- Can an untested commit reach production?
- Can somebody deploy around the pipeline without leaving evidence?
- Can we identify the exact source commit and container digest running right now?
- Do build, deployment, and runtime identities have separate permissions?
- Are we promoting one artifact or rebuilding differently in every environment?
- Will a technically successful but functionally broken release be detected?
- Can we restore the last known-good version quickly?
- Can an engineer understand what happened without reverse-engineering five disconnected systems?
If those answers are good, the Google Cloud CI/CD pipeline has become more than automation. It has become part of the engineering platform: a repeatable way to turn source code into a production change with evidence attached at every step.
Keep these release records available for troubleshooting and future reviews.
Official Google Cloud references
Implementation details evolve, so use the current Google Cloud documentation when you turn this architecture into a real platform:
- Cloud Build triggers — repository event automation for pushes and pull requests.
- User-specified Cloud Build service accounts — least-privilege build identities.
- Artifact Analysis scanning — vulnerability scanning for supported artifacts.
- Cloud Deploy overview — releases, rollouts, promotion, approvals, and rollback.
- Deploy to GKE using Cloud Deploy — delivery pipelines and targets.
- Cloud Deploy verification — post-deployment verification.
- Workload Identity Federation for GKE — workload access to Google Cloud APIs without long-lived keys.