/scaffold

Kubernetes Manifest Scaffolding Reference

Purpose: Scaffold raw Kubernetes manifests (Deployment, Service, Ingress, ConfigMap, Secret) and kustomize overlays for a workload that does not yet need full Helm packaging. Covers label conventions, namespace layout, resource requests/limits defaults, and probe wiring so the workload is admission-safe on first apply.

Scope Boundary

• Scaffold k8s: raw manifest authoring and kustomize overlay layout (Deployment/Service/Ingress/ConfigMap/Secret, labels, namespace, requests/limits, probes).
• Scaffold helm: reusable, versioned chart packaging. If the workload must ship to 3+ environments or multiple tenants, escalate from k8s to helm.
• Scaffold terraform: cluster-level provisioning (EKS / GKE / AKS control plane, node groups, IRSA). Manifests live in k8s; the cluster that runs them lives in terraform.
• Gear[gha] (elsewhere): CI/CD wiring — kubectl apply -k, kustomize build, ArgoCD / Flux sync. k8s produces the manifests; Gear[gha] wires the rollout.
• Gateway (elsewhere): public API contract, versioning, breaking-change review. Ingress routing lives in k8s; API rules (auth, rate limit, schema) live in Gateway.
• Native (elsewhere): mobile build/release concerns are out of scope for k8s — route those to Native.

If the request is "package this workload for reuse" → helm. If it is "ship this single workload to one cluster" → k8s.

Workload Shape (minimum set)

Pod Security Standards (2026 baseline)

By 2026, Pod Security Admission (PSA) enforcing the restricted policy is the default posture for new namespaces. Label every namespace explicitly — relying on the cluster-wide default has bitten too many teams:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

The restricted policy in 2026 requires every Pod spec to set:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 65532
    fsGroup: 65532
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: [ALL]

When the workload legitimately needs an escape (hostNetwork, privileged, write-able root FS), drop the namespace to baseline only with a written exception — never silently exempt workloads.

For stronger admission-time validation (custom policies, image-signature requirements), wire Kyverno (or OPA Gatekeeper) at the cluster level — PSA covers the standard cases, Kyverno covers the rest.

Label & Annotation Convention

Use the recommended Kubernetes labels — they are free, queryable, and required by many tools (ArgoCD, Prometheus operator).

metadata:
  labels:
    app.kubernetes.io/name: payments-api
    app.kubernetes.io/instance: payments-api-prod
    app.kubernetes.io/version: "1.24.3"
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: payments
    app.kubernetes.io/managed-by: kustomize

Selector labels (Deployment.spec.selector.matchLabels) must be a stable subset — never include version or instance in selectors, they break rolling updates.

Resource Requests / Limits Defaults

Never ship without requests. Unrequested pods land in BestEffort QoS and are evicted first.

Prefer omitting limits.cpu (CPU throttling hurts latency more than it helps); always set limits.memory to prevent noisy-neighbor OOM cascade.

Probes

Liveness and readiness are not optional. Startup probe is required for slow-booting apps (>10s).

readinessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 15
  failureThreshold: 3
  initialDelaySeconds: 30
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30  # allow 150s for JVM / cold-cache warmup

Never point liveness at / or any path that touches downstream dependencies — liveness failure restarts the pod and cascades an outage through the whole tier.

Kustomize Overlay Layout

k8s/
  base/
    kustomization.yaml
    namespace.yaml
    deployment.yaml
    service.yaml
    configmap.yaml
  overlays/
    dev/
      kustomization.yaml     # patches replicas=1, lower requests
      config-patch.yaml
    staging/
      kustomization.yaml
    prod/
      kustomization.yaml     # patches replicas, HPA, PDB, prod image tag
      hpa.yaml
      pdb.yaml

Use kustomization.yaml images: to pin tags per overlay. Use configMapGenerator / secretGenerator to avoid drift from hand-edited YAML.

Secret Handling

Raw Secret objects are base64, not encrypted. Three acceptable paths:

1. External Secrets Operator (ESO) → pulls from AWS Secrets Manager / GCP Secret Manager / Vault.
2. Secrets Store CSI Driver → mounts secrets as files, no K8s Secret object at all.
3. Sealed Secrets (Bitnami) → when the repo must be the source of truth and no secret manager exists.

Never commit a Secret with real data, even in dev/. Commit a placeholder and let ESO / CSI hydrate.

Anti-Patterns

• Deploying to default namespace — collides with cluster tooling and breaks RBAC granularity.
• latest image tag — Kubernetes will not pull-refresh a running pod, and rollback becomes impossible.
• Missing resources.requests — evicted first under pressure, kills HPA accuracy.
• readinessProbe pointing at a path that exercises the database — startup storms and thundering-herd restarts.
• Single-replica Deployment behind a Service with no PDB — rolling node drain = full outage.
• One monolithic ConfigMap shared across unrelated workloads — changing one key restarts every consumer.
• Including version label in selector.matchLabels — makes rolling updates orphan the old ReplicaSet.
• Baking environment config into the image — breaks the 12-factor boundary and forces a rebuild per env.
• Setting limits.cpu aggressively on latency-sensitive APIs — CPU throttling shows up as p99 spikes.
• Using hostNetwork: true or hostPath volumes without a documented reason — breaks admission policies and portability.

Handoff

When the k8s manifests are ready:

• To Gear[gha]: the kustomize build overlays/<env> command, the image tag convention, and the promotion flow (dev → staging → prod). Gear[gha] wires kubectl apply / ArgoCD sync / Flux Kustomization.
• To Sentinel: RBAC assumptions, ServiceAccount scope, NetworkPolicy posture (default-deny or not), and secret-handling path (ESO / CSI / Sealed Secrets).
• To Beacon: probe endpoints, Prometheus scrape annotations (prometheus.io/scrape, prometheus.io/port), and the metrics/log labels Beacon will key dashboards on.
• To Canvas: topology — namespaces, services, ingress routes — so Canvas can render the cluster view.
• Upgrade path: if the workload is about to be installed into a second cluster or handed to another team, re-scaffold as a Helm chart via scaffold helm.