Cloud Logging, Audit Attribution, APF & API Server Traces
Table of Contents
- 1. API Server Slow-Request Trace Logs (
trace.go) (Lines 11–54) - 2. Heavy API Clients (
httplog.go), Audit Attribution & SlowLISTs (Lines 56–117) - 3. API Priority & Fairness (APF) Throttling (
429/504/404) (Lines 119–143) - 4. Hot-Object
409 Conflicts & Leader-ElectionRenewDeadlineTimeouts (Lines 145–169) - 5.
pods/exec, Konnectivity, Cloud KMS & IAM Webhook Authorizer (Lines 171–211)
1. API Server Slow-Request Trace Logs (trace.go)
kube-apiserver component logs (API_SERVER) are an optional feature that is
not enabled by default and must be explicitly turned on
(--logging=SYSTEM,WORKLOAD,API_SERVER). State this and link to
https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs
when recommending apiserver log analysis.
- Search for
traceorTraceentries: Queryapiservercomponent logs (resource.type="k8s_control_plane_component" AND resource.labels.component_name="apiserver") fortraceorTrace(e.g.,textPayload:"Trace["). - Structure & interpretation:
kube-apiserveruses upstreamk8s.io/utils/trace(trace.go) to log requests exceeding a latency threshold (typically 500ms). The header records the operation ("List","Get","Create","Patch"),url,user-agent,audit-id,clientIP, start timestamp, andtotal time. Nested sub-traces appear in brackets[...]and steps exceeding the step threshold appear as---"<StepName>" <duration>ms. - Distinguishing storage bottlenecks vs. slow-client serialization:
- Slow backend storage (
etcd) read or write:cacher.GetList,List(recursive=true) etcd3, orGuaranteedUpdate etcd3(withinitial value restored/Txn call completed) dominatestotal time:I1015 11:36:56.835120 10 trace.go:236] Trace[123456789]: "List" url:/api/v1/pods,user-agent:my-controller/v1.0.0,audit-id:11111111-2222-3333-4444-555555555555,client:10.0.0.1 (15-Oct-2026 11:36:49.460) (total time: 7374ms): Trace[123456789]: ["List(recursive=true) etcd3" 6890ms (11:36:49.461)] Trace[123456789]: ---"Objects listed" 6891ms (11:36:56.352) Trace[123456789]: ---"SerializeObject" 480ms (11:36:56.832) - Slow client / response serialization bottleneck:
"Objects listed"is fast, while"SerializeObject"and---"Write call succeeded"(or---"Write call failed") dominatetotal timebecause a slow or bandwidth-constrained client takes seconds to read a large payload:I1015 11:40:12.104512 10 trace.go:236] Trace[987654321]: "List" url:/api/v1/pods,user-agent:slow-watcher/v0.1,audit-id:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee,client:10.0.0.2 (15-Oct-2026 11:40:01.002) (total time: 11102ms): Trace[987654321]: ---"Objects listed" 45ms (11:40:01.047) Trace[987654321]: ["SerializeObject" 11055ms (11:40:01.047)] Trace[987654321]: ---"Write call succeeded" 11054ms (11:40:12.101)
- Slow backend storage (
- Upstream docs & future evolution (cite both): Link to
https://github.com/kubernetes/utils/blob/master/trace/README.md for
trace.goand mention https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/647-apiserver-tracing (KEP-647: APIServer Tracing via OpenTelemetry) as a possible future change.
2. Heavy API Clients (httplog.go), Audit Attribution & Slow LISTs
httplog.goHTTP access logs: Requires enablingAPI_SERVERcomponent logs (--logging=SYSTEM,WORKLOAD,API_SERVER; link to https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs). State explicitly that thehttplog.golog structure is very stable across Kubernetes releases (making regex extraction and Log Analytics queries reliable), and include this example log line:I1015 11:36:56.835535 10 httplog.go:137] "HTTP" verb="GET" URI="/api/v1/namespaces/default/pods" latency="7.374935s" userAgent="my-controller/v1.0.0 (linux/amd64)" audit-ID="11111111-2222-3333-4444-555555555555" srcIP="10.0.0.1:12345" apf_pl="workload-low" apf_fs="everybody" apf_iseats=1 apf_fseats=0 apf_additionalLatency="0s" apf_execution_time="7.374765s" resp=200- Key fields:
user_agent(userAgentin raw text) identifies the client binary/operator;verbseparates expensiveLISTor mutatingPATCH/UPDATEcalls from lightweightGETs;apf_iseats(withapf_fseatsandapf_execution_time) measures API Priority and Fairness (APF) concurrency seats occupied (apf_iseats * apf_execution_timereveals top concurrency consumers).
- Key fields:
- Noisy-neighbor attribution & slow/heavy
LISTqueries: Because Prometheus metrics omit caller/user-agent labels to prevent cardinality explosion, always pairapiserver_request_total(grouped byverb,resource, andcodeto quantify QPS/error spikes) with GKE Cloud Audit Logs (which attribute requests to callers out-of-the-box even when optionalAPI_SERVERlogs are off):sum by (verb, resource, code) (rate(apiserver_request_total[5m]))- Query GKE Cloud Audit Logs (
cloudaudit.googleapis.com/activityanddata_access) andkube-apiserverTrace[/httplog.gologs, aggregating byprotoPayload.authenticationInfo.principalEmail,protoPayload.requestMetadata.callerSuppliedUserAgent, and target resource/namespace. - Correlate audit log offender attribution with
apiserver_request_total(grouped byverb,resource, andcode),apiserver_request_duration_seconds(verb="LIST"),apiserver_response_sizes(verb="LIST"), per-resource storage size (apiserver_resource_size_estimate_bytesorkubernetes.io/master/apiserver/resource_size_estimate_bytes, grouped bygroupandresource), andapiserver_watch_list_duration_seconds. - Remediation: Throttle or fix high-QPS client workloads, refactor or
paginate expensive cluster-wide
LISTqueries (e.g.,external-dns,argocd-server, or custom operators listing all pods), useresourceVersion="0"(watch cache reads), or switch to cachedSharedInformerwatches.
- Query GKE Cloud Audit Logs (
- Unindexed CRD custom
fieldSelectorscans & in-flight tail latency: Custom resources are stored as JSON and have no secondary indexes in backend storage or thekube-apiserverwatch cache for customfieldSelectorpaths (even whenspec.versions[].selectableFieldsis declared; onlymetadata.nameandmetadata.namespaceare indexed). EveryLISTwith a customfieldSelector(e.g., across 50,000 CRD instances) forceskube-apiserverto scan and JSON-deserialize every object into memory (unstructured.Unstructured) to evaluate the filter—acquiring multiple API Priority and Fairness (APF) concurrency seats (apf_iseatsinhttplog.go,apiserver_flowcontrol_priority_level_seat_utilization) for the full scan duration (apf_execution_time) and churning Go heap/garbage collection, which queues concurrent requests (apiserver_flowcontrol_request_wait_duration_seconds) and causes in-flight tail-latency regressions (apiserver_request_duration_seconds, visible in opt-inAPI_SERVERTrace[andhttplog.gologs). Replace customfieldSelectorLISTs with indexedlabelSelectors or watch via aSharedIndexInformerwith controller-sidecache.Indexers(IndexField).
3. API Priority & Fairness (APF) Throttling (429 / 504 / 404)
When workloads, batch job scripts, or CI runners fail with HTTP 429 Too Many Requests or 504 Gateway Timeout (context canceled):
- Metrics to query: Query
apiserver_flowcontrol_rejected_requests_total,apiserver_flowcontrol_current_inqueue_requests,apiserver_flowcontrol_request_wait_duration_seconds, andapiserver_flowcontrol_priority_level_seat_utilizationgrouped bypriority_levelandflow_schema, plusapiserver_request_totalby status code (404,429,504) and verb (GET,LIST). - Log attribution (
API_SERVERlogs): Enable control-plane logging (API_SERVERvia--logging=SYSTEM,WORKLOAD,API_SERVER) to pull detailed information onuser_agent(userAgent), target resources (URI/requestURI), and source IPs (src_ip/srcIP/sourceIPs) fromkube-apiserverlogs (or Cloud Audit Logs) for unpaginatedLISTcalls, tight404GETpolling loops, or single-digit millisecond?timeout=requests. - How APF works & remediation: APF throttles noisy clients in
lower-priority lanes (such as
workload-low) to protect critical system traffic (system,leader-election,node-high). Pace client burst QPS, replace tightGET/LISTpolling withSharedInformerwatches, add exponential backoff and client-side caching on404s, increase overly short client request timeouts, or isolate noisy service accounts into a dedicatedFlowSchemaandPriorityLevelConfiguration.
4. Hot-Object 409 Conflicts & Leader-Election RenewDeadline Timeouts
- Hot-object write contention (
409 Conflict& slowPATCH/PUT): When runaway controllers fight over or hot-loop updates against a singleConfigMap,CustomResourceDefinition, orLease, query Cloud Audit Logs (cloudaudit.googleapis.com/activity) grouped byprotoPayload.resourceName,protoPayload.methodName(updateorpatch), andprotoPayload.requestMetadata.callerSuppliedUserAgent(orprotoPayload.authenticationInfo.principalEmail) and inspectkube-apiserverGuaranteedUpdateTrace[logs to pinpoint the hot object and responsible agent. Mitigate by reducing the operator's checkpoint/sync frequency (e.g., increasingrefreshInterval) and lowering client QPS; if pods or microservices write frequent status heartbeats or distributed locks into a single sharedConfigMap(causing serialized optimistic-concurrencyresourceVersioncontention and409 Conflictstorms), migrate per-replica heartbeats or locks to dedicatedcoordination.k8s.ioLeaseobjects. - Leader-election
RenewDeadline / 2(?timeout=5s) timeouts: When third-party operators (e.g., KEDA, VictoriaMetrics) lose Lease leadership withcontext deadline exceededafter 5s despite higher general client timeouts, explain thatclient-go'sleaderelectionpackage sets the per-request HTTP timeout on LeasePUTrenewals toRenewDeadline / 2(so defaultRenewDeadline: 10senforces?timeout=5sonrequestURIin Cloud Audit Logs andTrace[logs). Increase the operator's leader-election flags (e.g.,--leader-elect-lease-duration=60sand--leader-elect-renew-deadline=40s).
5. pods/exec, Konnectivity, Cloud KMS & IAM Webhook Authorizer
- Excessive
pods/execsaturating webhooks & Konnectivity: Whenwarden-validating.common-webhooks.networking.gke.ioadmission timeouts (10s) or500 error dialing backend: dial timeout, backstoperrors occur onkubectl exec, query Cloud Audit Logs forio.k8s.core.v1.pods.exec.get(connect) volume and correlate withapiserver_admission_webhook_admission_duration_secondsforwarden-validating(highpods/execvolume evaluates against admission webhooks and Konnectivity tunnels). Replacekubectl exec-based metrics scraping and health checks with Kubelet/metrics/cadvisor, Cloud Monitoring container metrics (k8s_container), and direct pod-to-pod HTTP/gRPC endpoints. - Konnectivity (
No agent available), firewall, &ip-masq-agent: When webhook calls orkubectl exec/logsfail withNo agent availableorcontext deadline exceeded, checkkube-apiserverlogs (resource.type="k8s_control_plane_component") forNo agent availableor dial failures, inspectkonnectivity-agentpods (kubectl get pods -n kube-system -l k8s-app=konnectivity-agent), verify VPC firewall rules between the control-plane CIDR and node/pod CIDRs (ports10250,8132, and webhook ports), and check theip-masq-agentConfigMap so pod-to-API-server and API-server-to-webhook traffic is not masqueraded or blocked. - Disabled Cloud KMS keys breaking encryption (
kms-provider-0/failed to get dek primitive): Inspectgcloud container clusters describe(databaseEncryptionconfig and error status) and query Cloud Audit Logs forcloudkms.googleapis.com(DisableCryptoKeyVersion,DestroyCryptoKeyVersion, orPermissionDeniedonEncrypt/Decrypt). Re-enable the disabledCryptoKeyVersion, verify the GKE service agent hasroles/cloudkms.cryptoKeyEncrypterDecrypter, and follow https://docs.cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets#reencrypt-secrets to re-encrypt existing Secrets. - In-cluster GCP IAM OAuth tokens saturating the GKE IAM webhook authorizer
(
Failed to make webhook authorizer request): When high-QPS in-cluster clients (Helm, operators) authenticate with GCP IAM OAuth Bearer tokens without matching Kubernetes RBAC bindings,kube-apiserverinvokes the external GKE IAM webhook authorizer on every request, which throttles/times out under load. Inspect Cloud Audit Logs (authenticationInfoandauthorizationInfo) and migrate in-cluster workloads to native KubernetesServiceAccounttokens bound via Kubernetes RBAC (RoleBinding/ClusterRoleBinding) so authorization is evaluated locally in memory.