skilld
All skills
google avatar

/agent-platform-troubleshooting

@efcb6f0
by googlegoogle/skills21k stars
1,698

Troubleshoots Google Cloud Gemini Enterprise Agent Platform issues (Agent Gateway, Registry, Identity, Policies, Model Armor, Identity-Aware Proxy (IAP)). Use when agent requests fail with 403 (especially unauthorized egress), Agent Runtime queries return 500, or gateway/IAP logs show permission errors. Don't use for general Google Cloud Identity and Access Management (IAM) debugging or networking issues unrelated to the Agent Platform stack.

Ask your Agent

Use this Skill: https://skilld.dev/gh/google/skills/agent-platform-troubleshooting

This session only. Nothing lands on disk.

referencesknown-issues.md

≈16k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Best Known Issues (BKIs) and limitations

Concrete failure modes that come up repeatedly with the Agent Platform — symptoms and fixes. Read this when the symptom in front of you matches one of these patterns; it's faster than walking the full diagnostic flow.

Table of Contents

1. ReasoningEngine startup: "Failed to convert project number to project ID"

Symptom:

  • Log name: aiplatform.googleapis.com/reasoning_engine_stderr
  • Error text: google.api_core.exceptions.Unknown: None or Failed to convert project number to project ID.
  • Happens during ReasoningEngine startup, before any user code runs.

Cause: The ReasoningEngine's system identity (the principal set it runs under) lacks resourcemanager.projects.get, which the Agent Runtime SDK needs to resolve the project name during init.

Fix: Grant roles/browser to the ReasoningEngine principal set:

gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --member="principalSet://agents.global.org-${ORG_ID}.system.id.goog/attribute.platformContainer/aiplatform/projects/${PROJECT_NUMBER}" \
  --role="roles/browser"

2. ReasoningEngine startup: "Assembly Service failed to initialize"

Symptom: [1099] ERROR: Assembly Service failed to initialize. in reasoning_engine_stderr.

Cause: Generic "init failed" — usually a downstream symptom of either #1 (project-number resolution) or a runtime error in the agent's set_up() method.

Investigation: Pull all severity=ERROR from reasoning_engine_stderr around the same timestamp; the actual root cause is in the same window.

3. Private-preview limit: one Agent Runtime instance per project bonded to an Agent Gateway

Symptom: Updating an Agent Runtime instance to use an Agent Gateway fails with Internal error encountered or The specified parameters are invalid.

Cause: During the private preview, a project can have only one active ReasoningEngine ↔ AgentGateway bonding. A second bonding attempt fails.

Fix: Either delete the existing bonded RE / AGW, or reuse the existing AGW for any new REs.

4. 403 / "Egress request is not authorized" — missing authz_policy or UAP binding {#egress-not-authorized}

Symptom: Audit logs show GatekeeperAuthorizer.AuthorizeUser returning Permission Denied for iap.webServiceVersions.egressViaIAP or iap.googleapis.com/resources.egressViaIAP.

Cause:

  1. Missing AuthzPolicy Attachment: The IAP authz extension exists, but no authz_policy targets the specific Agent Gateway (target.resources[]).
  2. Missing or Misconfigured UAP Binding (Policy V2): When the gateway uses iapPolicyVersion: "V2", no PolicyBinding binds an AccessPolicy to the target CRM node (//cloudresourcemanager.googleapis.com/projects/${PROJECT_NUMBER}), or the target resource was specified using the string project ID instead of the numeric project number.

Fix:

  1. Verify the authz_policy targets the gateway:

    gcloud network-security authz-policies list --location=${LOCATION}

  2. If using UAP, verify the policy binding targets the numeric project number:

    gcloud iam policy-bindings list \
  --target-resource="//cloudresourcemanager.googleapis.com/projects/${PROJECT_NUMBER}" \
  --location=global

5. IAP ENFORCE mode / Blanket DENY blocks Agent Runtime startup

Symptom: Agent Runtime instances fail to deploy or start, with 403s or connection resets in startup logs (reasoning_engine_stderr).

Cause:

  1. Default-Deny without Bootstrap Endpoints: With IAP in ENFORCE mode on the Agent Gateway, all egress is denied by default — including bootstrap calls the agent runtime makes to telemetry.mtls.googleapis.com, cloudresourcemanager, aiplatform, logging, monitoring, etc.
  2. Blanket Unregistered DENY in UAP: An Org or Folder Admin created an explicit effect: DENY rule on (destination.is_registered == false). Because DENY has absolute precedence and cannot be overridden by project-level rules, Reasoning Engine boot traffic to telemetry.mtls.googleapis.com is dropped immediately.

Fix:

  • Ensure all required monitoring/telemetry endpoints (telemetry.mtls.googleapis.com, monitoring.googleapis.com, trace.mtls.googleapis.com, cloudtrace.googleapis.com) are registered in the Agent Registry and permitted by policy.
  • Never create a blanket DENY on destination.is_registered == false. Rely on gateway default-deny instead.
  • Set IAP to DRY_RUN (iamEnforcementMode: "DRY_RUN") to observe failing destinations without blocking container initialization.

6. Self-signed / Private CA destinations not supported (early versions)

Symptom: Agent fails to connect to internal MCP servers or tools that present self-signed or Private CA-issued certificates.

Cause: In private-preview / early Agent Gateway, the gateway's egress proxy can't validate self-signed cert chains.

Mitigation: Use publicly-trusted CA certificates for internal MCP servers, or wait for the platform enhancement that adds Private CA trust-anchor support.

7. Principal Access Boundary (PAB) policies overriding IAM Allow

Symptom: Agent identity has the right roles bound (roles/iap.egressor, roles/aiplatform.user, etc.) and the resource is registered correctly, but calls still 403.

Cause: A Principal Access Boundary policy is restricting the scope of resources the principal can access. PAB policies take precedence over IAM Allow policies — even a correct binding does nothing if a PAB blocks the target resource.

Investigation: List PAB policies and their bindings for the agent identity:

# List org-wide PAB policies
gcloud iam principal-access-boundary-policies list \
  --organization="${ORGANIZATION_ID}" --location=global

# Find what's bound to a specific principal set / agent identity
gcloud iam policy-bindings search-target-policy-bindings \
  --project="${PROJECT_ID}" --target="${PRINCIPAL_SET}"

# Inspect a specific PAB
gcloud iam principal-access-boundary-policies search-policy-bindings "${PAB_POLICY_ID}" \
  --organization="${ORGANIZATION_ID}" --location=global

Fix: Either widen the PAB to include the destination resource, or remove the PAB binding from the agent's principal set if it shouldn't apply.

8. Gateway proxy denies the request before IAP runs

Symptom: Requests fail with no IAP audit log entry — IAP never saw them.

Cause: The gateway's underlying egress proxy (a Google-managed Secure Web Proxy instance) enforces a deny-by-default posture before IAP authz runs. When the proxy itself denies a call, IAP doesn't get a chance to evaluate, so no IAP audit-log entry is produced for that call.

Specific Case: CONNECT to PSC VIP (240.0.0.2:443) with Default Denied In environments using Private Service Connect (PSC), clients resolve Google APIs to internal VIPs (e.g., 240.0.0.2:443). Consequently, the outer tunnel request is logged as CONNECT 240.0.0.2:443 in the proxy logs. This is expected behavior. Because TLS inspection is enabled on the Secure Web Proxy (SWP), the proxy intercepts this CONNECT tunnel and inspects the TLS handshake to extract the actual SNI hostname (e.g., us-central1-aiplatform.googleapis.com) for routing and policy evaluation. If the extracted SNI hostname is not registered in the Agent Registry, the proxy cannot match the route. The proxy then denies the connection, logging a 403 on the CONNECT request with hostname: 240.0.0.2:443 and matching the default_denied rule.

Investigation: Pull the gateway proxy load-balancer logs:

jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
resource.labels.gateway_type="SECURE_WEB_GATEWAY"

Alternative (CLI Fallback):

gcloud logging read 'jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry" AND resource.labels.gateway_type="SECURE_WEB_GATEWAY"' --project=PROJECT_ID --limit=50

Key fields to inspect in the match:

  • httpRequest.status: Look for 403.
  • jsonPayload.enforcedGatewaySecurityPolicy.hostname: Look for 240.0.0.2:443 (expected PSC CONNECT target).
  • jsonPayload.enforcedGatewaySecurityPolicy.matchedRules: Look for default_denied.
  • jsonPayload.mtls.clientCertChainVerified: Often false because the handshake was interrupted on route match failure.

Fix: Identify the inner hostname the agent was attempting to reach (e.g. us-central1-aiplatform.googleapis.com) and ensure ALL required hostname permutations are registered in the Agent Registry. The proxy relies on these registry entries to dynamically build its routing table.

9. Cross-Project Agent Gateway import / creation failing (Code 13 Internal Error)

Symptom:

  • Happening during gcloud network-services agent-gateways import or creation.
  • CLI rejects the command with a synchronous error: ERROR: (gcloud.network-services.agent-gateways.import) { "code": 13, "message": "an internal error has occurred" }
  • Gateway YAML specifies cross-project infrastructure targets (e.g. a PSC network attachment or target DNS peering network residing inside another network host project, e.g., Project B).

Cause: To satisfy Google Cloud's cross-project resource isolation controls, the API control-plane must validate that the gateway's billing project is authorized to map the target resource. The validation is executed under the Agent Gateway Control Plane Service Agent identity: service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com

If this service agent lacks read permissions on the target network project, or if the network configuration itself is invalid, the validation fails, causing the import command to exit with a synchronous Code 13 internal error.

Specifically, the following network configuration rules must be honored:

  • VPC Network Mismatch: The VPC network containing the PSC network attachment and the target network specified in the DNS peering configuration (dnsPeeringConfig.targetNetwork) must be the exact same VPC network. If they point to different VPC networks, the configuration validation fails.
  • DNS Zone Validation Failure: The domains specified in dnsPeeringConfig.domains must exist as private DNS zones (e.g., private zones or forwarding zones) in the target VPC network. If they do not exist, validation fails.

Investigation:

  1. Verify the caller (user EUC) has roles/compute.networkViewer and roles/dns.reader (or basic Owner/Editor/Viewer roles) on the target network host project.
  2. Check if the network attachment connectionPreference is set to ACCEPT_AUTOMATIC.
  3. Check the IAM policy in the target network project. Verify if the gateway's control-plane service agent is missing the required roles: service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com Note: The egress proxy service agent (service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com) does not require these permissions manually granted on the host project.
  4. Verify that the PSC network attachment's network matches the DNS peering target network exactly.
  5. Check if all domains specified under dnsPeeringConfig.domains correspond to existing private DNS zones associated with the target network VPC (e.g., they align with the DNS forwarding zone, or use . if a catch-all peering zone is configured).

Fix: You must grant the control-plane service agent the required validation permissions inside the target network host project, ensure the target operator has required reader roles, and align the VPC networks and domains in the configuration:

  1. Grant roles/compute.networkUser to validate the PSC network attachment:

    gcloud projects add-iam-policy-binding TARGET_NETWORK_PROJECT_ID \
  --member="serviceAccount:service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com" \
  --role="roles/compute.networkUser"

  2. Grant roles/dns.peer to validate the DNS peering target:

    gcloud projects add-iam-policy-binding TARGET_NETWORK_PROJECT_ID \
  --member="serviceAccount:service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com" \
  --role="roles/dns.peer"

  3. Ensure the operator has roles/compute.networkViewer and roles/dns.reader on the target network project:

    gcloud projects add-iam-policy-binding TARGET_NETWORK_PROJECT_ID \
  --member="user:OPERATOR_EMAIL" \
  --role="roles/compute.networkViewer"

gcloud projects add-iam-policy-binding TARGET_NETWORK_PROJECT_ID \
  --member="user:OPERATOR_EMAIL" \
  --role="roles/dns.reader"

  4. Configure the gateway YAML definition such that the VPC network containing the network attachment match the target network for DNS peering (dnsPeeringConfig.targetNetwork) exactly.

  5. Ensure any domain configured under dnsPeeringConfig.domains is registered as a private DNS zone in the target VPC network (e.g. matching the exact peered zone or using . for root/catch-all peering).

10. Cross-Project Runtime-to-Gateway Binding Failures: PERMISSION_DENIED, INVALID_ARGUMENT, or CAA Token Suppression

Symptom:

  • Symptom A (Deployer PERMISSION_DENIED): Calling Vertex AI client.agent_engines.create(...) or REST API in Project 1 (AE_PROJECT_ID) referencing an Agent Gateway in Project 2 (AGW_PROJECT_ID) fails synchronously with: ERROR: (gcloud.ai.reasoning-engines.create) PERMISSION_DENIED: Permission 'networkservices.agentGateways.use' denied on resource '//networkservices.googleapis.com/projects/AGW_PROJECT_ID/locations/REGION/agentGateways/GATEWAY_NAME'
  • Symptom B (Control-Plane INVALID_ARGUMENT): Deployment fails with: INVALID_ARGUMENT: Agent Gateway project in spec.deployment_spec.agent_gateway_config.agent_to_anywhere_config.agent_gateway must match the Vertex AI project. or region mismatch INVALID_ARGUMENT: Agent Gateway location in spec... must match the Vertex AI location.
  • Symptom C (CAA 401 Unauthenticated Egress): Runtime container deploys successfully, but egress requests to external endpoints or MCP tools fail immediately with: Error Code: "401", Error Details: "Context-Aware Access requirements are not met" or Request had invalid authentication credentials.
  • Symptom D (Proxy 403 PERMISSION_DENIED on MCP Tool / Service Name): Egress through the gateway fails with HTTP 403 PERMISSION_DENIED, even though roles/iap.egressor was granted to the friendly service name in Agent Registry.
  • Symptom E (Unsupported Folder-Level principalSet): Policy binding using principalSet://TRUST_DOMAIN/attribute.platformContainer/aiplatform/folders/FOLDER_NUMBER fails silently or is rejected as an invalid principal identifier.
  • Symptom F (VPC-SC Perimeter Block): Egress calls fail with VPC_SERVICE_CONTROLS_VIOLATION across project boundaries.

Cause:

  1. Cross-Project Feature Rollout & Colocation Invariants: Following the EnableCrossProjectAgentGateway_Default_Launch rollout, cross-project binding (spec.deploymentSpec.agentGatewayConfig.agentToAnywhereConfig.agentGateway) is supported. However, both the runtime and the gateway must reside in the exact same Google Cloud region (e.g. us-central1). Cross-region bindings are rejected with INVALID_ARGUMENT. Furthermore, all runtimes in a given project and region must bind to the identical gateway instance.
  2. Missing Deployer and Service Agent IAM Permissions:
    • The deploying principal (user EUC, Cloud Build SA, or pipeline identity) must have networkservices.agentGateways.use and networkservices.agentGateways.get on the gateway in AGW_PROJECT_ID.
    • The runtime project's Vertex AI Service Agent (service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com) must hold roles/networkservices.viewer (or custom role ae_agw_cross_project_sa with networkservices.agentGateways.get and networkservices.operations.get) on AGW_PROJECT_ID.
  3. Context-Aware Access (CAA) Token Suppression: By default, Google Cloud SDKs and google-auth enforce bound token protection. In cross-project egress, bound token requirements reject requests unless the runtime explicitly sets the environment variable GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES: False.
  4. Friendly Service Name vs. Server-Generated Resource ID Trap: In Agent Registry, services, endpoints, and mcp-servers are written via gcloud agent-registry services create. However, IAP authorization (gcloud iap web add-iam-policy-binding) must bind to the server-generated resource ID (agentregistry-00000000-..., extracted from the registryResource field), NOT the friendly service name ([MCP_SERVICE_NAME]). Binding to the friendly service name attaches the policy to a non-existent IAP resource.
  5. SPIFFE principalSet Syntax Limitation: attribute.platformContainer/aiplatform/folders/FOLDER_NUMBER is not an indexed attribute in IAM SPIFFE URI evaluation. Scoping must strictly use project-level syntax (attribute.platformContainer/aiplatform/projects/PROJECT_NUMBER) or trust-domain-wide syntax (principalSet://TRUST_DOMAIN/*), or use Principal Access Boundary (PAB) policies.
  6. VPC Service Controls Perimeter Boundary: If Project 1 (AE_PROJECT_ID) and Project 2 (AGW_PROJECT_ID) reside in different perimeters, perimeter bridging or bidirectional ingress/egress rules covering aiplatform.googleapis.com, networkservices.googleapis.com, and agentregistry.googleapis.com are required.

Investigation:

  1. Check Deployer Permissions on Gateway:

    gcloud network-services agent-gateways describe GATEWAY_NAME \
  --project=AGW_PROJECT_ID \
  --location=REGION

    If this fails with PERMISSION_DENIED, the deployer lacks read/use permissions.

  2. Check Vertex AI Service Agent IAM on Gateway Project:

    gcloud projects get-iam-policy AGW_PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" \
  --format="table(bindings.role)"

    Verify that roles/networkservices.viewer or custom role ae_agw_cross_project_sa is present.

  3. Verify Runtime Region vs. Gateway Region: Ensure both the Reasoning Engine runtime and the Agent Gateway are deployed in the same region.

  4. Inspect Runtime CAA Environment Variable: Inspect the Reasoning Engine deployment spec or Python definition. Verify that env_vars contains "GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES": False.

  5. Verify IAP Binding Target Resource ID:

    # Fetch the server-generated registryResource identifier
gcloud agent-registry services describe MCP_SERVICE_NAME \
  --project=AGW_PROJECT_ID \
  --location=REGION \
  --format='value(registryResource)'

    Confirm that gcloud iap web add-iam-policy-binding used the trailing UUID (agentregistry-00000000-...) as --mcp-server or --agent, not the friendly name.

  6. Check VPC-SC Status: Confirm whether AE_PROJECT_ID and AGW_PROJECT_ID reside in the same VPC-SC perimeter or have perimeter ingress/egress rules allowing communication.

Fix:

  1. Grant Deployer Permissions: Grant roles/networkservices.viewer (or permissions networkservices.agentGateways.use and networkservices.agentGateways.get) to the deploying identity on AGW_PROJECT_ID:

    gcloud projects add-iam-policy-binding AGW_PROJECT_ID \
  --member="user:DEPLOYER_EMAIL" \
  --role="roles/networkservices.viewer"

  2. Grant Vertex AI Service Agent Permissions: Grant the Agent Runtime's service agent access to the governance project: Option A (Standard Viewer Role):

    gcloud projects add-iam-policy-binding AGW_PROJECT_ID \
  --member="serviceAccount:service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" \
  --role="roles/networkservices.viewer"

    Option B (Least-Privilege Custom Role):

    gcloud iam roles create ae_agw_cross_project_sa \
  --project="AGW_PROJECT_ID" \
  --title="AE AGW Cross Project SA" \
  --description="Custom role for cross-project service agent to access Agent Gateways and Operations" \
  --permissions="networkservices.agentGateways.get,networkservices.operations.get" \
  --stage="GA"

gcloud projects add-iam-policy-binding AGW_PROJECT_ID \
  --member="serviceAccount:service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" \
  --role="projects/AGW_PROJECT_ID/roles/ae_agw_cross_project_sa"

  3. Configure Runtime with CAA Opt-Out: In the agent runtime deployment configuration:

    remote_agent = client.agent_engines.create(
    agent=local_agent,
    config={
        "agent_gateway_config": {
            "agent_to_anywhere_config": {
                "agent_gateway": (
                    "projects/AGW_PROJECT_ID/locations/REGION/agentGateways/AGENT_GATEWAY_TO_ANYWHERE_NAME"
                )
            },
        },
        "identity_type": types.IdentityType.AGENT_IDENTITY,
        "env_vars": {
            "GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES": (
                False
            ),
        },
    },
)

  4. Bind IAP Role to the Server-Generated Resource ID:

    REGISTRY_RESOURCE=$(gcloud agent-registry services describe MCP_SERVICE_NAME \
  --project=AGW_PROJECT_ID \
  --location=REGION \
  --format='value(registryResource)')
RESOURCE_ID=$(basename "$REGISTRY_RESOURCE")

AGENT_PRINCIPAL="principal://agents.global.org-ORG_ID.system.id.goog/resources/aiplatform/projects/AE_PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_ID"

gcloud iap web add-iam-policy-binding \
  --project=AGW_PROJECT_ID \
  --region=REGION \
  --resource-type=agent-registry \
  --mcp-server="${RESOURCE_ID}" \
  --member="$AGENT_PRINCIPAL" \
  --role="roles/iap.egressor"

  5. Use Valid principalSet Syntax: For baseline Google APIs access, avoid folder-level scoping. Use project-level scoping:

    principalSet://agents.global.org-ORG_ID.system.id.goog/attribute.platformContainer/aiplatform/projects/AE_PROJECT_NUMBER

    or trust-domain-wide scoping:

    principalSet://agents.global.org-ORG_ID.system.id.goog/*

11. Reasoning Engine Container Crash: Python Dependency Mismatch / Startup Exceptions

Symptom:

  • Log name: aiplatform.googleapis.com/reasoning_engine_stderr
  • Error text: ModuleNotFoundError: No module named '...' or ImportError: cannot import name '...' or Python tracebacks.
  • The Reasoning Engine fails to deploy or dies immediately after creation.

Cause: The Python code uploaded for the Reasoning Engine depends on external packages that were not specified or incorrectly specified during deployment.

Fix:

  1. Identify the missing or mismatched package from the traceback in reasoning_engine_stderr.
  2. Update your requirements.txt file or the requirements parameter in your deployment script (e.g. deploy_agent.py) to include the correct package and version constraint.
  3. Re-deploy the Reasoning Engine.

12. mTLS Egress Failure: Invalid or Expired Client Certificate

Symptom:

  • Gateway proxy load-balancer logs show connection drops or 403 errors.
  • Logs contain jsonPayload.mtls.clientCertChainVerified: false or handshake failures.
  • The agent fails to reach the destination MCP server when using mTLS.

Cause: The client certificate used by the Agent Gateway is expired, not yet valid, or not trusted by the destination server (CA mismatch).

Fix:

  1. Verify the certificate validity dates and CA chain.
  2. If expired or invalid, upload a new certificate to Certificate Manager and update the corresponding ClientTlsPolicy resource.
  3. Ensure the destination server trusts the CA that signed the gateway's client certificate.

13. Model Armor integration on Agent Gateway fails with 500 / PERMISSION_DENIED

Symptom:

  • Calling remote_agent.async_stream_query returns 500 Internal Server Error.
  • Gateway Request Logs show grpcStatus: "PERMISSION_DENIED" for the Model Armor backend and authzPolicyInfo.result: DENIED.

How to Identify: Search in Cloud Logging in the consumer project using the following query:

resource.type="networkservices.googleapis.com/Gateway"
jsonPayload.serviceExtensionInfo.backendTargetName : "modelarmor"
jsonPayload.serviceExtensionInfo.backendTargetType="BACKEND_SERVICE"
jsonPayload.serviceExtensionInfo.grpcStatus="PERMISSION_DENIED"

Cause: The gateway proxy (Secure Web Proxy) executing the inline Model Armor scan lacks the necessary permissions in the consumer project to call the Model Armor service. The callout runs under the Consumer's Service Extensions Service Agent (gcp-sa-dep).

[!WARNING] Granting the legacy basic roles/owner or roles/editor roles to the Service Extensions Service Agent is insufficient. Granular data-plane permissions must be explicitly granted via specialized roles.

The required roles are:

  1. roles/modelarmor.user: Required to read and apply the template.
  2. roles/modelarmor.calloutUser: Required to initiate the egress gRPC callout.
  3. roles/serviceusage.serviceUsageConsumer: Required to consume quota and bill it.

Fix: Grant all three required roles to the consumer-side delegated Service Extensions Service Agent (gcp-sa-dep) in the consumer project.

gcloud projects add-iam-policy-binding CONSUMER_PROJECT_ID \
    --member="serviceAccount:service-CONSUMER_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com" \
    --role="roles/modelarmor.user"

gcloud projects add-iam-policy-binding CONSUMER_PROJECT_ID \
    --member="serviceAccount:service-CONSUMER_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com" \
    --role="roles/modelarmor.calloutUser"

gcloud projects add-iam-policy-binding CONSUMER_PROJECT_ID \
    --member="serviceAccount:service-CONSUMER_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com" \
    --role="roles/serviceusage.serviceUsageConsumer"

15. Egress Blocked: Empty Agent Registry IAM Policy (403 Forbidden)

Symptom: All egress attempts through the gateway fail with a 403 Permission Denied (logged as Egress request is not authorized).

Cause: The IAM policy on the agent-registry resource in the consumer project is empty or lacks bindings for the agent. The agent's service account must be explicitly granted the roles/iap.egressor role.

Fix: Grant the roles/iap.egressor role to the agent's service account at the registry level.

  1. Create a policy.json file:

    {
  "bindings": [
    {
      "role": "roles/iap.egressor",
      "members": [
        "serviceAccount:YOUR_AGENT_SERVICE_ACCOUNT_EMAIL"
      ]
    }
  ]
}

  2. Apply the policy to the Agent Registry:

    gcloud iap web set-iam-policy policy.json \
    --resource-type=agent-registry \
    --project=YOUR_PROJECT_ID

16. Reasoning Engine deployment fails with Code 13 (Internal Error)

Symptom:

  • Agent Runtime deployment fails with generic Code 13: Failed to deploy to Agent Platform: Failed to update Agent Runtime: {'code': 13, 'message': 'Please refer to our troubleshooting pages...'}.
  • The instance is immediately cleaned up.

Root Cause A: Route Conflict in Shared Tenant Project (Wildcard Route Collision)

  • Underlying Cause: Agent Runtime allocates a single, shared tenant project per customer project and region to host Reasoning Engines. If a customer already has a Reasoning Engine bonded to one Agent Gateway (gateway-A), the tenant project contains wildcard routes pointing to gateway-A. Trying to deploy another Reasoning Engine bonded to gateway-B in the same region fails due to route conflict.

  • Fix: A project/location can only support one active Agent Gateway bonding for Reasoning Engines.

    1. Delete any active Reasoning Engines in the region bonded to the old gateway.

      curl -X DELETE -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/reasoningEngines/${REASONING_ENGINE_ID}?force=true"

    2. If the old Reasoning Engines are already deleted but routes are stuck, delete the old gateway resource to force cleanup:

      gcloud network-services agent-gateways delete OLD_GATEWAY_NAME --location=LOCATION --project=PROJECT_ID

    3. Retry deployment bonded to the new gateway.

Root Cause B: Container Startup Timeout or Initialization Failure

  • Underlying Cause: The container fails to complete initialization within the startup timeout limit (due to syntax errors, missing dependencies, slow network calls, or permission denials).
  • Fix:
    1. Verify the container can start locally.
    2. Check Cloud Logging at the time of deployment for Python tracebacks.
    3. De-risk initialization code by lazy loading heavy dependencies.

17. Reasoning Engine outbound SSL handshake failure (CERTIFICATE_VERIFY_FAILED) with Egress Agent Gateway

Symptom:

  • Log show: CERTIFICATE_VERIFY_FAILED: self signed certificate in certificate chain
  • Occurs when a Reasoning Engine makes outbound calls while bound to an Egress Agent Gateway with AGENT_IDENTITY enabled.

Cause: When outgoing connections pass through Secure Web Proxy (SWG) on the Agent Gateway, SWG performs TLS decryption and inspection using a dynamic certificate signed by its custom root CA. If the Reasoning Engine trust store does not contain these root certificates, outbound handshakes fail.

For custom container or prebuilt image deployments, the automated trust certificate injection is skipped, so the gateway's root certificates are not baked into the container.

Fix / Workaround:

  • For Source/Config-based Deployments (Agent Runtime SDK): Create the Agent Gateway first and wait until provisioning completes. Verify root certificates are ready:

    gcloud network-services agent-gateways describe GATEWAY_NAME \
    --location=LOCATION \
    --format="value(agentGatewayCard.rootCertificates)"

    Once this returns PEM certificates, deploy/update the Reasoning Engine. The pipeline will automatically bake them in.

  • For Custom Container / Prebuilt Image Deployments: Export the gateway's root certificates using the gcloud network-services gateways export-cacerts command and manually copy/bake them into your Dockerfile:

    COPY agw_root.crt /usr/local/share/ca-certificates/agw-gateway.crt
RUN chmod 644 /usr/local/share/ca-certificates/agw-gateway.crt && update-ca-certificates
ENV GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

18. VPC Service Controls (VPC-SC) Perimeter Blocks Agent Gateway Provisioning

[!NOTE] Native VPC-SC Creation Support: As of September 8, 2026, Agent Gateway creation inside a VPC-SC perimeter works natively out of the box without requiring manual control plane ingress policies, provided that the Agent Connectivity Template (ACT) specifies vpcEgress: ALL_TRAFFIC. The requirement for manual ingress policies is no longer needed for standard provisioning.

Symptom:

  • Deployment of Agent Gateway fails or runtime egress is blocked when the project is inside a VPC-SC perimeter.
  • Violation logs show NETWORK_NOT_IN_SAME_SERVICE_PERIMETER or similar VPC-SC blocks for calls between the consumer project and Google-managed infrastructure.

Cause:

  1. ACT Egress Mode Misconfiguration (Primary Cause): The Agent Connectivity Template is configured with vpcEgress: PRIVATE_RANGES_ONLY or the gateway is deployed in Standalone mode (without an ACT). In these modes, traffic bypasses the customer's VPC perimeter boundaries, causing VPC-SC to reject the deployment or egress calls.
  2. Legacy or Strict Custom Perimeters (Historical): Prior to September 8, 2026, control plane provisioning workflows required explicit ingress policies for Google-managed service agents. In legacy or custom perimeters that block Google-managed actuation identities, explicit ingress policies may still be required.

Fix:

  1. Verify ACT specifies ALL_TRAFFIC: Ensure the Agent Connectivity Template uses vpcEgress: ALL_TRAFFIC and binds to a valid PSC-I Network Attachment in the consumer VPC with Cloud NAT configured on the subnet.
  2. Configure Ingress Policies (Legacy / Restrictive Perimeters Fallback): If operating in a strict perimeter where Google-managed actuation identities are blocked, configure ingress policies in your VPC-SC perimeter to allow the actuation identities. Two ingress rules are required:

Rule 1: Gateway Control Plane Actuation

Allows the control plane service agent to manage gateway resources.

  • From:
    • Identities: serviceAccount:actuation-a@networkservices-prod.iam.gserviceaccount.com
    • Sources: Access Level *
  • To:
    • Operations:
      • Service: compute.googleapis.com
        • Methods:
          • FirewallsService.Delete, FirewallsService.Get
          • GlobalOperationsService.Get
          • HealthChecksService.Delete, HealthChecksService.Get
          • InstanceTemplatesService.Delete, InstanceTemplatesService.Get
          • NetworksService.Delete, NetworksService.Get
          • RegionAddressesService.Delete
          • RegionBackendServicesService.Delete, RegionBackendServicesService.Get
          • RegionForwardingRulesService.Delete, RegionForwardingRulesService.Get
          • RegionInstanceGroupManagersService.Delete, RegionInstanceGroupManagersService.Get
          • RegionOperationsService.Get
          • RegionRoutersService.Delete, RegionRoutersService.Get
          • RoutesService.Delete, RoutesService.Get
          • ServiceAttachmentsService.Delete, ServiceAttachmentsService.Get
          • SubnetworksService.Delete, SubnetworksService.Get, SubnetworksService.List
          • ZonesService.List
      • Service: cloudresourcemanager.googleapis.com
        • Methods: Projects.SetIamPolicy
      • Service: networkservices.googleapis.com (Methods: *)
      • Service: networksecurity.googleapis.com (Methods: *)
      • Service: serviceusage.googleapis.com (Methods: *)
      • Service: certificatemanager.googleapis.com (Methods: *)
      • Service: privateca.googleapis.com (Methods: *)
    • Resources: projects/CUSTOMER_PROJECT_NUMBER

Rule 2: Agent Runtime

Allows the Agent Runtime pipeline agent to access security resources.

  • From:
    • Identities: serviceAccount:cloud-aiplatform-pipeline-robot-prod@system.gserviceaccount.com
    • Sources: Access Level *
  • To:
    • Operations:
      • Service: networksecurity.googleapis.com (Methods: *)
    • Resources: projects/CUSTOMER_PROJECT_NUMBER

Replace CUSTOMER_PROJECT_NUMBER with your actual project number.

19. PSC Subnet Exhaustion Fails Agent Gateway Deployment

Symptom:

  • Agent Gateway deployment fails or gets stuck during provisioning.
  • Resource allocation errors related to IP addresses in GCE logs.

Cause: The Private Service Connect (PSC) Network Attachment requires free IP addresses in its allocated subnet to establish connections. If the subnet (especially if it is small, like /28) runs out of free IP addresses, the gateway cannot allocate the necessary interfaces and deployment fails.

Fix: Verify and expand the PSC subnet.

  1. Find the Agent Gateway Name (if not provided): List the gateways in the project:

    gcloud network-services agent-gateways list --location=$LOCATION --project=$PROJECT_ID

  2. Identify the subnetwork used by the gateway's Network Attachment: Describe the gateway:

    gcloud network-services agent-gateways describe AGENT_GATEWAY_NAME --location=$LOCATION --project=$PROJECT_ID

    Look for networkConfig.egress.networkAttachment or networkConfig.ingress.networkAttachment.

  3. Describe the Network Attachment to find the subnet name:

    gcloud compute network-attachments describe <ATTACHMENT_NAME> --region=$LOCATION --project=$PROJECT_ID

  4. Describe the subnetwork to check the CIDR range:

    gcloud compute networks subnetworks describe <SUBNET_NAME> --region=$LOCATION --project=$PROJECT_ID

  5. If the subnet is small (e.g. /28 or /29), it may be exhausted. Allocate a larger subnet (minimum /26 recommended) or expand the existing one, and recreate the network attachment and gateway if necessary.

20. Config Validation Failure: Missing Agent Registry API enablement

Symptom: During gcloud network-services agent-gateways import, the command fails with:

ERROR: (gcloud.alpha.network-services.agent-gateways.import) {
  "code": 3,
  "details": [
    {
      "@type": "type.googleapis.com/google.rpc.BadRequest",
      "fieldViolations": [
        {
          "description": "most likely missing permissions in project <PROJECT_NUMBER>, location <LOCATION>: agentregistry.agents.list, agentregistry.endpoints.list, and agentregistry.mcpServers.list",
          "field": "resource.registries"
        }
      ]
    }
  ],
  "message": "Config validation failed"
}

Cause: The Agent Registry API (agentregistry.googleapis.com) is not enabled in the project. The error message misleadingly suggests a permission issue.

Fix: Enable the Agent Registry API in the project:

gcloud services enable agentregistry.googleapis.com --project=YOUR_PROJECT_ID

21. Agent Identity Egress to Cloud Run / Functions fails with 401/403 (JWT Auth issue)

Symptom: The agent is registered in the Agent Registry with an Agent Gateway. When attempting to call an MCP server or tool hosted on Cloud Run or Cloud Functions, the request fails with an egress error or HTTP 401 Unauthorized / 403 Forbidden.

Cause: Direct Agent Identity (using principalSet) to Cloud Run OIDC authentication is not supported natively. Cloud Run requires a standard OIDC ID token, which the default agent identity cannot obtain directly.

Fix / Workaround: The agent must use Service Account impersonation to obtain an OIDC ID token to authenticate with Cloud Run.

  1. Configure IAM Permissions:

    • The Agent Identity (principalSet://...) must have the roles/iam.serviceAccountTokenCreator role on a target Service Account (e.g. mcp-invoker-sa).
    • The target Service Account must have roles/run.invoker on the Cloud Run service.

  2. Enable Service Account Token Creator API: Ensure iamcredentials.googleapis.com is enabled in the project.

  3. Impersonate in Code: Use the Google Auth SDK to impersonate the Service Account and generate ID tokens for the Cloud Run audience. Example Python (ADK):

    from google.auth import impersonated_credentials
import google.auth

source_creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
impersonated = impersonated_credentials.Credentials(
    source_credentials=source_creds,
    target_principal="YOUR_TARGET_SA_EMAIL",
    target_scopes=["https://www.googleapis.com/auth/cloud-platform"],
)
id_token_creds = impersonated_credentials.IDTokenCredentials(
    target_credentials=impersonated,
    target_audience="YOUR_CLOUD_RUN_URL_ORIGIN",
    include_email=True,
)
# Use id_token_creds to authenticate requests to Cloud Run.

22. Agent deployed but not visible in Registry (Boundary issue)

Symptom: You have successfully deployed an agent (e.g., via Agent Runtime SDK), but when you list agents from your designated Management Project using gcloud agent-registry agents list, the agent is missing.

Cause: The Agent Registry uses a "Management Boundary" to define which projects' agents are governed and visible. If the project where the agent was deployed is not included in the Management Boundary of the Management Project, it will not be discoverable.

Fix: Update the Agent Management Boundary of your Management Project to include the project or folder where the agent is hosted.

  1. Verify the current boundary settings (if supported) or update the boundary to include the target CRM node (project or folder):

    gcloud agent-registry boundary update --crmNode=projects/YOUR_AGENT_HOST_PROJECT_ID

    Or if you want to manage a whole folder:

    gcloud agent-registry boundary update --crmNode=folders/YOUR_FOLDER_ID

23. Telemetry Egress Blocked by Egress Agent Gateway (Startup Failure)

Symptom: Reasoning Engine deployment fails to start up (container crashes during set_up()). Logs show connection reset or handshake errors to telemetry.mtls.googleapis.com.

Cause: Your project is using an Egress Agent Gateway (default-deny). The agent runtime connects to Google telemetry services during startup. Because these endpoints are not registered in the Agent Registry and allowed via policy, the gateway blocks the connection, causing the startup probe to fail.

Fix: Register the required telemetry endpoints in the Agent Registry and allow access via policy.

  1. Verify & Register Telemetry and Tracing Endpoints: You MUST explicitly check and verify whether all required monitoring and tracing endpoints (telemetry.mtls.googleapis.com, monitoring.googleapis.com, trace.mtls.googleapis.com, cloudtrace.googleapis.com) are registered as Endpoints in the Agent Registry:

    gcloud agent-registry endpoints list --location=us-central1 --project=$PROJECT_ID

    If any of them are missing, create endpoint entries for them (e.g. telemetry.mtls.googleapis.com, monitoring.googleapis.com, trace.mtls.googleapis.com, cloudtrace.googleapis.com):

    gcloud agent-registry endpoints create google-telemetry \
    --fqdn=telemetry.mtls.googleapis.com \
    --location=us-central1

  2. Authorize the Egress: Ensure your AuthorizationPolicy is correctly bound to the Gateway and allows the agent's principal set to egress to these endpoints.

24. UAP Org Policy Blocker (constraints/iam.managed.disableAccessPolicyBinding)

Symptom: Creating an IAM Policy Binding (gcloud iam policy-bindings create or Terraform google_iam_policy_binding) fails with:

ERROR: (gcloud.beta.iam.policy-bindings.create) CUSTOM_ORG_POLICY_VIOLATION: Operation violates organization policy constraint 'constraints/iam.managed.disableAccessPolicyBinding'.

Cause: Enterprise Google Cloud organizations often enforce constraints/iam.managed.disableAccessPolicyBinding by default to prevent unauthorized IAM v3 bindings.

Fix: An Organization or Folder Administrator must apply an Org Policy V2 override setting enforce: false at the target resource level:

gcloud org-policies set-policy --project=${PROJECT_ID} override.yaml

Where override.yaml defines:

name: projects/PROJECT_ID/policies/iam.managed.disableAccessPolicyBinding
spec:
  rules:
    - enforce: false

Allow 30–60 seconds for IAM policy control plane propagation before retrying.

25. UAP Policy Binding Destruction Race Condition (400 FAILED_PRECONDITION)

Symptom: During automated teardown or terraform destroy, destroying google_iam_access_policy fails with:

HTTP 400 FAILED_PRECONDITION: Access policy is referenced by active policy binding(s).

Cause: Deleting a PolicyBinding is subject to a 10–30 second replication delay in the IAM control plane. If an orchestrator deletes the binding and immediately issues a deletion for the parent AccessPolicy, IAM rejects the call because the binding is still visible in eventual consistency.

Fix: Ensure teardown workflows poll GET on the PolicyBinding resource URL until an HTTP 404 is confirmed before deleting the referenced AccessPolicy. In Terraform scripts or destroy provisioners, never exit 0 on timeout.

26. Agent Gateway Multi-Registry Validation Error (Missing Global Registry)

Symptom: gcloud network-services agent-gateways import or Terraform apply fails with:

Invalid argument: maximum of two registries are supported. When it's two registries, one of them must be global, the other one can be either regional or multi-regional.

Cause: The Agent Gateway was configured with two registries, but neither was global (e.g. two regional registries like us-central1 and us-west1), or both were global.

Fix: In multi-registry mode, ensure exactly one registry is global, and the second registry is regional or multi-regional:

registries:
  - //agentregistry.googleapis.com/projects/PROJECT_NUMBER/locations/global
  - //agentregistry.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1

27. CEL Null Dereference Crash on Non-Tool MCP Methods

Symptom: Gatekeeper returns HTTP 500 or PERMISSION_DENIED with internal evaluation error logs when an agent invokes MCP lifecycle or non-tool methods (prompts/list, resources/read).

Cause: When invoking non-tool MCP operations, the tool message inside mcp_server is null. A CEL rule that dereferences destination.agent_registry.mcp_server.tool.annotations.* or tool.name without first checking the method throws an unhandled null-dereference exception in the CEL evaluator.

Fix: Guard all tool attribute dereferences with a method check:

(destination.agent_registry.mcp_server.method == 'tools') && (destination.agent_registry.mcp_server.tool.annotations.read_only_hint == true)

28. ACT ALL_TRAFFIC Silent Outbound Hang (Missing Cloud NAT)

Symptom: When an Agent Gateway uses an Agent Connectivity Template with vpcEgress: ALL_TRAFFIC, Reasoning Engine calls to public external APIs (api.weather.gov, api.ipify.org, SaaS tools) hang and eventually fail with connection timeouts. Egress to Google APIs continues to function normally.

Cause: In ALL_TRAFFIC mode, all non-Google outbound packets exit the PSC-I Network Attachment into the consumer VPC subnet (agw-psci-subnet). If that subnet does not have an active Cloud NAT gateway, public traffic is dropped. (Google APIs bypass Cloud NAT via Andromeda Private Google Access).

Fix: Configure Cloud Router and Cloud NAT on the consumer VPC covering the PSC-I subnet:

gcloud compute routers nats create agw-cloud-nat \
    --router=agw-nat-router \
    --region=LOCATION \
    --nat-custom-subnet-ip-ranges=agw-psci-subnet \
    --auto-allocate-nat-external-ips

29. CCFE Agent Gateway Mutation Concurrency Lock (HTTP 409 ABORTED)

Symptom: Submitting a PATCH, update, or import command against an Agent Gateway returns:

HTTP 409 ABORTED: unable to queue the operation

Cause: Cloud Control Forwarding Engine (CCFE) enforces strict mutual exclusion per Agent Gateway resource. Concurrent mutations are aborted while an asynchronous update operation is already active.

Fix: Wait ~1.5 to 2 minutes for the in-flight Long-Running Operation (LRO) to reach done: true before initiating another mutation.

30. Dangling Reference on ACT Deletion (FROM_AGENT_GATEWAY)

Symptom: Attempting to delete an Agent Connectivity Template fails with:

HTTP 400 FAILED_PRECONDITION: Resource projects/.../agentConnectivityTemplates/... is already being used by resource(s) projects/.../agentGateways/...

even after the referenced Agent Gateway has been deleted.

Cause: Network Services retained an internal cross-reference protection (FROM_AGENT_GATEWAY<ID>) after gateway deletion.

Fix: Delete the dangling reference via Stubby:

stubby call blade:network-services-prod-${REGION} google.internal.cloud.reference.References.DeleteReference \
    "name: 'projects/${PROJECT_NUMBER}/locations/${REGION}/agentConnectivityTemplates/${ACT_NAME}/references/FROM_AGENT_GATEWAY_${GATEWAY_ID}'"

31. Reasoning Engine Deletion Preconditions (Active Sessions Require ?force=true)

Symptom: Deleting a Vertex AI Reasoning Engine instance fails with:

HTTP 400 FAILED_PRECONDITION: Cannot delete ReasoningEngine with active sessions.

Cause: Active agent sessions or conversations remain attached to the engine runtime.

Fix: Append ?force=true to the delete API call or use the force flag in the CLI/REST client to cascade session cleanup.

32. REST API Path Matching Gotcha (.json Extensions and Query Strings)

Symptom: An agent calling an external REST API (e.g. Zendesk) receives HTTP 403 Egress request is not authorized, even though a UAP rule allows /api/v2/tickets.

Cause: SDK clients typically append .json (e.g. /api/v2/tickets.json) or query strings (?page=1). Strict equality checks reject these variations, while naive .startsWith('/api/v2/tickets') introduces security escapes to /api/v2/tickets_delete_all.

Fix: Construct explicit boundary-aware CEL path conditions:

((destination.unregistered.path == '/api/v2/tickets') ||
 (destination.unregistered.path == '/api/v2/tickets.json') ||
 (destination.unregistered.path.startsWith('/api/v2/tickets/')) ||
 (destination.unregistered.path.startsWith('/api/v2/tickets?')) ||
 (destination.unregistered.path.startsWith('/api/v2/tickets.json?')))

33. Secure Web Proxy (SWP) & Policy-Based Routing (PBR) Egress Failures

Symptom: When attempting to route egress traffic from an Agent Gateway Network Attachment through a downstream Secure Web Proxy (SWP) for Layer 7 URL filtering, packets drop silently, connections time out, or requests fail with HTTP 403 Forbidden, HTTP 503 Service Unavailable: remote connection failure, or INVALID_ARGUMENT.

Causes & Platform Constraints:

  1. PSC Endpoints Cannot Be Route Next-Hops: Google Cloud route tables (both static routes and Policy-Based Routes) do not allow Private Service Connect (PSC) forwarding rules (--load-balancing-scheme="") or service attachments as next hops.
  2. SWP via PSC Service Attachment Lacks Next-Hop Mode: Publishing Secure Web Proxy behind a PSC Service Attachment operates strictly in Explicit Proxy Mode (HTTP CONNECT). Packets emerging from an Agent Gateway Network Attachment are Layer 3/Layer 4 transparent IP packets; they cannot reach an explicit proxy without client-side proxy configuration (which Vertex AI Reasoning Engine containers do not support).
  3. Static Route Tag Defect: Next-hop routing mode with static routes requires VM network tags (--tags). Network Attachments (compute network-attachments) are consumer network interfaces, not VM instances, and cannot bear network tags. Static routes cannot match or steer traffic originating from an Agent Gateway Network Attachment.
  4. SWP Gateway Resource Definition Syntax Error: Creating an SWP gateway with type: SECURE_WEB_PROXY fails validation. The Cloud Control Forwarding Engine API requires type: SECURE_WEB_GATEWAY and routingMode: NEXT_HOP_ROUTING_MODE.
  5. Cloud NAT Missing ENDPOINT_TYPE_SWG: SWP Envoy proxy instances reside on the private proxy-only subnet and lack external IPs. Outbound internet egress requires Cloud NAT configured with --endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG. If ENDPOINT_TYPE_SWG is omitted, proxy outbound traffic hangs silently.
  6. ADK Streaming Session Allowlist Omission: In ALL_TRAFFIC ACT mode, Agent Development Kit (ADK) streaming queries initialize sessions via POST https://<region>-aiplatform.mtls.googleapis.com/.../sessions. Because ALL_TRAFFIC forces all runtime egress to the Network Attachment, PBR steers this session call to SWP. If <region>-aiplatform.mtls.googleapis.com is omitted from the GatewaySecurityPolicy allowlist, SWP drops it (default_denied), causing ADK streaming queries to fail with HTTP 503 (remote connection failure), while direct query() calls succeed.

Fix:

  1. Deploy Next-Hop SWP directly in the consumer VPC: Create an Envoy proxy-only subnet (purpose: REGIONAL_MANAGED_PROXY, role: ACTIVE, minimum /26) and deploy SWP listening on an internal IP on the PSC-I subnet (e.g. 10.20.1.250) with type: SECURE_WEB_GATEWAY and routingMode: NEXT_HOP_ROUTING_MODE.

  2. Configure Policy-Based Routing (PBR): Create a PBR matching source CIDR 10.20.1.0/24 (the Network Attachment subnet) and destination 0.0.0.0/0 with --next-hop-ilb-ip=10.20.1.250 (priority 200). Add a default fallback PBR with priority 1000:

    gcloud network-connectivity policy-based-routes create agw-psci-to-swp-pbr \
  --network=projects/PROJECT_ID/global/networks/VPC_NAME \
  --source-range=10.20.1.0/24 \
  --destination-range=0.0.0.0/0 \
  --next-hop-ilb-ip=10.20.1.250 \
  --priority=200 \
  --project=PROJECT_ID

  3. Configure Cloud NAT with ENDPOINT_TYPE_SWG: Update the Cloud NAT on the Cloud Router to include --endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG.

  4. Allowlist External Domains and Regional AI Platform in GatewaySecurityPolicy: Ensure the CEL policy rule permits both external target domains and the regional AI platform session endpoint: inUrlList(host(), ['api.weather.gov', 'api.ipify.org', '<region>-aiplatform.mtls.googleapis.com']).

Source: SKILL.md on GitHub

Third-party checks

No alerts8d3 checks · Risk SAFE
  • Gen Agent Trust Hub8d

    This skill is a diagnostic tool designed to troubleshoot the Google Cloud Gemini Enterprise Agent Platform. It provides structured procedures and reference documentation for analyzing connectivity, identity, and policy issues within a Google Cloud environment. No security issues were detected during the analysis.

  • Socket8d

    No alerts

  • Snyk8d

    Risk: LOW · No issues

Provenance

Signed by skilld at efcb6f0. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last week

Capability

metadata
{
  "version": "1.0.0",
  "category": "AiAndMachineLearning"
}

README badge

README badge for google/skills/agent-platform-troubleshooting