skilld
All skills
aws avatar

/connecting-to-data-source

@b33847d
by Amazon Web Servicesaws/agent-toolkit-for-aws2.8k stars
337

Create and troubleshoot AWS Glue connections to JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS), Redshift, Snowflake, and BigQuery. Gathers connection hints from user, discovers existing connections and RDS/Redshift candidates, registers credentials in Secrets Manager or IAM DB auth, configures VPC, and tests. Triggers on: connect to database, set up Glue connection, register data source, connect to Snowflake/BigQuery/RDS, connection timeout, test connection, troubleshoot connection. Do NOT use for moving data (use ingesting-into-data-lake), creating tables (use creating-data-lake-table), queries (use querying-data-lake), catalog exploration (use exploring-data-catalog), or SaaS (Salesforce, ServiceNow, SAP, MongoDB, Kafka).

Ask your Agent

Use this Skill: https://skilld.dev/gh/aws/agent-toolkit-for-aws/connecting-to-data-source

This session only. Nothing lands on disk.

referencesbigquery-setup.md

≈662 tokens on demand. Your agent reads this file only when SKILL.md points to it.

BigQuery Connection Setup

AWS Glue native BigQuery connection (type BIGQUERY). Authentication is via a GCP service account; credentials flow through AWS Secrets Manager.

Contents

Prerequisites

  • GCP project with BigQuery enabled
  • Service account in that project with BigQuery access (typically roles/bigquery.dataViewer plus roles/bigquery.jobUser for running jobs)
  • Service account JSON key file from GCP
  • AWS Secrets Manager secret in the same region as the Glue job

Service Account Setup

Service account and key generation happen in GCP, not AWS. For current steps see GCP service account docs and BigQuery access control.

Minimum GCP IAM roles for read-only ingestion:

  • roles/bigquery.dataViewer on the target dataset
  • roles/bigquery.jobUser on the project (to run queries)

For cross-project reads, grant both roles in each source project.

Secrets Manager Storage

Base64-encode the service account JSON and store in Secrets Manager. The Glue BigQuery connection expects the secret value to be the base64 string directly, not a JSON wrapper.

base64 -i <service-account>.json | tr -d '\n' > sa.b64
aws secretsmanager create-secret \
  --name glue/bigquery/<project-id>/credentials \
  --secret-string file://sa.b64 \
  --region <region>
rm sa.b64

Rotate by creating a new key in GCP and updating the secret value. Glue picks up the new value on next job run.

Connection JSON Template

{
  "Name": "bigquery-<project-id>",
  "ConnectionType": "BIGQUERY",
  "ConnectionProperties": {
    "SECRET_ID": "glue/bigquery/<project-id>/credentials"
  }
}

Glue's BigQuery connection talks to Google APIs over the internet. No PhysicalConnectionRequirements needed unless the Glue job itself must run in a specific VPC for other reasons (e.g., also reading from a private RDS). In that case, ensure the subnet has NAT gateway egress so Glue can reach bigquery.googleapis.com.

Further Reading

Source: SKILL.md on GitHub

Third-party checks

No alerts17d3 checks · Risk SAFE
  • Gen Agent Trust Hub17d

    This skill includes some security considerations such as dynamic script generation for troubleshooting and a surface for indirect prompt injection. While these warrant review, they are used within the skill's intended functionality to register and test data source connections. See detailed analysis for context.

  • Socket17d

    No alerts

  • Snyk17d

    Risk: LOW · No issues

Provenance

Signed by skilld at b33847d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago

Capability

Other metadata
metadata
{
  "version": "1",
  "argument-hint": "'[source-type|connection-name|hostname]'"
}

README badge

README badge for aws/agent-toolkit-for-aws/connecting-to-data-source