/chdb-datastore

DataStore Connectors — All Data Sources

Quick reference for connecting DataStore to any data source. After connecting, all sources share the same pandas API.

Table of Contents

• Local Files
• Cloud Storage
• Databases
• Data Lakes
• URI Shorthand
• In-Memory Data

Local Files

DataStore.from_file(path, format=None, structure=None, compression=None, **kwargs)

Format is auto-detected by extension: .parquet, .csv, .tsv, .json, .jsonl, .arrow, .orc, .avro, .xml.

from datastore import DataStore

ds = DataStore.from_file("sales.parquet")
ds = DataStore.from_file("data.csv")
ds = DataStore.from_file("events.jsonl")
ds = DataStore.from_file("logs/*.csv")              # glob pattern
ds = DataStore.from_file("data/2024-*/events.parquet")  # nested glob
ds = DataStore.from_file("data.csv.gz")             # compressed, auto-detected
ds = DataStore.from_file("data.tsv", format="TabSeparatedWithNames")  # explicit format

Notes:

• Glob patterns (*, **) work for querying multiple files at once
• Compression (.gz, .zst, .bz2, .xz, .lz4) is auto-detected from extension
• Use structure parameter to specify column types: structure="id UInt64, name String"

Cloud Storage

S3

DataStore.from_s3(url, access_key_id=None, secret_access_key=None, format=None, nosign=False, **kwargs)

# Public bucket (no auth)
ds = DataStore.from_s3("s3://public-data/dataset.parquet", nosign=True)

# Private bucket
ds = DataStore.from_s3("s3://my-bucket/data.parquet",
    access_key_id="AKIA...", secret_access_key="secret...")

# Glob pattern
ds = DataStore.from_s3("s3://bucket/logs/2024-*.parquet", nosign=True)

GCS (Google Cloud Storage)

DataStore.from_gcs(url, hmac_key=None, hmac_secret=None, format=None, nosign=False, **kwargs)

ds = DataStore.from_gcs("gs://my-bucket/data.parquet", nosign=True)
ds = DataStore.from_gcs("gs://private/data.parquet", hmac_key="KEY", hmac_secret="SECRET")

Azure Blob Storage

DataStore.from_azure(connection_string, container, path="", format=None, **kwargs)

ds = DataStore.from_azure(
    connection_string="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...",
    container="data", path="analytics/events.parquet")

HDFS

DataStore.from_hdfs(uri, format=None, structure=None, **kwargs)

ds = DataStore.from_hdfs("hdfs://namenode:9000/warehouse/events/*.parquet")

HTTP/HTTPS URL

DataStore.from_url(url, format=None, structure=None, headers=None, **kwargs)

ds = DataStore.from_url("https://example.com/data.csv")

Databases

MySQL

DataStore.from_mysql(host, database=None, table=None, user=None, password="", port=None, **kwargs)

ds = DataStore.from_mysql(
    host="db.example.com:3306", database="shop",
    table="orders", user="root", password="pass")

Note: Port must be included in host string (e.g., "db:3306") or passed via port parameter.

PostgreSQL

DataStore.from_postgresql(host, database=None, table=None, user=None, password="", port=None, **kwargs)

ds = DataStore.from_postgresql(
    host="pg:5432", database="analytics",
    table="events", user="user", password="pass")

ClickHouse (Remote)

DataStore.from_clickhouse(host, database=None, table=None, user="default", password="", secure=False, port=None, **kwargs)

ds = DataStore.from_clickhouse(host="ch:9000", database="logs", table="access_log")
ds = DataStore.from_clickhouse(host="ch:9440", database="logs", table="hits",
    user="reader", password="pass", secure=True)

MongoDB

DataStore.from_mongodb(host, database, collection, user, password="", **kwargs)

ds = DataStore.from_mongodb(
    host="mongo:27017", database="app",
    collection="users", user="user", password="pass")

SQLite

DataStore.from_sqlite(database_path, table, **kwargs)

ds = DataStore.from_sqlite("/data/local.db", "users")

Redis

DataStore.from_redis(host, key, structure, password=None, db_index=0, **kwargs)

ds = DataStore.from_redis("localhost:6379", key="mydata",
    structure="id UInt64, name String, value Float64")

Data Lakes

Apache Iceberg

DataStore.from_iceberg(url, access_key_id=None, secret_access_key=None, **kwargs)

ds = DataStore.from_iceberg("s3://warehouse/iceberg/events",
    access_key_id="KEY", secret_access_key="SECRET")

Delta Lake

DataStore.from_delta(url, access_key_id=None, secret_access_key=None, **kwargs)

ds = DataStore.from_delta("s3://warehouse/delta/transactions",
    access_key_id="KEY", secret_access_key="SECRET")

Apache Hudi

DataStore.from_hudi(url, access_key_id=None, secret_access_key=None, **kwargs)

ds = DataStore.from_hudi("s3://warehouse/hudi/logs",
    access_key_id="KEY", secret_access_key="SECRET")

URI Shorthand

DataStore.uri(uri_string, **kwargs)

Universal one-liner that auto-detects source type from the URI scheme:

from datastore import DataStore

ds = DataStore.uri("s3://public-data/dataset.parquet?nosign=true")
ds = DataStore.uri("mysql://root:pass@localhost:3306/shop/orders")
ds = DataStore.uri("postgresql://analyst:pass@pg:5432/analytics/events")
ds = DataStore.uri("clickhouse://ch:9440/analytics/hits?user=reader&password=pass")
ds = DataStore.uri("mongodb://user:pass@mongo:27017/logs.app_events")
ds = DataStore.uri("sqlite:///data/local.db?table=users")
ds = DataStore.uri("deltalake:///data/delta/events")

In-Memory Data

From dict

ds = DataStore({"name": ["Alice", "Bob"], "age": [25, 30]})

From pandas DataFrame

ds = DataStore(df)
ds = DataStore.from_df(df, name="my_data")

Generated sequences

ds = DataStore.from_numbers(100)                   # 0..99
ds = DataStore.from_numbers(10, start=5, step=2)   # 5, 7, 9, ...

Random data (for testing)

ds = DataStore.from_random(
    structure="id UInt64, name String, value Float64",
    random_seed=42, max_string_length=10)