---
name: clickhouse-data-handling
description: |-
  Handle data lifecycle in ClickHouse — TTL expiration, data deletion (GDPR),
  column-level encryption, and audit logging with real ClickHouse SQL.
  Use when implementing data retention, fulfilling GDPR/CCPA deletion requests,
  or managing sensitive data in ClickHouse.
  Trigger with "clickhouse data retention", "clickhouse TTL", "clickhouse GDPR",
  "delete data clickhouse", "clickhouse data lifecycle", "clickhouse PII".
allowed-tools: Read, Write, Edit
version: 1.7.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
tags:
  - saas
  - database
  - analytics
  - clickhouse
  - olap
compatibility: Designed for Claude Code
title: clickhouse-data-handling
canonical_url: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling
last_updated: 2026-10-07T06:06:28.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [references/examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/examples.md), [references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/implementation.md).
>
> If the user asked to install this Skill, run `npx skilld install jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/clickhouse-data-handling`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/clickhouse-data-handling`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/clickhouse-data-handling --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/clickhouse-data-handling --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# ClickHouse Data Handling

## Overview

Manage the full data lifecycle in ClickHouse: TTL-based expiration, GDPR/CCPA
deletion, data masking, partition management, and audit trails. This skill
produces migration SQL and TypeScript client code you write into your project,
then verifies the results against ClickHouse `system.*` tables.

The workflow below is the high-level path — each step links to the full,
copy-ready SQL/TypeScript in [references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/implementation.md),
with end-to-end scenarios in [references/examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/examples.md).

## Prerequisites

Before starting, confirm you have:

- Populated ClickHouse tables to operate on (schema comes from the companion
  skill `clickhouse-core-workflow-a`).
- A written data-retention policy: how long each data class is kept, and which
  columns hold PII. The [Data Classification](#data-classification) table maps
  each class to its ClickHouse handling.
- ClickHouse 23.3+ if you plan to use lightweight `DELETE FROM`; older versions
  must use mutation-based `ALTER TABLE ... DELETE`.
- Access to `system.mutations` and `system.parts` to verify deletions.

## Instructions

Work the six steps in order for a new table, or jump to the one you need. Use
`Write`/`Edit` to place the generated SQL into a migration file (or the
TypeScript into your data-access layer), then run it against ClickHouse and
verify via the `system.*` queries. Full code for each step lives in
[references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/implementation.md).

1. **TTL-based expiration** — attach a `TTL` clause so data self-deletes, or use
   tiered `TO VOLUME` storage (hot → cold → delete) and column-level TTL to null
   out PII while keeping the row. Skeleton:

   ```sql
   ALTER TABLE analytics.events
       MODIFY TTL created_at + INTERVAL 90 DAY;
   ```

2. **GDPR/CCPA deletion** — choose lightweight `DELETE FROM` (23.3+), verifiable
   `ALTER TABLE ... DELETE` (the compliant path), or `DROP PARTITION` for bulk.
   Always confirm completion in `system.mutations`.
3. **Masking & anonymization** — expose a `CREATE VIEW` that `sipHash64`-hashes
   identifiers and shows only email domains, gated by a dictionary allowlist.
4. **DSAR export & delete** — the TypeScript `exportUserData` / `deleteUserData`
   helpers loop every table for one `user_id` and log each deletion.
5. **Audit trail** — an immutable, TTL-free `audit_log` table partitioned by
   month so retention actions are provable.
6. **Retention monitoring** — a `system.tables`/`system.parts` join that reports
   size, age span, and any MergeTree table missing a TTL.

## Data Classification

| Category | Examples | Handling in ClickHouse |
|----------|----------|------------------------|
| PII | Email, name, IP | Column-level TTL, masking views, deletion support |
| Sensitive | API keys, tokens | Never store in ClickHouse — use secret managers |
| Business | Event counts, metrics | Standard TTL, aggregate for long-term retention |
| Audit | Access logs | No TTL, immutable, partitioned by month |

## Output

Applying this skill produces:

- **Migration SQL** — `CREATE TABLE`/`ALTER TABLE` statements adding TTL clauses,
  masking views, and the immutable `audit_log` table, ready to commit as a
  migration file.
- **TypeScript client code** — `exportUserData` and `deleteUserData` functions
  for DSAR and erasure requests against `@clickhouse/client`.
- **Verification queries** — `system.mutations` / `system.parts` / `system.tables`
  SELECTs that prove a deletion finished and flag tables missing retention.
- **An audit record** — one immutable `audit_log` row per compliance action.

## Error Handling

| Issue | Cause | Solution |
|-------|-------|----------|
| Mutation stuck | Large table rewrite | Check `system.mutations`, cancel if needed |
| TTL not expiring | No merges running | `OPTIMIZE TABLE ... FINAL` to force |
| DELETE not working | Old ClickHouse version | Use `ALTER TABLE DELETE` (mutation) |
| Export timeout | Too much user data | Add LIMIT or export in batches |

## Examples

A minimal TTL attach — the smallest useful action:

```sql
ALTER TABLE analytics.events
    MODIFY TTL created_at + INTERVAL 90 DAY;
OPTIMIZE TABLE analytics.events FINAL;   -- force the cleanup now
```

Full worked scenarios — a complete GDPR erasure (export → verifiable delete →
audit log), standing up a retention-safe table with tiered storage, and auditing
for tables missing a retention policy — are in
[references/examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/examples.md). The step-by-step SQL and
TypeScript each example composes lives in
[references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/implementation.md).

## Resources

- [TTL for Data Management](https://clickhouse.com/docs/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl)
- [DELETE Statement](https://clickhouse.com/docs/sql-reference/statements/delete)
- [Mutations](https://clickhouse.com/docs/guides/developer/mutations)
- [references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/implementation.md) — full SQL + TypeScript for all six steps
- [references/examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/clickhouse-data-handling/references/examples.md) — end-to-end GDPR / retention scenarios

## Next Steps

For role-based access control that restricts who can run these deletion and
export operations, see the companion skill `clickhouse-enterprise-rbac`. For the
table schemas these lifecycle rules attach to, see `clickhouse-core-workflow-a`.
