Index of every page on this site: llms.txt
All skills
posthog avatar

/exploring-llm-clusters

@028173b official
by posthogposthog/posthog41k stars
3,718
Run it yourself

Investigate AI observability clusters — understand usage patterns in AI/LLM traffic, compare cluster behavior, compute cost/latency metrics, and drill into individual traces within clusters.

Use this Skill: https://skilld.dev/gh/posthog/posthog/exploring-llm-clusters

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈54 tokens for metadata: the name and description. ≈4.6k when used: this file.

Description uses 2.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈215 characters, or 2.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Exploring LLM clusters

Use this skill when investigating AI observability clusters — understanding what patterns exist in your AI/LLM traffic, comparing cluster behavior, and drilling into individual clusters.

Tools

Tool Purpose
posthog:llma-clustering-job-list List clustering job configurations for the team
posthog:llma-clustering-job-get Get a specific clustering job by ID
posthog:execute-sql Query cluster run events and compute metrics
posthog:query-llm-traces-list Find traces belonging to a cluster
posthog:query-llm-trace Inspect a specific trace in detail

How clustering works

PostHog clusters LLM traces, individual generations, or evaluation events by embedding similarity. A Temporal workflow runs periodically or on-demand, producing cluster events stored as $ai_trace_clusters (trace-level), $ai_generation_clusters (generation-level), or $ai_evaluation_clusters (evaluation-level).

Each cluster event contains:

  • $ai_clustering_run_id — unique run identifier (format: <team_id>_<level>_<YYYYMMDD>_<HHMMSS>[_<job_id>])
  • $ai_clustering_level — "trace", "generation", or "evaluation"
  • $ai_window_start / $ai_window_end — time window of the data that was analyzed
  • $ai_total_items_analyzed — number of traces, generations, or evaluations processed
  • $ai_clusters — JSON array of cluster objects
  • $ai_clustering_params — algorithm parameters used

The analyzed window closes when a run starts, and the cluster event lands once the run finishes. So the cluster event's own timestamp is always after $ai_window_end, by anything from seconds to hours. Use the window only to bound the traces, generations, and evaluations that were analyzed. To find the cluster event itself, filter on $ai_clustering_run_id with a plain recent-time bound.

Cluster object shape (inside $ai_clusters)

{
  "cluster_id": 0,
  "size": 42,
  "title": "User authentication flows",
  "description": "Traces involving login, signup, and token refresh operations",
  "traces": {
    "<trace_or_generation_id>": {
      "distance_to_centroid": 0.123,
      "rank": 0,
      "x": -2.34,
      "y": 1.56,
      "timestamp": "2026-03-28T10:00:00Z",
      "trace_id": "abc-123",
      "generation_id": "gen-456"
    }
  },
  "centroid_x": -2.1,
  "centroid_y": 1.4
}
  • cluster_id: -1 is the noise/outlier cluster (items that didn't fit any cluster)
  • Items in traces are keyed by trace ID (trace-level), generation event UUID (generation-level), or evaluation event UUID (evaluation-level)
  • rank orders items by proximity to centroid (0 = closest)
  • x, y are 2D coordinates for visualization (UMAP/PCA/t-SNE reduced)

Clustering jobs

Each team can have up to 10 clustering jobs. A job defines:

  • name — human-readable label
  • analysis_level — "trace", "generation", or "evaluation"
  • event_filters — property filters scoping which items are included
  • enabled — whether the job runs on schedule

Default jobs named "Default - traces", "Default - generations", and "Default - evaluations" are auto-created and disabled when a custom job is created for the same level.

Workflow: explore clusters

Check setup and scheduled results

If there are no recent results, read posthog:llma-clustering-config-get before diagnosing missing data. Clustering is off by default. If the user wants automatic clustering, use posthog:llma-clustering-config-set-enabled with enabled: true; organization AI data processing consent is required. Reading historical clusters does not require enabling new runs.

Enabling clustering lets existing schedules prepare new data and find groups; it does not start an immediate run or backfill historical traces. Each job needs at least 10 eligible prepared items, and enough similar items to form groups. Trace and generation jobs look back seven days; evaluation jobs look back three days and exclude numeric and categorical results. If results are missing, inspect the enabled jobs and their saved filters, and confirm that matching data continues to arrive.

Step 1 — List recent clustering runs

posthog:execute-sql
SELECT
    toString(properties.$ai_clustering_run_id) AS run_id,
    toString(properties.$ai_clustering_level) AS level,
    toString(properties.$ai_clustering_job_id) AS job_id,
    toString(properties.$ai_clustering_job_name) AS job_name,
    toString(properties.$ai_window_start) AS window_start,
    toString(properties.$ai_window_end) AS window_end,
    toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
    AND timestamp >= now() - INTERVAL 14 DAY
ORDER BY timestamp DESC
LIMIT 10

For a supplied run ID, add AND toString(properties.$ai_clustering_run_id) = '<run_id>' to this query's WHERE clause to retrieve its level and analyzed window. Widen the timestamp lookback to include that run's emitted cluster event; its analyzed window ends before the event is emitted.

Step 2 — Get clusters from a specific run

Return a page of summaries first. Keep the run's level and analyzed window from Step 1 for the later queries.

posthog:execute-sql
WITH cluster_run AS (
    SELECT coalesce(toString(properties.$ai_clusters), '[]') AS clusters
    FROM events
    WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
        AND timestamp >= now() - INTERVAL 14 DAY
        AND toString(properties.$ai_clustering_run_id) = '<run_id>'
    ORDER BY timestamp DESC
    LIMIT 1
)
SELECT
    JSONExtractInt(cluster, 'cluster_id') AS cluster_id,
    JSONExtractInt(cluster, 'size') AS size,
    left(JSONExtractString(cluster, 'title'), 200) AS title,
    left(JSONExtractString(cluster, 'description'), 500) AS description,
    left(JSONExtractRaw(cluster, 'metrics'), 500) AS metrics_preview
FROM (
    SELECT arrayJoin(JSONExtractArrayRaw(clusters)) AS cluster
    FROM cluster_run
)
ORDER BY size DESC, cluster_id
LIMIT 10 OFFSET 0

Keep the lookback bound wide enough to cover the timestamp Step 1 reported for the run. Never bound these cluster-run queries with $ai_window_start / $ai_window_end. The cluster event is emitted after the window closes, so those bounds return zero rows.

After each complete page, increase OFFSET by the number of rows returned. Stop when a complete page contains fewer rows than the current LIMIT, including an empty final page when needed. Titles, descriptions, and metrics are capped previews. For more description text, filter to the chosen cluster_id and use substringUTF8(JSONExtractString(cluster, 'description'), 501, 500), then starts 1001, 1501, and so on. Keep each text chunk bounded, or report the capped excerpt. For a specific metric, select only that field for the chosen cluster_id. For example, replace metrics_preview with JSONExtractRaw(cluster, 'metrics', 'pass_rate') AS pass_rate to preserve missing or null values. Cap any string metric with left or retrieve it in bounded chunks too.

To get representative members of one cluster, extract only their IDs and ranks:

posthog:execute-sql
WITH cluster_run AS (
    SELECT coalesce(toString(properties.$ai_clusters), '[]') AS clusters
    FROM events
    WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
        AND timestamp >= now() - INTERVAL 14 DAY
        AND toString(properties.$ai_clustering_run_id) = '<run_id>'
    ORDER BY timestamp DESC
    LIMIT 1
),
cluster_objects AS (
    SELECT arrayJoin(JSONExtractArrayRaw(clusters)) AS cluster
    FROM cluster_run
)
SELECT
    tupleElement(member, 1) AS item_id,
    JSONExtractString(tupleElement(member, 2), 'trace_id') AS trace_id,
    JSONExtractString(tupleElement(member, 2), 'generation_id') AS generation_id,
    JSONExtractInt(tupleElement(member, 2), 'rank') AS rank,
    JSONExtractFloat(tupleElement(member, 2), 'distance_to_centroid') AS distance_to_centroid
FROM (
    SELECT arrayJoin(JSONExtractKeysAndValuesRaw(cluster, 'traces')) AS member
    FROM cluster_objects
    WHERE JSONExtractInt(cluster, 'cluster_id') = <cluster_id>
)
ORDER BY rank, item_id
LIMIT 5 OFFSET 0

Use trace_id with the trace tools. Use item_id for membership: it is the trace ID at trace level and the event UUID at generation or evaluation level. For metrics across the whole cluster, start with LIMIT 100 OFFSET 0 and paginate using the same offset and completion rules as the summaries. Compute metrics for each batch with Step 3; representative members alone do not describe the whole cluster.

Large tool responses may be truncated without a local file. If a page is incomplete, reduce its limit or field selection and retry the same offset before continuing.

Step 3 — Compute metrics for clusters

For trace-level clusters, compute cost/latency/token metrics. The latency rule is the same as in query-llm-trace and query-llm-traces-list, so the numbers match the trace view:

posthog:execute-sql
SELECT
    properties.$ai_trace_id AS trace_id,
    round(sumIf(toFloat(properties.$ai_total_cost_usd),
                event IN ('$ai_generation', '$ai_embedding')), 10) AS total_cost,
    round(
        coalesce(
            -- The root $ai_trace event reports the wall-clock latency of the whole trace.
            nullIf(maxIf(toFloat(properties.$ai_latency),
                         event = '$ai_trace' AND toFloat(properties.$ai_latency) > 0), 0),
            CASE
                WHEN countIf(toFloat(properties.$ai_latency) > 0 AND event != '$ai_generation') = 0
                     AND countIf(toFloat(properties.$ai_latency) > 0 AND event = '$ai_generation') > 0
                THEN sumIf(toFloat(properties.$ai_latency),
                           event = '$ai_generation' AND toFloat(properties.$ai_latency) > 0)
                ELSE sumIf(toFloat(properties.$ai_latency),
                           properties.$ai_parent_id IS NULL
                           OR toString(properties.$ai_parent_id) = toString(properties.$ai_trace_id))
            END
        ), 2
    ) AS latency,
    countIf(event = '$ai_trace' AND toFloat(properties.$ai_latency) > 0) > 0 AS has_root_latency,
    sumIf(toInt(properties.$ai_input_tokens),
          event IN ('$ai_generation', '$ai_embedding')) AS input_tokens,
    sumIf(toInt(properties.$ai_output_tokens),
          event IN ('$ai_generation', '$ai_embedding')) AS output_tokens,
    countIf(isNotNull(properties.$ai_error) OR properties.$ai_is_error = 'true') AS error_count
FROM events
WHERE event IN ('$ai_trace', '$ai_generation', '$ai_embedding', '$ai_span')
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND properties.$ai_trace_id IN ('<trace_id_1>', '<trace_id_2>', ...)
GROUP BY trace_id

How latency is resolved, in order:

  1. A positive $ai_latency on the root $ai_trace event. This is the measured wall-clock duration of the trace.
  2. If no root latency exists and only generations carry latency, the sum of the generation latencies.
  3. Otherwise, the sum of the latencies on the trace's direct children (events with no $ai_parent_id, or a parent equal to the trace ID).

Steps 2 and 3 are approximations. They overstate the duration when children ran in parallel, and they count a nested event that has no $ai_parent_id as a direct child. Use has_root_latency to tell measured durations from approximations, and mention approximations when you compare clusters.

Do not use max($ai_latency) across child events as the trace latency. It returns the longest single step, not the trace duration. For example, a trace runs three root-level spans one after another: 12 s, 9 s, and 6 s, with some time between them. Its $ai_trace event reports 30 s. max returns 12 s and the recipe above returns 30 s. Cluster averages built from max understate latency and can put the clusters in the wrong order.

Cost and tokens only sum $ai_generation and $ai_embedding events, the same as the trace view. The query reads $ai_trace and $ai_span events for latency only, so their properties cannot count the same spend twice. The trace window bounds can cut off the root $ai_trace event, because it is usually the last event of the trace. If has_root_latency is false for traces that should have one, widen <window_end> a little.

For generation-level clusters, match by event UUID:

posthog:execute-sql
SELECT
    toString(uuid) as generation_id,
    toFloat(properties.$ai_total_cost_usd) as cost,
    toFloat(properties.$ai_latency) as latency,
    toInt(properties.$ai_input_tokens) as input_tokens,
    toInt(properties.$ai_output_tokens) as output_tokens,
    if(properties.$ai_is_error = 'true', 1, 0) as is_error
FROM events
WHERE event = '$ai_generation'
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND uuid IN ('<gen_uuid_1>', '<gen_uuid_2>', ...)

For evaluation-level clusters, first check each cluster's metrics from Step 2 (for example pass rate, N/A rate, dominant evaluator name, and average judge cost). Retrieve specific fields when the preview is incomplete. When you need individual evaluation rows, match by event UUID:

posthog:execute-sql
SELECT
    toString(uuid) AS evaluation_id,
    toString(properties.$ai_trace_id) AS trace_id,
    toString(properties.$ai_target_event_id) AS generation_id,
    toString(properties.$ai_evaluation_name) AS evaluation_name,
    toString(properties.$ai_evaluation_result) AS evaluation_result,
    toString(properties.$ai_evaluation_reasoning) AS evaluation_reasoning,
    toFloatOrNull(toString(properties.$ai_total_cost_usd)) AS judge_cost,
    timestamp
FROM events
WHERE event = '$ai_evaluation'
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND uuid IN ('<eval_uuid_1>', '<eval_uuid_2>', ...)

Step 4 — Drill into specific traces

Once you've identified interesting clusters, use the trace tools to inspect individual traces:

posthog:query-llm-trace
{
  "traceId": "<trace_id_from_cluster>",
  "dateRange": {"date_from": "<window_start>", "date_to": "<window_end>"}
}

When you need message content

Use events for cluster events, IDs, cost/latency/token metrics, and evaluation rows. Do not query events.properties.$ai_input, $ai_output, or $ai_output_choices when you need user messages or full model inputs/outputs — those heavy fields live on posthog.ai_events.

For a few representative examples, prefer query-llm-trace; it reads posthog.ai_events for you and returns the full event tree. For batch extraction, first get the trace IDs from the cluster, then query posthog.ai_events anchored on trace_id:

posthog:execute-sql
SELECT
    trace_id,
    timestamp,
    span_id,
    event,
    model,
    input,
    output_choices
FROM posthog.ai_events
WHERE trace_id IN ('<trace_id_1>', '<trace_id_2>', ...)
ORDER BY trace_id, timestamp

posthog.ai_events has a shorter retention window than events; older clusters may still have metadata and metrics but no message content. For the full $ai_* event schema, load the exploring-llm-traces skill.

Investigation patterns

"What kinds of LLM usage do we have?"

  1. List recent clustering runs (Step 1)
  2. Load the latest run's clusters (Step 2)
  3. Review cluster titles and descriptions — each represents a distinct usage pattern
  4. Compare cluster sizes to understand traffic distribution

"Which cluster is most expensive / slowest?"

  1. Load clusters from a run (Step 2)
  2. Extract trace IDs from each cluster
  3. Compute metrics per cluster (Step 3)
  4. Aggregate: avg(cost), avg(latency), sum(cost) per cluster
  5. Compare across clusters

"What's in this cluster?"

  1. Load the cluster's ranked members (Step 2)
  2. Use the lowest rank values (closest to centroid = most representative)
  3. Inspect the top 3-5 traces via query-llm-trace to understand the pattern
  4. Check the cluster title and description for the AI-generated summary

"Are there error-heavy clusters?"

  1. Compute metrics (Step 3) with error_count
  2. Calculate error rate per cluster: items_with_errors / total_items
  3. Focus on clusters with high error rates
  4. Drill into errored traces to find root causes
  5. If a cluster shows a failure that keeps recurring, offer to catch it with an online evaluation scoped to that cluster (creating-online-evaluations)

"How do clusters compare across runs?"

  1. List multiple runs (Step 1)
  2. Load clusters from each run
  3. Compare cluster titles — similar titles across runs indicate stable patterns
  4. Track cluster size changes to detect shifts in traffic patterns

Constructing UI links

  • Clusters overview: https://app.posthog.com/ai-observability/clusters
  • Specific run: https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id>
  • Cluster detail: https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id>/<cluster_id>

Always surface these links so the user can verify visually in the PostHog UI.

Tips

  • Always set a time range in SQL queries — cluster events without time bounds are slow
  • Bound a search for a cluster event by when it was emitted, not by the window it analyzed — a run's $ai_window_end is earlier than the event's own timestamp
  • Start with run listing to orient, then drill into specific clusters
  • Cluster titles and descriptions are AI-generated summaries — verify by inspecting traces
  • The noise cluster (cluster_id: -1) contains outliers that didn't fit any pattern
  • Use llma-clustering-job-list to understand what clustering configs are active
  • Trace IDs in clusters can be used directly with query-llm-trace for deep inspection
  • Message content lives on posthog.ai_events, not events.properties; use query-llm-trace unless you need custom batch SQL
  • For large clusters, inspect the top-ranked traces (closest to centroid) for representative examples

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts6mo3 checks · Risk SAFE
  • Gen Agent Trust Hub6mo

    This skill facilitates the investigation of LLM analytics clusters within the PostHog platform. It uses official tools to query cluster data and provides a helper script for summarizing large datasets. No security issues were detected.

  • Socket6mo

    No alerts

  • Snyk6mo

    Risk: LOW · No issues

Signed by skilld at 028173b. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 3 days ago
  • posthog
  • llm
  • observability
  • clustering
  • traces
  • ai-monitoring
  • cost-analysis
  • latency
  • embeddings

README badge

README badge for posthog/posthog/exploring-llm-clusters

Analyzes AI observability clusters from PostHog to understand LLM traffic patterns, compare cluster behavior, and compute cost and latency metrics. Helps identify expensive or error-prone usage patterns by drilling into individual traces within clusters grouped by embedding similarity.

Generated from the current SKILL.md.

What's the difference between trace-level and generation-level clustering?
Trace-level clustering groups complete LLM workflows (multiple generations, embeddings, spans), while generation-level clustering groups individual model calls. The skill supports both via the `analysis_level` property in clustering jobs.
How do I find traces belonging to a specific cluster?
Load the cluster run with the SQL query in Step 2, then extract trace IDs from the `traces` field within the cluster object. You can then drill into individual traces using `query-llm-trace` or compute metrics across the trace set.
Can I compare clustering results across multiple time periods?
Yes. List recent clustering runs (Step 1), then load clusters from each run separately (Step 2). Compare cluster titles and sizes to detect stable patterns or shifts in traffic patterns over time.
What does the noise cluster (cluster_id: -1) represent?
The noise cluster contains outliers — traces or generations that didn't fit closely enough to any other cluster based on embedding similarity.
How do I compute cost and latency metrics per cluster?
Use Step 3 SQL queries to aggregate cost, latency, and token counts across all traces or generations in a cluster, then divide by cluster size to get per-item averages.

Generated from the current SKILL.md. These answers refresh after source changes.