---
name: detecting-data-anomalies
description: 'Process identify anomalies and outliers in datasets using machine learning algorithms.

  Use when analyzing data for unusual patterns, outliers, or unexpected deviations from normal behavior.

  Trigger with phrases like "detect anomalies", "find outliers", or "identify unusual patterns".'
allowed-tools: Read, Bash(python:*), Grep, Glob
version: 1.27.0
author: Jeremy Longshore <jeremy@intentsolutions.io>
license: MIT
tags:
  - ai
  - ml
  - detecting-data
compatibility: Designed for Claude Code
title: detecting-data-anomalies
canonical_url: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies
last_updated: 2026-10-23T05:08:30.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [assets/README.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/assets/README.md), [references/errors.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/references/errors.md), [references/examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/references/examples.md), [references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/references/implementation.md), [references/README.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/references/README.md), [scripts/README.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/scripts/README.md).
>
> If the user asked to install this Skill, run `npx skilld install jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/detecting-data-anomalies`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/detecting-data-anomalies`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/detecting-data-anomalies --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/detecting-data-anomalies --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Detecting Data Anomalies

## Overview

Identify anomalies and outliers in datasets using statistical and machine learning algorithms including Isolation Forest, One-Class SVM, Local Outlier Factor, and autoencoders. This skill handles the full detection pipeline from data ingestion and feature scaling through algorithm selection, threshold tuning, and result interpretation with anomaly scoring.

## Prerequisites

- Python 3.9+ with scikit-learn >= 1.3 (`pip install scikit-learn`)
- pandas and NumPy for data manipulation (`pip install pandas numpy`)
- matplotlib or seaborn for anomaly visualizations (`pip install matplotlib seaborn`)
- Dataset in CSV, JSON, Parquet, or database-queryable format
- Minimum 500 data points for statistical significance (1000+ recommended)
- Optional: PyTorch or TensorFlow for autoencoder-based detection on complex patterns

## Instructions

1. Load the dataset using the Read tool and verify schema, column types, and row count
2. Profile feature distributions using descriptive statistics to understand baseline behavior
3. Handle missing values via imputation (median for numeric, mode for categorical) or row exclusion
4. Apply StandardScaler or MinMaxScaler to numeric features to normalize magnitude differences
5. Select the detection algorithm based on data characteristics:
   - **Isolation Forest**: high-dimensional data, no assumptions on distribution
   - **One-Class SVM**: well-defined normal class with clear decision boundary
   - **Local Outlier Factor**: density-varying data with local anomaly patterns
   - **Autoencoder**: complex temporal or image data with non-linear relationships
6. Set the contamination parameter to the expected anomaly proportion (start with 0.01-0.05)
7. Fit the model on the training partition and generate anomaly scores for each data point
8. Apply the decision threshold to classify points as normal (-1) or anomalous (1)
9. Analyze flagged anomalies for common characteristics, temporal clusters, or feature correlations
10. Generate a summary report with detection counts, score distributions, and visualization plots

See [`references/implementation.md`](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/references/implementation.md) for the detailed implementation guide.

## Output

- Anomaly detection summary: total points, anomaly count, contamination rate
- Per-record anomaly scores with classification labels
- Algorithm configuration: model type, contamination, distance metric, threshold
- Feature importance ranking showing which dimensions drive anomaly flags
- Visualization: scatter plot of anomaly scores, distribution histogram, t-SNE cluster plot
- CSV export of flagged records with anomaly scores and contributing features

## Error Handling

| Error | Cause | Solution |
|-------|-------|----------|
| Insufficient data volume | Fewer than 100 data points for model fitting | Collect additional data or switch to simple statistical methods (z-score, IQR) |
| High false positive rate | Contamination parameter set too high or features not scaled | Lower contamination to 0.01; verify StandardScaler applied; refine feature selection |
| Algorithm OOM on large dataset | Isolation Forest or LOF exceeds available memory | Subsample data for training; use `max_samples` parameter; switch to streaming approach |
| Feature scaling mismatch | Mixed numeric and categorical features without proper encoding | One-hot encode categoricals separately; scale numeric features independently |
| No ground truth for validation | Unlabeled dataset prevents accuracy measurement | Use domain expert review on top-N anomalies; implement feedback loop to refine threshold |

See [`references/errors.md`](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/detecting-data-anomalies/references/errors.md) for the full error reference.

## Examples

**Scenario 1: Network Intrusion Detection** -- Apply Isolation Forest to 50K network flow records with features: packet count, byte volume, duration, protocol type. Expected contamination: 2%. Target: flag port-scan and DDoS patterns with precision above 0.85.

**Scenario 2: Manufacturing Quality Control** -- Run LOF on sensor readings (temperature, vibration, pressure) from 10K production cycles. Detect equipment degradation anomalies. Visualize flagged cycles on a time-series plot with normal operating bands.

**Scenario 3: Financial Transaction Monitoring** -- Train an autoencoder on 100K legitimate transactions. Reconstruct test transactions and flag those with reconstruction error above the 99th percentile. Report flagged transactions with amount, merchant category, and time-of-day features.

## Resources

- [scikit-learn Anomaly Detection](https://scikit-learn.org/stable/modules/outlier_detection.html) -- Isolation Forest, LOF, One-Class SVM
- [PyOD Library](https://pyod.readthedocs.io/) -- 40+ outlier detection algorithms with unified API
- Autoencoder anomaly detection: Keras/PyTorch reconstruction-error approach
- Feature scaling: StandardScaler, RobustScaler, MinMaxScaler selection guide
- Evaluation without labels: silhouette analysis, domain expert review protocols
