Index of every page on this site: llms.txt
All skills
wshobson avatar

/spark-optimization

@4220308
by Seth Hobsonwshobson/agents43k stars
4,518
Run it yourself

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Use this Skill: https://skilld.dev/gh/wshobson/agents/spark-optimization

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈53 tokens for metadata: the name and description. ≈730 when used: this file. ≈2.5k more on demand in 1 file.

Description uses 2.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈213 characters, or 2.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

When to Use This Skill

  • Optimizing slow Spark jobs
  • Tuning memory and executor configuration
  • Implementing efficient partitioning strategies
  • Debugging Spark performance issues
  • Scaling Spark pipelines for large datasets
  • Reducing shuffle and data skew

Core Concepts

1. Spark Execution Model

Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)

2. Key Performance Factors

Factor Impact Solution
Shuffle Network I/O, disk I/O Minimize wide transformations
Data Skew Uneven task duration Salting, broadcast joins
Serialization CPU overhead Use Kryo, columnar formats
Memory GC pressure, spills Tune executor memory
Partitions Parallelism Right-size partitions

Quick Start

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

# Create optimized Spark session
spark = (SparkSession.builder
    .appName("OptimizedJob")
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
    .config("spark.sql.adaptive.skewJoin.enabled", "true")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .config("spark.sql.shuffle.partitions", "200")
    .getOrCreate())

# Read with optimized settings
df = (spark.read
    .format("parquet")
    .option("mergeSchema", "false")
    .load("s3://bucket/data/"))

# Efficient transformations
result = (df
    .filter(F.col("date") >= "2024-01-01")
    .select("id", "amount", "category")
    .groupBy("category")
    .agg(F.sum("amount").alias("total")))

result.write.mode("overwrite").parquet("s3://bucket/output/")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Enable AQE - Adaptive query execution handles many issues
  • Use Parquet/Delta - Columnar formats with compression
  • Broadcast small tables - Avoid shuffle for small joins
  • Monitor Spark UI - Check for skew, spills, GC
  • Right-size partitions - 128MB - 256MB per partition

Don'ts

  • Don't collect large data - Keep data distributed
  • Don't use UDFs unnecessarily - Use built-in functions
  • Don't over-cache - Memory is limited
  • Don't ignore data skew - It dominates job time
  • Don't use .count() for existence - Use .take(1) or .isEmpty()

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo5 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill provides legitimate technical documentation and code patterns for optimizing Apache Spark jobs. It covers standard performance tuning topics such as partitioning, memory management, and shuffle optimization without any malicious code or security risks.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

  • Runlayer8mo

    1/1 file flagged

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 4220308. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 3 days ago.

Activeupdated 6 months ago
  • apache-spark
  • performance-tuning
  • partitioning
  • shuffle-optimization
  • memory-management
  • data-skew
  • pyspark
  • distributed-computing

README badge

README badge for wshobson/agents/spark-optimization

Optimizes Apache Spark jobs through partitioning strategies, memory tuning, shuffle reduction, and data skew mitigation. Targets production Spark pipelines and includes configuration patterns for adaptive query execution, serialization, and broadcast joins.

Generated from the current SKILL.md.

Does this skill cover Spark on Kubernetes or just standalone/YARN clusters?
The skill focuses on general Spark optimization patterns (partitioning, caching, shuffle, memory tuning) that apply across all cluster managers. Kubernetes-specific deployment or resource allocation is not covered.
Does this include Spark Streaming or structured streaming optimization?
No. The skill targets batch Spark jobs and SQL workloads. Streaming-specific patterns like micro-batch tuning or backpressure handling are not included.
What Python and Spark versions does this assume?
The skill uses PySpark and assumes Spark 3.x with Adaptive Query Execution (AQE) available. Specific version pinning is not mentioned in the documentation.
Does this cover Delta Lake optimization or just Parquet?
The skill recommends Delta Lake and Parquet as columnar formats but does not include Delta-specific optimization patterns like Z-order clustering or merge/upsert tuning.
Are GPU or machine learning workloads covered?
No. The skill is limited to CPU-based distributed data processing optimization. GPU acceleration and ML-specific tuning are out of scope.

Generated from the current SKILL.md. These answers refresh after source changes.