Index of every page on this site: llms.txt
All skills
jeffallan avatar

/spark-engineer

@67c1961
by jeffallanjeffallan/claude-skills12k stars
1,202
Run it yourself

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.

Use this Skill: https://skilld.dev/gh/jeffallan/claude-skills/spark-engineer

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈110 tokens for metadata: the name and description. ≈1.4k when used: this file. ≈20k more on demand in 5 files.

Description uses 5.5% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈439 characters, or 5.5%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Spark Engineer

Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.

Core Workflow

  1. Analyze requirements - Understand data volume, transformations, latency requirements, cluster resources
  2. Design pipeline - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
  3. Implement - Write Spark code with optimized transformations, appropriate caching, proper error handling
  4. Optimize - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
  5. Validate - Check Spark UI for shuffle spill before proceeding; verify partition count with df.rdd.getNumPartitions(); if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets

Reference Guide

Load detailed guidance based on context:

Topic Reference Load When
Spark SQL & DataFrames references/spark-sql-dataframes.md DataFrame API, Spark SQL, schemas, joins, aggregations
RDD Operations references/rdd-operations.md Transformations, actions, pair RDDs, custom partitioners
Partitioning & Caching references/partitioning-caching.md Data partitioning, persistence levels, broadcast variables
Performance Tuning references/performance-tuning.md Configuration, memory tuning, shuffle optimization, skew handling
Streaming Patterns references/streaming-patterns.md Structured Streaming, watermarks, stateful operations, sinks

Code Examples

Quick-Start Mini-Pipeline (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType

spark = SparkSession.builder \
    .appName("example-pipeline") \
    .config("spark.sql.shuffle.partitions", "400") \
    .config("spark.sql.adaptive.enabled", "true") \
    .getOrCreate()

# Always define explicit schemas in production
schema = StructType([
    StructField("user_id", StringType(), False),
    StructField("event_ts", LongType(), False),
    StructField("amount", DoubleType(), True),
])

df = spark.read.schema(schema).parquet("s3://bucket/events/")

result = df \
    .filter(F.col("amount").isNotNull()) \
    .groupBy("user_id") \
    .agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))

# Verify partition count before writing
print(f"Partition count: {result.rdd.getNumPartitions()}")

result.write.mode("overwrite").parquet("s3://bucket/output/")

Broadcast Join (small dimension table < 200 MB)

from pyspark.sql.functions import broadcast

# Spark will automatically broadcast dim_table; hint makes intent explicit
enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")

Handling Data Skew with Salting

import pyspark.sql.functions as F

SALT_BUCKETS = 50

# Add salt to the skewed key on both sides
skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \
    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))

other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \
    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))

result = skewed_df.join(other_df, on="salted_key", how="inner") \
    .drop("salt", "salted_key")

Correct Caching Pattern

# Cache ONLY when the DataFrame is reused multiple times
df_cleaned = df.filter(...).withColumn(...).cache()
df_cleaned.count()  # Materialize immediately; check Spark UI for spill

report_a = df_cleaned.groupBy("region").agg(...)
report_b = df_cleaned.groupBy("product").agg(...)

df_cleaned.unpersist()  # Release when done

Constraints

MUST DO

  • Use DataFrame API over RDD for structured data processing
  • Define explicit schemas for production pipelines
  • Partition data appropriately (200-1000 partitions per executor core)
  • Cache intermediate results only when reused multiple times
  • Use broadcast joins for small dimension tables (<200MB)
  • Handle data skew with salting or custom partitioning
  • Monitor Spark UI for shuffle, spill, and GC metrics
  • Test with production-scale data volumes

MUST NOT DO

  • Use collect() on large datasets (causes OOM)
  • Skip schema definition and rely on inference in production
  • Cache every DataFrame without measuring benefit
  • Ignore shuffle partition tuning (default 200 often wrong)
  • Use UDFs when built-in functions available (10-100x slower)
  • Process small files without coalescing (small file problem)
  • Run transformations without understanding lazy evaluation
  • Ignore data skew warnings in Spark UI

Output Templates

When implementing Spark solutions, provide:

  1. Complete Spark code (PySpark or Scala) with type hints/types
  2. Configuration recommendations (executors, memory, shuffle partitions)
  3. Partitioning strategy explanation
  4. Performance analysis (expected shuffle size, memory usage)
  5. Monitoring recommendations (key Spark UI metrics to watch)

Knowledge Reference

Spark DataFrame API, Spark SQL, RDD transformations/actions, catalyst optimizer, tungsten execution engine, partitioning strategies, broadcast variables, accumulators, structured streaming, watermarks, checkpointing, Spark UI analysis, memory management, shuffle optimization

Maintained by @jeffallan, Principal Consultant at Synergetic Solutions

Documentation

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 alert1mo5 checks · Risk CRITICAL
  • Gen Agent Trust Hub1mo

    This skill provides comprehensive, high-quality documentation and code examples for Spark engineering. It contains no malicious patterns, credentials, or dangerous execution methods.

  • Socket1mo

    No alerts

  • Snyk1mo

    Risk: LOW · No issues

  • Runlayer8mo

    5/6 files flagged

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at 67c1961. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 4 days ago.

Activeupdated 2 months ago
Other metadata
metadata
{
  "author": "https://github.com/Jeffallan",
  "company": "https://synergetic.solutions",
  "version": "1.1.0",
  "domain": "data-ml",
  "triggers": "Apache Spark, PySpark, Spark SQL, distributed computing, big data, DataFrame API, RDD, Spark Streaming, structured streaming, data partitioning, Spark performance, cluster computing, data processing pipeline",
  "role": "expert",
  "scope": "implementation",
  "output-format": "code",
  "related-skills": "python-pro, sql-pro, devops-engineer"
}
  • apache-spark
  • pyspark
  • spark-sql
  • distributed-computing
  • big-data
  • dataframe
  • etl
  • performance-tuning
  • data-partitioning
  • structured-streaming

README badge

README badge for jeffallan/claude-skills/spark-engineer

Writes and optimizes Apache Spark jobs, including DataFrame transformations, Spark SQL queries, RDD pipelines, and structured streaming applications. Handles partitioning strategy, shuffle tuning, data skew mitigation, and performance analysis via Spark UI metrics for large-scale distributed data processing.

Generated from the current SKILL.md.

Does this skill work with both PySpark and Scala?
The skill covers both PySpark and Scala. Examples are primarily in PySpark, but the core concepts (DataFrame API, RDD operations, partitioning, tuning) apply to both languages.
What should I do if I see shuffle spill in the Spark UI?
The skill's validation workflow requires returning to the optimization step if spill is detected. Common fixes include increasing shuffle partitions, using salting for skewed joins, or tuning executor memory via configuration.
When should I use RDDs instead of DataFrames?
The skill prioritizes DataFrame API for structured data processing. RDDs are covered for unstructured transformations, custom partitioning, or pair operations where Catalyst optimization doesn't apply.
Does this skill cover Structured Streaming?
Yes. The skill includes a reference guide for streaming patterns covering Structured Streaming, watermarks, stateful operations, and sinks, though the focus is primarily on batch ETL pipelines.
What file formats does this skill handle?
The skill specifically mentions Parquet as a primary format. General DataFrame read/write patterns apply to other formats (CSV, JSON, ORC), but Parquet is the recommended production format.

Generated from the current SKILL.md. These answers refresh after source changes.