Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/apify-performance-tuning

@820dfc8
Run it yourself

Optimize Apify Actor performance: crawl speed, memory usage, concurrency, and proxy rotation. Use when Actors are slow, consuming too much memory, or being blocked by target sites, or when a crawl is too expensive per run. Trigger with "apify performance", "optimize apify actor", "apify slow", "crawlee concurrency", "apify memory tuning", "scraper performance".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-performance-tuning

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈97 tokens for metadata: the name and description. ≈1.4k when used: this file. ≈2.1k more on demand in 2 files.

Description uses 4.9% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈390 characters, or 4.9%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Apify Performance Tuning

Overview

Optimize Apify Actors for speed, cost, and reliability. Covers Crawlee concurrency settings, memory profiling, proxy rotation strategies, request batching, and crawler selection for different workloads.

The workflow is a repeatable loop: measure a baseline, apply one lever, re-measure. The single highest-impact lever is usually crawler choice — swapping a browser crawler for CheerioCrawler on non-JS pages is a 5-10x speedup on its own. The full six-step walkthrough, with every code block, lives in references/implementation.md.

Prerequisites

  • Existing Actor with measurable baseline performance
  • Understanding of apify-sdk-patterns
  • Access to Actor run stats in Apify Console
  • APIFY_TOKEN in the environment for reading run stats via ApifyClient

Instructions

Work the levers in order. Each step is expanded — with copy-paste code — in the reference file linked below.

  1. Measure a baseline. Pull runTimeSecs, requestsFinished, memAvgBytes, and usageTotalUsd from the run stats before changing anything. You cannot judge an optimization without a before number.
  2. Choose the right crawler. HttpCrawler/CheerioCrawler for static HTML or JSON (low memory, fast); PlaywrightCrawler/PuppeteerCrawler only when the page genuinely needs JavaScript rendering.
  3. Tune concurrency. Raise maxConcurrency for Cheerio (up to ~50); keep it low (~3-5) for browser crawlers because each page costs ~200MB. Let autoscaledPoolOptions adjust within the band.
  4. Optimize memory. Push data immediately instead of accumulating arrays; for browser crawlers, block images/CSS/fonts in preNavigationHooks and cap concurrent browsers.
  5. Right-size the memory allocation. Compute units bill on memory x duration — start low (512 MB for Cheerio) and only raise it if the Actor is memory-starved.
  6. Rotate proxies and tune requests. Start on datacenter proxies, fall back to residential on 403/blocked; use a session pool for IP rotation and ban detection.

Full step-by-step walkthrough with all code: references/implementation.md.

The minimal starting skeleton — swap a browser crawler for Cheerio and push immediately:

import { CheerioCrawler } from 'crawlee';
import { Actor } from 'apify';

const crawler = new CheerioCrawler({
  maxConcurrency: 50,             // Cheerio is cheap — parallelize hard
  maxRequestsPerMinute: 300,      // But cap the rate to protect the target
  requestHandler: async ({ $, request }) => {
    await Actor.pushData({ url: request.url, title: $('title').text().trim() });
  },
});

Output

Applying this skill produces:

  • A baseline vs. tuned metrics comparison (pages/min, avg/max memory, compute units, cost/run) drawn from the run stats.
  • A crawler and concurrency recommendation matched to whether the target pages need JS rendering.
  • A memory allocation value sized to the workload, with the compute-unit cost tradeoff made explicit.
  • A proxy and session strategy (datacenter-first with residential fallback) for reliability under anti-bot blocking.

Instrument the running crawl to confirm the gains in real time — see references/monitoring.md for the throughput logger and a before/after impact table.

Error Handling

Issue Cause Solution
Out of memory crash Too many concurrent browsers Reduce maxConcurrency
Slow crawl speed Low concurrency Increase maxConcurrency
High failure rate Anti-bot blocking Add proxy, reduce concurrency
Expensive runs Over-provisioned memory Profile and reduce allocation
Stalled crawl Request handler timeout Set requestHandlerTimeoutSecs

Examples

Slow Playwright crawl on static pages. The Actor renders every page in a browser at 3 pages/min. The pages are server-rendered HTML, so switch to CheerioCrawler and raise maxConcurrency — ~30 pages/min (10x). Code: references/implementation.md Steps 1-2.

Out-of-memory crashes under load. A PlaywrightCrawler at maxConcurrency: 20 OOMs. Drop concurrency to 3, block images/CSS/fonts in preNavigationHooks, and call window.stop() post-navigation. Code: references/implementation.md Step 3.

Getting blocked (403s) mid-crawl. Start on datacenter proxies, and on a 403 re-enqueue the request with a residential proxy and retire the session to force a new IP. Code: references/implementation.md Step 5.

Runs cost too much. A 4GB allocation bills 8x more than needed for HTML parsing. Right-size to 512 MB and re-measure cost/run. Code: references/implementation.md Step 4; impact table in references/monitoring.md.

Resources

Next Steps

For cost optimization beyond performance tuning, see the apify-cost-tuning skill in this pack — it covers compute-unit budgeting, storage costs, and scheduling strategies that this skill's memory right-sizing feeds into.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill provides legitimate optimization techniques for Apify Actors, including crawler selection and resource management. It correctly uses environment variables for sensitive tokens. A low-severity risk of indirect prompt injection exists because the skill processes data from external websites, which is inherent to web scraping tools.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 4 months ago
What it can do
Reads files Edits files
version
1.5.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
[
  "saas",
  "scraping",
  "automation",
  "apify"
]
All 3 allowed tools
ReadWriteEdit
Other metadata
compatibility
Designed for Claude Code

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/apify-performance-tuning