---
name: apify-rate-limits
description: 'Handle Apify API rate limits with proper backoff and request queuing. Use when hitting 429 errors, optimizing API request throughput, or implementing rate-aware client wrappers. Trigger with "apify rate limit", "apify throttling", "apify 429", "apify retry", "apify backoff", "too many requests apify".'
allowed-tools: Read, Write, Edit
version: 1.5.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
tags:
  - saas
  - scraping
  - automation
  - apify
compatibility: Designed for Claude Code
title: apify-rate-limits
canonical_url: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits
last_updated: 2026-10-07T06:06:28.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [references/examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/examples.md), [references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/implementation.md).
>
> If the user asked to install this Skill, run `npx skilld install jeremylongshore/tons-of-skills-marketplace/apify-rate-limits`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/apify-rate-limits`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/apify-rate-limits`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/apify-rate-limits --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/apify-rate-limits --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Apify Rate Limits

## Overview

The Apify API enforces rate limits per resource. The `apify-client` library
auto-retries 429s (up to 8 times with exponential backoff), so most workloads never
notice a limit. You reach for this skill when bulk operations, custom API calls, or
large fan-outs push past what the built-in retry can absorb — you then batch, queue,
stagger, and monitor to stay under the ceiling.

Full runnable code for every step is in
[implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/implementation.md); combined scenarios are in
[examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/examples.md).

### Apify rate limit rules

| Scope | Limit | Notes |
|-------|-------|-------|
| Per resource (default) | 60 req/sec | Applies to each Actor, dataset, KV store independently |
| Dataset push | 60 req/sec per dataset | Batch items to reduce call count |
| Actor runs | 60 req/sec per Actor | Start runs in sequence or with delays |
| Platform-wide | Higher limit | Aggregate across all resources |

**"Per resource" means:** calls to dataset A and dataset B each get 60 req/sec
independently. Every response carries `X-RateLimit-Limit`,
`X-RateLimit-Remaining`, and `X-RateLimit-Reset` (epoch seconds) headers.

## Prerequisites

- An Apify account with API access and `APIFY_TOKEN` set in the environment.
- The `apify-client` package installed (`npm install apify-client`).
- For custom queuing: `p-queue` (`npm install p-queue`); `crawlee` for `sleep` and
  crawler-level concurrency.

## Instructions

The workflow is five steps. Each is summarized here with its core lever; the full
runnable code for every step is in
[implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/implementation.md).

1. **Understand built-in retries** — `apify-client` already retries 429/500+ with
   exponential backoff. Tune `maxRetries` / `minDelayBetweenRetriesMillis` only when
   the defaults are wrong for your endpoint:

   ```typescript
   import { ApifyClient } from 'apify-client';
   const client = new ApifyClient({
     token: process.env.APIFY_TOKEN,
     maxRetries: 5,                      // Default: 8
     minDelayBetweenRetriesMillis: 500,  // Default: 500
   });
   ```

2. **Batch operations** (biggest lever) — collapse per-item loops into one batched
   call (up to 9 MB), chunking only for very large datasets:

   ```typescript
   await client.dataset(dsId).pushItems(items);   // 1 call, not N
   ```

3. **Queue custom calls** — gate raw API calls through `p-queue`
   (`concurrency` + `intervalCap`) so fan-out reads never exceed 60 req/sec. See
   [implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/implementation.md) § Step 3.

4. **Stagger Actor starts** — insert a ~200 ms delay between `start()` calls so the
   runs endpoint never 429s, then `waitForFinish()` in parallel. See
   [implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/implementation.md) § Step 4.

5. **Monitor headers** — feed `X-RateLimit-*` into a small monitor that warns before
   the wall and pauses exactly until reset. See
   [implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/implementation.md) § Step 5.

Target-website throttling is a separate ceiling from the platform API — cap it with
Crawlee's `maxConcurrency` / `maxRequestsPerMinute`
([implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/implementation.md) § Crawlee-level concurrency).

## Output

Applying this skill produces a rate-aware Apify integration:

- A configured `ApifyClient` with an explicit retry envelope.
- Batched/chunked dataset writes that cut API-call count by orders of magnitude.
- A `p-queue`-gated call path that holds requests under 60 req/sec per resource.
- Staggered Actor starts and, optionally, a header-driven monitor that pauses before
  exhaustion — the net effect being zero (or transparently retried) 429s under load.

## Error Handling

| Scenario | Detection | Response |
|----------|-----------|----------|
| API 429 | `apify-client` auto-retries | Usually transparent; increase delays if persistent |
| Target site 429 | `statusCode === 429` in handler | Reduce `maxConcurrency`, add proxy rotation |
| Burst of starts | Starting 100+ runs at once | Stagger with 200ms delays |
| Large data push | Single 50MB dataset push | Chunk into 9MB batches |

## Examples

Worked end-to-end scenarios live in [examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-rate-limits/references/examples.md):

- **Bulk dataset push without 429s** — 50,000 rows in ~50 calls via chunked batching.
- **Fan-out reads through a queue** — 500 Actor reads held under 50 req/sec.
- **Launch 100 runs safely** — staggered starts, then parallel wait-for-finish.
- **Pause on header-driven exhaustion** — sleep exactly until the limit resets.

## Resources

- [Apify API Rate Limits](https://docs.apify.com/api/v2)
- [p-queue Documentation](https://github.com/sindresorhus/p-queue)
- [Crawlee Auto-scaling](https://crawlee.dev/js/docs/guides/configuration)

For security configuration, see `apify-security-basics`.
