Apify Rate Limits — Full Implementation
Complete, runnable code for every step of the rate-limit workflow. The SKILL.md body summarizes each step and its core lever; this file carries the full detail.
Step 1: Understand built-in retries
The apify-client package handles rate limits automatically — it retries 429 and
500+ responses with exponential backoff. Tune the retry envelope only when the
defaults are too aggressive (hammering a struggling endpoint) or too shallow.
import { ApifyClient } from 'apify-client';
// Default: retries up to 8 times on 429 and 500+ errors
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
// Customize retry behavior
const client = new ApifyClient({
token: process.env.APIFY_TOKEN,
maxRetries: 5, // Default: 8
minDelayBetweenRetriesMillis: 500, // Default: 500
});Step 2: Batch operations to reduce API calls
The cheapest request is the one you never make. Collapse per-item loops into a single batched call (up to a 9 MB payload), and chunk only when the dataset is larger than one payload can carry.
// BAD: 1000 API calls (easily rate limited)
for (const item of items) {
await client.dataset(dsId).pushItems([item]);
}
// GOOD: 1 API call (up to 9MB payload)
await client.dataset(dsId).pushItems(items);
// GOOD: Chunked for very large datasets
function chunkArray<T>(arr: T[], size: number): T[][] {
const chunks: T[][] = [];
for (let i = 0; i < arr.length; i += size) {
chunks.push(arr.slice(i, i + size));
}
return chunks;
}
for (const chunk of chunkArray(items, 1000)) {
await client.dataset(dsId).pushItems(chunk);
}Step 3: Queue-based rate limiting for custom calls
When you make raw API calls outside the client's retry path (custom endpoints,
fan-out reads), gate them through a p-queue so you never exceed the per-resource
ceiling. intervalCap bounds requests per interval; concurrency bounds parallel
in-flight calls.
import PQueue from 'p-queue';
// 50 requests per second with max 10 concurrent
const apiQueue = new PQueue({
concurrency: 10,
interval: 1000,
intervalCap: 50,
});
// All API calls go through the queue
async function rateLimitedCall<T>(fn: () => Promise<T>): Promise<T> {
return apiQueue.add(fn) as Promise<T>;
}
// Usage
const results = await Promise.all(
actorIds.map(id =>
rateLimitedCall(() => client.actor(id).get())
)
);Step 4: Stagger Actor starts
Launching many runs at once hits the runs endpoint's 429. Insert a small delay between starts, then wait for all runs to finish in parallel.
import { sleep } from 'crawlee';
// Start multiple Actor runs with delays to avoid 429 on the runs endpoint
async function staggeredRuns(
actorId: string,
inputs: Record<string, unknown>[],
delayMs = 200,
) {
const runs = [];
for (const input of inputs) {
const run = await client.actor(actorId).start(input);
runs.push(run);
await sleep(delayMs);
}
// Wait for all to finish
const finished = await Promise.all(
runs.map(run => client.run(run.id).waitForFinish())
);
return finished;
}Step 5: Rate limit monitor
Read the X-RateLimit-* headers off responses to warn before you hit the wall and
to decide when to pause. Feed updateFromHeaders from every response, check
shouldPause() before a batch, and sleep getWaitMs() when it returns true.
class ApifyRateLimitMonitor {
private remaining = 60;
private resetAt = Date.now();
private warningThreshold: number;
constructor(warningThreshold = 10) {
this.warningThreshold = warningThreshold;
}
updateFromHeaders(headers: Record<string, string>) {
if (headers['x-ratelimit-remaining']) {
this.remaining = parseInt(headers['x-ratelimit-remaining']);
}
if (headers['x-ratelimit-reset']) {
this.resetAt = parseInt(headers['x-ratelimit-reset']) * 1000;
}
if (this.remaining < this.warningThreshold) {
const waitMs = Math.max(0, this.resetAt - Date.now());
console.warn(
`Rate limit warning: ${this.remaining} requests remaining. ` +
`Resets in ${waitMs}ms.`
);
}
}
shouldPause(): boolean {
return this.remaining <= 1 && Date.now() < this.resetAt;
}
getWaitMs(): number {
return Math.max(0, this.resetAt - Date.now());
}
}Crawlee-level concurrency (target website rate limits)
Separate from Apify API rate limits, you must also respect the target website. The crawler's concurrency knobs cap how hard you hit the site being scraped — a different ceiling from the platform API.
const crawler = new CheerioCrawler({
// Limit concurrent requests to the target site
maxConcurrency: 10, // Max parallel requests
minConcurrency: 1, // Min parallel requests
maxRequestsPerMinute: 120, // Hard cap per minute
// Auto-scale based on system resources
autoscaledPoolOptions: {
desiredConcurrency: 5,
maxConcurrency: 20,
},
// Delay between requests
requestHandlerTimeoutSecs: 30,
});