Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/apify-reference-architecture

@820dfc8
Run it yourself

Production-grade architecture patterns for Apify-powered applications. Use when designing scraping infrastructure, building multi-Actor pipelines, or integrating Apify into a larger system architecture. Trigger with "apify architecture", "apify best practices", "apify project structure", "scraping architecture", "apify system design".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-reference-architecture

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈92 tokens for metadata: the name and description. ≈1.4k when used: this file. ≈3k more on demand in 2 files.

Description uses 4.6% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈367 characters, or 4.6%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Apify Reference Architecture

Overview

Production-ready architecture patterns for applications built on Apify. Three patterns scale from a single scraper to a full-stack integration:

  1. Standalone Actor — one scraper deployed to the Apify platform.
  2. Multi-Actor Pipeline — a discover → scrape → transform chain of Actors.
  3. Full-Stack Integration — an application using Apify as a data source behind a service layer.

This skill helps you choose the right pattern, lay out the directory structure, and wire the skeleton code. Full directory trees, diagrams, and code for every pattern live in references/architecture-patterns.md; the service layer, configuration loader, and health check live in references/implementation.md.

Prerequisites

  • Runtime: Node.js >=18, TypeScript, and the Apify CLI (npm i -g apify-cli).
  • Packages: apify + crawlee (inside an Actor), apify-client (calling Actors from an app), zod (input validation).
  • Auth: an Apify API token. Set APIFY_TOKEN in the environment; the Apify SDK and apify-client read it automatically (or pass it explicitly to new ApifyClient({ token })). Never hardcode the token — inject it via env var and validate at startup.
  • Access: Read and Grep the target repository so you can match the recommended layout against the code already on disk before proposing changes.

Instructions

  1. Pick the pattern. One scraper → Pattern 1. A staged workflow that discovers, scrapes, then cleans → Pattern 2. An app that consumes scraped data → Pattern 3.
  2. Grep the existing repo for apify, apify-client, and Actor.main to see what is already wired, so you extend rather than duplicate structure.
  3. Lay out the directory from the pattern's tree in references/architecture-patterns.md. Keep routing, extraction, and validation in separate modules.
  4. Add typed input validation with zod (see src/types.ts in the reference) so bad input fails fast at the Actor boundary instead of mid-crawl.
  5. Isolate every Apify call behind a service layer (Pattern 3) using the ApifyService class in references/implementation.md — the rest of the app never imports apify-client directly.
  6. Load configuration once at startup via loadConfig() and layer per-environment overrides on a single base object; validate required env vars before serving traffic.
  7. Expose an Apify health check so a bad token or platform outage surfaces before a user-facing scrape fails.

Output

Applying this skill produces an architecture, not a running command. Expect:

  • A recommended directory layout for the chosen pattern.
  • Skeleton TypeScript modules (main.ts, types.ts, service layer, config loader, health check).
  • A per-environment configuration strategy and an Apify health signal.
  • For pipelines, an orchestrator that reports per-stage item counts and total USD cost, e.g.:
=== Pipeline Summary ===
Discovered: 320 URLs
Scraped:    298 items
Clean:      271 items
Total cost: $0.4120

Error Handling

Issue Cause Solution
Circular dependencies Service imports service Use dependency injection
Missing config Env var not set Validate at startup with loadConfig()
Pipeline stage failure Actor crash mid-pipeline Add retry logic per stage
State management Tracking run status Use webhook handler + database
Run not ready error Fetching results before SUCCEEDED Poll getRunStatus or use a completion webhook

Examples

Standalone Actor entry point — the minimal skeleton; full file in references/architecture-patterns.md:

// src/main.ts
import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';
import { router } from './routes/listing';
import { validateInput, ScraperInput } from './types';

await Actor.main(async () => {
  const input = validateInput(await Actor.getInput<ScraperInput>());
  const crawler = new CheerioCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: input.maxItems ?? 100,
    maxConcurrency: input.concurrency ?? 10,
  });
  await crawler.run(input.startUrls.map(s => s.url));
});

Calling an Actor from an app — via the service layer:

const apify = new ApifyService(process.env.APIFY_TOKEN!);
const { runId } = await apify.startScrape(['https://example.com']);
const results = await apify.getResults<ProductOutput>(runId);

More: the multi-stage pipeline orchestrator and the full ApifyService class are in references/architecture-patterns.md and references/implementation.md. For multi-environment setup, see the companion apify-deploy-integration skill.

Resources

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill provides production-grade architectural templates and best practices for building web scraping applications using the Apify platform. It emphasizes security best practices, such as managing API tokens via environment variables rather than hardcoding them, and implementing robust input validation using the Zod library. The code provided consists of clean TypeScript templates for standalone scrapers, multi-actor pipelines, and full-stack integrations. No security issues or malicious patterns were identified.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 4 months ago
What it can do
Reads files
version
1.5.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
[
  "saas",
  "scraping",
  "automation",
  "apify"
]
All 2 allowed tools
ReadGrep
Other metadata
compatibility
Designed for Claude Code

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/apify-reference-architecture