Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/apify-reference-architecture

@820dfc8
Run it yourself

Production-grade architecture patterns for Apify-powered applications. Use when designing scraping infrastructure, building multi-Actor pipelines, or integrating Apify into a larger system architecture. Trigger with "apify architecture", "apify best practices", "apify project structure", "scraping architecture", "apify system design".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-reference-architecture

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesimplementation.md

≈850 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Apify Integration — Implementation Reference

Service-layer, configuration, and health-check building blocks for the full-stack integration pattern (Pattern 3 in architecture-patterns.md). These isolate every Apify call behind a typed boundary so the rest of the application never touches apify-client directly.

Service Layer

A single ApifyService class wraps apify-client and exposes intent-level methods (startScrape, getRunStatus, getResults, checkHealth). Dependency-inject the token so the class stays testable and environment-agnostic.

// src/services/apify-service.ts
import { ApifyClient } from 'apify-client';

export class ApifyService {
  private client: ApifyClient;

  constructor(token: string) {
    this.client = new ApifyClient({ token });
  }

  async startScrape(urls: string[]): Promise<{ runId: string }> {
    const run = await this.client.actor('username/scraper').start({
      startUrls: urls.map(url => ({ url })),
    });
    return { runId: run.id };
  }

  async getRunStatus(runId: string): Promise<{
    status: string;
    progress?: { finished: number; failed: number };
  }> {
    const run = await this.client.run(runId).get();
    return {
      status: run.status,
      progress: {
        finished: run.stats?.requestsFinished ?? 0,
        failed: run.stats?.requestsFailed ?? 0,
      },
    };
  }

  async getResults<T>(runId: string): Promise<T[]> {
    const run = await this.client.run(runId).get();
    if (run.status !== 'SUCCEEDED') {
      throw new Error(`Run not ready: ${run.status}`);
    }
    const { items } = await this.client
      .dataset(run.defaultDatasetId)
      .listItems();
    return items as T[];
  }

  async checkHealth(): Promise<boolean> {
    try {
      const user = await this.client.user().get();
      return !!user.username;
    } catch {
      return false;
    }
  }
}

Configuration Management

Load configuration once at startup, validate that required env vars are present, and layer per-environment overrides on top of a single base object. Fail fast here rather than deep inside a request handler.

// src/config/apify.ts
interface ApifyConfig {
  token: string;
  actorId: string;
  defaultMemory: number;
  defaultTimeout: number;
  webhookUrl?: string;
}

export function loadConfig(): ApifyConfig {
  const env = process.env.NODE_ENV || 'development';

  const base: ApifyConfig = {
    token: process.env.APIFY_TOKEN!,
    actorId: process.env.APIFY_ACTOR_ID!,
    defaultMemory: 1024,
    defaultTimeout: 3600,
  };

  const overrides: Record<string, Partial<ApifyConfig>> = {
    development: { defaultMemory: 256, defaultTimeout: 300 },
    staging: { defaultMemory: 512 },
    production: { webhookUrl: process.env.APIFY_WEBHOOK_URL },
  };

  return { ...base, ...overrides[env] };
}

Health Check

Expose Apify reachability as a first-class health signal so orchestrators and dashboards can detect a bad token or platform outage before a user-facing scrape fails.

// src/health.ts
export async function healthCheck(apifyService: ApifyService) {
  const start = Date.now();
  const healthy = await apifyService.checkHealth();

  return {
    service: 'apify',
    status: healthy ? 'healthy' : 'unhealthy',
    latencyMs: Date.now() - start,
    timestamp: new Date().toISOString(),
  };
}

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill provides production-grade architectural templates and best practices for building web scraping applications using the Apify platform. It emphasizes security best practices, such as managing API tokens via environment variables rather than hardcoding them, and implementing robust input validation using the Zod library. The code provided consists of clean TypeScript templates for standalone scrapers, multi-actor pipelines, and full-stack integrations. No security issues or malicious patterns were identified.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago
What it can do
Reads files
version
1.5.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
[
  "saas",
  "scraping",
  "automation",
  "apify"
]
All 2 allowed tools
ReadGrep
Other metadata
compatibility
Designed for Claude Code

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/apify-reference-architecture