Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/apify-core-workflow-a

@820dfc8
Run it yourself

Build a complete web scraping Actor with Crawlee and deploy to Apify. Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform. Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesimplementation.md

≈1.1k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Apify Core Workflow A — Full Implementation Walkthrough

This reference holds the complete, copy-ready code for each step of the build-and-deploy workflow. SKILL.md carries the lean skeleton and the high-level sequence; drill in here for the full Actor source, the Dockerfile, and the programmatic result-retrieval code.

Step 2: Build the Actor with Router Pattern

The router pattern separates listing-page handling (enqueue product links + pagination) from product-detail handling (extract structured fields). This keeps each handler small and testable, and lets Crawlee route requests by label.

// src/main.ts
import { Actor } from 'apify';
import { CheerioCrawler, createCheerioRouter, Dataset, log } from 'crawlee';

interface ProductInput {
  startUrls: { url: string }[];
  maxItems?: number;
  proxyConfig?: { useApifyProxy: boolean; groups?: string[] };
}

interface Product {
  url: string;
  name: string;
  price: number | null;
  currency: string;
  description: string;
  imageUrl: string | null;
  inStock: boolean;
  scrapedAt: string;
}

const router = createCheerioRouter();

// LISTING pages — extract product links
router.addDefaultHandler(async ({ request, $, enqueueLinks, log }) => {
  log.info(`Listing page: ${request.url}`);

  await enqueueLinks({
    selector: 'a.product-card',
    label: 'PRODUCT',
  });

  // Handle pagination
  await enqueueLinks({
    selector: 'a.next-page',
    label: 'LISTING',
  });
});

// PRODUCT detail pages — extract structured data
router.addHandler('PRODUCT', async ({ request, $, log }) => {
  log.info(`Product page: ${request.url}`);

  const product: Product = {
    url: request.url,
    name: $('h1.product-title').text().trim(),
    price: parseFloat($('.price').text().replace(/[^0-9.]/g, '')) || null,
    currency: $('.currency').text().trim() || 'USD',
    description: $('div.description').text().trim(),
    imageUrl: $('img.product-image').attr('src') || null,
    inStock: !$('.out-of-stock').length,
    scrapedAt: new Date().toISOString(),
  };

  await Actor.pushData(product);
});

// Entry point
await Actor.main(async () => {
  const input = await Actor.getInput<ProductInput>();
  if (!input?.startUrls?.length) throw new Error('startUrls required');

  const proxyConfiguration = input.proxyConfig?.useApifyProxy
    ? await Actor.createProxyConfiguration({
        groups: input.proxyConfig.groups,
      })
    : undefined;

  const crawler = new CheerioCrawler({
    requestHandler: router,
    proxyConfiguration,
    maxRequestsPerCrawl: input.maxItems ?? 100,
    maxConcurrency: 10,
    requestHandlerTimeoutSecs: 60,

    async failedRequestHandler({ request }, error) {
      log.error(`Failed: ${request.url} — ${error.message}`);
      await Actor.pushData({
        url: request.url,
        error: error.message,
        '#isFailed': true,
      });
    },
  });

  await crawler.run(input.startUrls.map(s => s.url));

  // Save run summary to key-value store
  const dataset = await Dataset.open();
  const info = await dataset.getInfo();
  await Actor.setValue('SUMMARY', {
    itemCount: info?.itemCount ?? 0,
    finishedAt: new Date().toISOString(),
    startUrls: input.startUrls.map(s => s.url),
  });

  log.info(`Done. Scraped ${info?.itemCount ?? 0} products.`);
});

Step 3: Configure Dockerfile

A two-stage build compiles TypeScript in the builder stage, then copies only the built dist/ into a lean runtime image with production-only dependencies.

# .actor/Dockerfile
FROM apify/actor-node:20 AS builder
COPY package*.json ./
RUN npm ci --include=dev --audit=false
COPY . .
RUN npm run build

FROM apify/actor-node:20
COPY package*.json ./
RUN npm ci --omit=dev --audit=false
COPY --from=builder /usr/src/app/dist ./dist
COPY .actor .actor
CMD ["npm", "start"]

Step 6: Retrieve Results Programmatically

After deployment, call the Actor and pull its dataset from any client using the apify-client SDK. The token comes from process.env.APIFY_TOKEN (never hard-code it).

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

// Run the deployed Actor
const run = await client.actor('username/my-actor').call({
  startUrls: [{ url: 'https://target-store.com/products' }],
  maxItems: 500,
});

// Get results
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Scraped ${items.length} products`);

// Download as CSV
const csv = await client.dataset(run.defaultDatasetId).downloadItems('csv');

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill provides a standard workflow for developing and deploying web scrapers using Crawlee and the Apify platform. It correctly handles sensitive API tokens by recommending environment variables. There is an inherent risk of indirect prompt injection from external web content being scraped, which is common for this type of tool.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: MEDIUM · 1 issue

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago
What it can do
Reads files Edits files Runs commands
version
1.5.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
[
  "saas",
  "scraping",
  "automation",
  "apify"
]
All 7 allowed tools
ReadWriteEditBash(npm:*)Bash(npx:*)Bash(apify:*)Grep
Other metadata
compatibility
Designed for Claude Code

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a