Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/apify-core-workflow-a

@820dfc8
Run it yourself

Build a complete web scraping Actor with Crawlee and deploy to Apify. Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform. Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈107 tokens for metadata: the name and description. ≈1.6k when used: this file. ≈1.6k more on demand in 2 files.

Description uses 5.3% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈428 characters, or 5.3%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Apify Core Workflow A — Build & Deploy a Scraper

Overview

End-to-end workflow: define input schema, build a Crawlee-based Actor, extract structured data, store results in datasets, test locally, and deploy to Apify platform. This is the primary money-path workflow for Apify.

Prerequisites

  • npm install apify crawlee in your project
  • npm install -g apify-cli and apify login completed
  • For programmatic retrieval (Step 6), an API token in APIFY_TOKEN — read it from the environment (process.env.APIFY_TOKEN), never hard-code it
  • Familiarity with apify-sdk-patterns

Instructions

Step 1: Define Input Schema

Create .actor/INPUT_SCHEMA.json:

{
  "title": "E-Commerce Scraper",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrls": {
      "title": "Start URLs",
      "type": "array",
      "description": "Product listing page URLs to scrape",
      "editor": "requestListSources",
      "prefill": [{ "url": "https://example-store.com/products" }]
    },
    "maxItems": {
      "title": "Max items",
      "type": "integer",
      "description": "Maximum number of products to scrape",
      "default": 100,
      "minimum": 1,
      "maximum": 10000
    },
    "proxyConfig": {
      "title": "Proxy configuration",
      "type": "object",
      "description": "Select proxy to use",
      "editor": "proxy",
      "default": { "useApifyProxy": true }
    }
  },
  "required": ["startUrls"]
}

Step 2: Build the Actor with Router Pattern

Use a Crawlee router that splits handling by page type: the default handler enqueues product links + pagination from listing pages, and a PRODUCT-labeled handler extracts structured fields from detail pages. The entry point wires proxy config, concurrency, a failed-request handler, and a run summary into the key-value store. Skeleton:

// src/main.ts
import { Actor } from 'apify';
import { CheerioCrawler, createCheerioRouter, Dataset, log } from 'crawlee';

const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks }) => {
  await enqueueLinks({ selector: 'a.product-card', label: 'PRODUCT' });
  await enqueueLinks({ selector: 'a.next-page', label: 'LISTING' });
});
router.addHandler('PRODUCT', async ({ request, $ }) => {
  await Actor.pushData({ url: request.url, name: $('h1.product-title').text().trim() });
});

await Actor.main(async () => {
  const input = await Actor.getInput();
  const crawler = new CheerioCrawler({ requestHandler: router, maxRequestsPerCrawl: input?.maxItems ?? 100 });
  await crawler.run(input.startUrls.map(s => s.url));
});

The full typed Actor — Product/ProductInput interfaces, proxy configuration, failedRequestHandler, and the SUMMARY key-value write — is in implementation.md, Step 2.

Step 3: Configure Dockerfile

Use the apify/actor-node:20 base with a two-stage build (compile TypeScript in a builder stage, ship only dist/ + production deps). Full Dockerfile: implementation.md, Step 3.

Step 4: Test Locally

# Create test input
mkdir -p storage/key_value_stores/default
echo '{"startUrls":[{"url":"https://example.com"}],"maxItems":5}' \
  > storage/key_value_stores/default/INPUT.json

# Run locally
apify run

# Check results
ls storage/datasets/default/
cat storage/key_value_stores/default/SUMMARY.json

Step 5: Deploy to Apify Platform

# Push to Apify (creates Actor if it doesn't exist)
apify push

# Or push to a specific Actor
apify push username/my-actor

# Run on platform
apify actors call username/my-actor

Step 6: Retrieve Results Programmatically

From any client, use the apify-client SDK to call the deployed Actor, list its dataset items, and download results (JSON/CSV). The token comes from process.env.APIFY_TOKEN — never hard-code it. Full retrieval code: implementation.md, Step 6.

Output

  • Deployable Actor with typed input schema
  • Router-based crawler handling listing + detail pages
  • Structured product data in default dataset
  • Run summary in default key-value store
  • Failed requests tracked with error messages

Error Handling

Error Cause Solution
Actor build failed Dockerfile/deps issue Check build logs on platform
Selector returns empty Page structure changed Update CSS selectors
maxRequestsPerCrawl hit Too many pages enqueued Increase limit or filter URLs
Proxy errors Anti-bot blocking Switch to residential proxy
TIMED-OUT status Actor exceeded timeout Increase timeout or reduce scope

Examples

A quick example — seed a local input, run the Actor, and check results:

mkdir -p storage/key_value_stores/default
echo '{"startUrls":[{"url":"https://example-store.com/products"}],"maxItems":5}' \
  > storage/key_value_stores/default/INPUT.json
apify run
cat storage/key_value_stores/default/SUMMARY.json

Three fuller worked scenarios live in examples.md:

  • Scrape a catalog locally, then deploy — the full seed → apify run → inspect → apify push loop, with the expected SUMMARY.json output.
  • Run the deployed Actor and export CSV — call the Actor via apify-client and download the dataset as CSV.
  • Route through residential proxy — pass a proxyConfig group at run time to get past anti-bot blocking.

Resources

Next Steps

Once your Actor is deployed and producing data, move on to dataset and key-value store management — pagination over large datasets, deduplication, exporting to external stores, and scheduling recurring runs — covered in apify-core-workflow-b.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    This skill provides a standard workflow for developing and deploying web scrapers using Crawlee and the Apify platform. It correctly handles sensitive API tokens by recommending environment variables. There is an inherent risk of indirect prompt injection from external web content being scraped, which is common for this type of tool.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: MEDIUM · 1 issue

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 4 months ago
What it can do
Reads files Edits files Runs commands
version
1.5.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
[
  "saas",
  "scraping",
  "automation",
  "apify"
]
All 7 allowed tools
ReadWriteEditBash(npm:*)Bash(npx:*)Bash(apify:*)Grep
Other metadata
compatibility
Designed for Claude Code

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a