---
name: apify-core-workflow-a
description: 'Build a complete web scraping Actor with Crawlee and deploy to Apify.

  Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform.

  Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".'
allowed-tools: Read, Write, Edit, Bash(npm:*), Bash(npx:*), Bash(apify:*), Grep
version: 1.5.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
tags:
  - saas
  - scraping
  - automation
  - apify
compatibility: Designed for Claude Code
title: apify-core-workflow-a
canonical_url: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a
last_updated: 2026-10-07T06:06:28.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [references/examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/examples.md), [references/implementation.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/implementation.md).
>
> If the user asked to install this Skill, run `npx skilld install jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/apify-core-workflow-a`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/apify-core-workflow-a`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/apify-core-workflow-a --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/apify-core-workflow-a --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Apify Core Workflow A — Build & Deploy a Scraper

## Overview

End-to-end workflow: define input schema, build a Crawlee-based Actor, extract structured data, store results in datasets, test locally, and deploy to Apify platform. This is the primary money-path workflow for Apify.

## Prerequisites

- `npm install apify crawlee` in your project
- `npm install -g apify-cli` and `apify login` completed
- For programmatic retrieval (Step 6), an API token in `APIFY_TOKEN` — read it from
  the environment (`process.env.APIFY_TOKEN`), never hard-code it
- Familiarity with `apify-sdk-patterns`

## Instructions

### Step 1: Define Input Schema

Create `.actor/INPUT_SCHEMA.json`:

```json
{
  "title": "E-Commerce Scraper",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrls": {
      "title": "Start URLs",
      "type": "array",
      "description": "Product listing page URLs to scrape",
      "editor": "requestListSources",
      "prefill": [{ "url": "https://example-store.com/products" }]
    },
    "maxItems": {
      "title": "Max items",
      "type": "integer",
      "description": "Maximum number of products to scrape",
      "default": 100,
      "minimum": 1,
      "maximum": 10000
    },
    "proxyConfig": {
      "title": "Proxy configuration",
      "type": "object",
      "description": "Select proxy to use",
      "editor": "proxy",
      "default": { "useApifyProxy": true }
    }
  },
  "required": ["startUrls"]
}
```

### Step 2: Build the Actor with Router Pattern

Use a Crawlee router that splits handling by page type: the default handler
enqueues product links + pagination from listing pages, and a `PRODUCT`-labeled
handler extracts structured fields from detail pages. The entry point wires proxy
config, concurrency, a failed-request handler, and a run summary into the key-value
store. Skeleton:

```typescript
// src/main.ts
import { Actor } from 'apify';
import { CheerioCrawler, createCheerioRouter, Dataset, log } from 'crawlee';

const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks }) => {
  await enqueueLinks({ selector: 'a.product-card', label: 'PRODUCT' });
  await enqueueLinks({ selector: 'a.next-page', label: 'LISTING' });
});
router.addHandler('PRODUCT', async ({ request, $ }) => {
  await Actor.pushData({ url: request.url, name: $('h1.product-title').text().trim() });
});

await Actor.main(async () => {
  const input = await Actor.getInput();
  const crawler = new CheerioCrawler({ requestHandler: router, maxRequestsPerCrawl: input?.maxItems ?? 100 });
  await crawler.run(input.startUrls.map(s => s.url));
});
```

The full typed Actor — `Product`/`ProductInput` interfaces, proxy configuration,
`failedRequestHandler`, and the `SUMMARY` key-value write — is in
[implementation.md, Step 2](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/implementation.md).

### Step 3: Configure Dockerfile

Use the `apify/actor-node:20` base with a two-stage build (compile TypeScript in a
`builder` stage, ship only `dist/` + production deps). Full Dockerfile:
[implementation.md, Step 3](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/implementation.md).

### Step 4: Test Locally

```bash
# Create test input
mkdir -p storage/key_value_stores/default
echo '{"startUrls":[{"url":"https://example.com"}],"maxItems":5}' \
  > storage/key_value_stores/default/INPUT.json

# Run locally
apify run

# Check results
ls storage/datasets/default/
cat storage/key_value_stores/default/SUMMARY.json
```

### Step 5: Deploy to Apify Platform

```bash
# Push to Apify (creates Actor if it doesn't exist)
apify push

# Or push to a specific Actor
apify push username/my-actor

# Run on platform
apify actors call username/my-actor
```

### Step 6: Retrieve Results Programmatically

From any client, use the `apify-client` SDK to call the deployed Actor, list its
dataset items, and download results (JSON/CSV). The token comes from
`process.env.APIFY_TOKEN` — never hard-code it. Full retrieval code:
[implementation.md, Step 6](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/implementation.md).

## Output

- Deployable Actor with typed input schema
- Router-based crawler handling listing + detail pages
- Structured product data in default dataset
- Run summary in default key-value store
- Failed requests tracked with error messages

## Error Handling

| Error | Cause | Solution |
|-------|-------|----------|
| `Actor build failed` | Dockerfile/deps issue | Check build logs on platform |
| Selector returns empty | Page structure changed | Update CSS selectors |
| `maxRequestsPerCrawl` hit | Too many pages enqueued | Increase limit or filter URLs |
| Proxy errors | Anti-bot blocking | Switch to residential proxy |
| `TIMED-OUT` status | Actor exceeded timeout | Increase timeout or reduce scope |

## Examples

A quick example — seed a local input, run the Actor, and check results:

```bash
mkdir -p storage/key_value_stores/default
echo '{"startUrls":[{"url":"https://example-store.com/products"}],"maxItems":5}' \
  > storage/key_value_stores/default/INPUT.json
apify run
cat storage/key_value_stores/default/SUMMARY.json
```

Three fuller worked scenarios live in [examples.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/examples.md):

- **Scrape a catalog locally, then deploy** — the full seed → `apify run` →
  inspect → `apify push` loop, with the expected `SUMMARY.json` output.
- **Run the deployed Actor and export CSV** — call the Actor via `apify-client`
  and download the dataset as CSV.
- **Route through residential proxy** — pass a `proxyConfig` group at run time to
  get past anti-bot blocking.

## Resources

- [Crawlee Quick Start](https://crawlee.dev/js/docs/quick-start)
- [Actor Deployment](https://docs.apify.com/platform/actors/development/deployment)
- [Input Schema Spec](https://docs.apify.com/platform/actors/development/actor-definition/input-schema)
- [Full implementation walkthrough](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/implementation.md) — complete Actor source, Dockerfile, and retrieval code
- [Worked examples](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-a/references/examples.md) — three end-to-end run scenarios

## Next Steps

Once your Actor is deployed and producing data, move on to dataset and key-value
store management — pagination over large datasets, deduplication, exporting to
external stores, and scheduling recurring runs — covered in `apify-core-workflow-b`.
