---
name: apify-core-workflow-b
description: |-
  Manage Apify datasets, key-value stores, and request queues programmatically,
  and orchestrate multi-Actor pipelines.

  Use when you need to read or write Apify datasets, export scraped data to
  CSV/JSON/XLSX, store config or binary artifacts in a key-value store, manage a
  resumable request queue, chain Actors into a scrape → transform → export
  pipeline, or monitor Actor run status and cost.

  Trigger with "apify dataset", "apify key-value store", "apify storage",
  "export apify data", "apify pipeline", "apify request queue".
allowed-tools: Read, Write, Edit, Bash(npm:*), Bash(npx:*), Grep
version: 1.5.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
tags:
  - saas
  - scraping
  - automation
  - apify
compatibility: Designed for Claude Code
title: apify-core-workflow-b
canonical_url: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b
last_updated: 2026-10-07T06:06:28.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> Supporting files, fetch one when the Skill refers to it: [references/pipelines.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/pipelines.md), [references/storage-operations.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/storage-operations.md).
>
> If the user asked to install this Skill, run `npx skilld install jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/apify-core-workflow-b`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/apify-core-workflow-b`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/apify-core-workflow-b --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/apify-core-workflow-b --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Apify Core Workflow B — Storage & Pipelines

## Overview

Manage Apify's three storage types (datasets, key-value stores, request queues)
and orchestrate multi-Actor pipelines using the `apify-client` JS SDK. Covers
CRUD operations, data export, automatic pagination, and chaining Actors
together (scrape → transform → export).

This SKILL.md gives you the high-level workflow plus the essential first example
for each storage type. Drill into the reference files for the complete,
copy-ready code:

- **[Storage operations — full reference](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/storage-operations.md)** —
  every dataset, key-value store, and request queue operation with pagination,
  format export, and binary records.
- **[Pipelines & run monitoring — full reference](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/pipelines.md)** —
  the multi-Actor pipeline function and Actor-run status/cost/abort monitoring.

## Prerequisites

- Node.js with `apify-client` installed (`npm install apify-client`).
- An Apify account token exported as `APIFY_TOKEN` (see Authentication below).
- Familiarity with `apify-core-workflow-a` (Actor invocation and run lifecycle),
  since pipelines chain Actor runs and read their default storages.

## Authentication

All operations authenticate with an Apify API token. Never hard-code it —
read it from the environment and construct the client once:

```typescript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
```

Generate a token at Apify Console → Settings → Integrations, then export it
(`export APIFY_TOKEN=apify_api_...`) or load it from your secrets manager.

## Storage Types at a Glance

| Storage | Best For | Analogy | Retention |
|---------|----------|---------|-----------|
| Dataset | Lists of similar items (products, pages) | Append-only table | 7 days (unnamed) |
| Key-Value Store | Config, screenshots, summaries, any file | S3 bucket | 7 days (unnamed) |
| Request Queue | URLs to crawl (managed by Crawlee) | Job queue | 7 days (unnamed) |

Named storages persist indefinitely. Unnamed (default run) storages expire after 7 days.

## Instructions

Pick the storage type you need, use the skeleton below to get started, then open
the linked reference for the full operation set.

### Datasets — append-only item lists

`getOrCreate` a named dataset, push items, and list them (pagination is manual):

```typescript
const dataset = await client.datasets().getOrCreate('product-catalog');
const dsClient = client.dataset(dataset.id);
await dsClient.pushItems([{ sku: 'ABC123', name: 'Widget', price: 9.99 }]);
const { items, total } = await dsClient.listItems({ limit: 100, offset: 0 });
```

Full auto-pagination loop, CSV/JSON/XLSX export, and field filtering:
**[storage-operations.md, Step 1](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/storage-operations.md)**.

### Key-value stores — config, files, and Actor OUTPUT

Store JSON or binary records by key, then retrieve them:

```typescript
const store = await client.keyValueStores().getOrCreate('scraper-config');
const kvClient = client.keyValueStore(store.id);
await kvClient.setRecord({ key: 'settings', value: { maxRetries: 3 }, contentType: 'application/json' });
const record = await kvClient.getRecord('settings');
```

Binary records, key listing, and reading a run's default `OUTPUT`:
**[storage-operations.md, Step 2](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/storage-operations.md)**.

### Request queues — resumable crawl URLs

Create a named queue and add requests (deduplicated by `uniqueKey`):

```typescript
const queue = await client.requestQueues().getOrCreate('my-crawl-queue');
const rqClient = client.requestQueue(queue.id);
await rqClient.addRequest({ url: 'https://example.com/page1', uniqueKey: 'page1' });
```

Batch adds and queue stats:
**[storage-operations.md, Step 3](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/storage-operations.md)**.

### Multi-Actor pipelines & monitoring

Chain Actors (scrape → transform → export) and monitor run status and cost.
Full `runPipeline()` function and run-monitoring code:
**[pipelines.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/pipelines.md)**.

## Output

- **Datasets** return `{ items, total, count, offset, limit }` from `listItems()`;
  `downloadItems(format)` returns a `Buffer` in `csv` / `json` / `xlsx`.
- **Key-value stores** return `{ key, value, contentType }` from `getRecord()`
  and `{ items }` (each `{ key, size }`) from `listKeys()`.
- **Request queues** return `{ pendingRequestCount, handledRequestCount, ... }`
  from `get()`.
- **Pipelines** return the named export dataset id; run monitoring yields
  `{ status, statusMessage, stats, usage, usageTotalUsd }` per run.

## Error Handling

| Error | Cause | Solution |
|-------|-------|----------|
| `Dataset not found` | Expired (unnamed, >7 days) | Use named datasets for persistence |
| `Record too large` | KV store 9MB record limit | Split into multiple records |
| `Push failed` | Dataset items >9MB batch | Push in smaller batches |
| `Request already exists` | Duplicate uniqueKey | Expected behavior, queue deduplicates |

## Examples

**Export a named dataset to CSV** — get the client, download the buffer, write it:

```typescript
const csvBuffer = await client.dataset('product-catalog').downloadItems('csv');
require('fs').writeFileSync('products.csv', csvBuffer);
```

**Read an Actor run's OUTPUT record** — after a run completes:

```typescript
const run = await client.actor('apify/web-scraper').call(input);
const output = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');
```

Longer end-to-end examples — the full pagination loop, binary record storage,
and the three-stage `runPipeline()` — live in the reference files:
[storage-operations.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/storage-operations.md) and
[pipelines.md](https://skilld.dev/api/skills-raw/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b/references/pipelines.md).

## Resources

- [Dataset Documentation](https://docs.apify.com/platform/storage/dataset)
- [Key-Value Store Documentation](https://docs.apify.com/platform/storage/key-value-store)
- [Request Queue Documentation](https://docs.apify.com/platform/storage/request-queue)
- [JS Client API Reference](https://docs.apify.com/api/client/js/reference)

## Next Steps

For common errors and their fixes across the Apify pack, see the
`apify-common-errors` skill. For Actor invocation and run lifecycle basics that
pipelines build on, see `apify-core-workflow-a`.
