Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/apify-core-workflow-b

@820dfc8
Run it yourself

Manage Apify datasets, key-value stores, and request queues programmatically, and orchestrate multi-Actor pipelines. Use when you need to read or write Apify datasets, export scraped data to CSV/JSON/XLSX, store config or binary artifacts in a key-value store, manage a resumable request queue, chain Actors into a scrape → transform → export pipeline, or monitor Actor run status and cost. Trigger with "apify dataset", "apify key-value store", "apify storage", "export apify data", "apify pipeline", "apify request queue".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈138 tokens for metadata: the name and description. ≈1.6k when used: this file. ≈1.6k more on demand in 2 files.

Description uses 6.9% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈550 characters, or 6.9%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Apify Core Workflow B — Storage & Pipelines

Overview

Manage Apify's three storage types (datasets, key-value stores, request queues) and orchestrate multi-Actor pipelines using the apify-client JS SDK. Covers CRUD operations, data export, automatic pagination, and chaining Actors together (scrape → transform → export).

This SKILL.md gives you the high-level workflow plus the essential first example for each storage type. Drill into the reference files for the complete, copy-ready code:

Prerequisites

  • Node.js with apify-client installed (npm install apify-client).
  • An Apify account token exported as APIFY_TOKEN (see Authentication below).
  • Familiarity with apify-core-workflow-a (Actor invocation and run lifecycle), since pipelines chain Actor runs and read their default storages.

Authentication

All operations authenticate with an Apify API token. Never hard-code it — read it from the environment and construct the client once:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

Generate a token at Apify Console → Settings → Integrations, then export it (export APIFY_TOKEN=apify_api_...) or load it from your secrets manager.

Storage Types at a Glance

Storage Best For Analogy Retention
Dataset Lists of similar items (products, pages) Append-only table 7 days (unnamed)
Key-Value Store Config, screenshots, summaries, any file S3 bucket 7 days (unnamed)
Request Queue URLs to crawl (managed by Crawlee) Job queue 7 days (unnamed)

Named storages persist indefinitely. Unnamed (default run) storages expire after 7 days.

Instructions

Pick the storage type you need, use the skeleton below to get started, then open the linked reference for the full operation set.

Datasets — append-only item lists

getOrCreate a named dataset, push items, and list them (pagination is manual):

const dataset = await client.datasets().getOrCreate('product-catalog');
const dsClient = client.dataset(dataset.id);
await dsClient.pushItems([{ sku: 'ABC123', name: 'Widget', price: 9.99 }]);
const { items, total } = await dsClient.listItems({ limit: 100, offset: 0 });

Full auto-pagination loop, CSV/JSON/XLSX export, and field filtering: storage-operations.md, Step 1.

Key-value stores — config, files, and Actor OUTPUT

Store JSON or binary records by key, then retrieve them:

const store = await client.keyValueStores().getOrCreate('scraper-config');
const kvClient = client.keyValueStore(store.id);
await kvClient.setRecord({ key: 'settings', value: { maxRetries: 3 }, contentType: 'application/json' });
const record = await kvClient.getRecord('settings');

Binary records, key listing, and reading a run's default OUTPUT: storage-operations.md, Step 2.

Request queues — resumable crawl URLs

Create a named queue and add requests (deduplicated by uniqueKey):

const queue = await client.requestQueues().getOrCreate('my-crawl-queue');
const rqClient = client.requestQueue(queue.id);
await rqClient.addRequest({ url: 'https://example.com/page1', uniqueKey: 'page1' });

Batch adds and queue stats: storage-operations.md, Step 3.

Multi-Actor pipelines & monitoring

Chain Actors (scrape → transform → export) and monitor run status and cost. Full runPipeline() function and run-monitoring code: pipelines.md.

Output

  • Datasets return { items, total, count, offset, limit } from listItems(); downloadItems(format) returns a Buffer in csv / json / xlsx.
  • Key-value stores return { key, value, contentType } from getRecord() and { items } (each { key, size }) from listKeys().
  • Request queues return { pendingRequestCount, handledRequestCount, ... } from get().
  • Pipelines return the named export dataset id; run monitoring yields { status, statusMessage, stats, usage, usageTotalUsd } per run.

Error Handling

Error Cause Solution
Dataset not found Expired (unnamed, >7 days) Use named datasets for persistence
Record too large KV store 9MB record limit Split into multiple records
Push failed Dataset items >9MB batch Push in smaller batches
Request already exists Duplicate uniqueKey Expected behavior, queue deduplicates

Examples

Export a named dataset to CSV — get the client, download the buffer, write it:

const csvBuffer = await client.dataset('product-catalog').downloadItems('csv');
require('fs').writeFileSync('products.csv', csvBuffer);

Read an Actor run's OUTPUT record — after a run completes:

const run = await client.actor('apify/web-scraper').call(input);
const output = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');

Longer end-to-end examples — the full pagination loop, binary record storage, and the three-stage runPipeline() — live in the reference files: storage-operations.md and pipelines.md.

Resources

Next Steps

For common errors and their fixes across the Apify pack, see the apify-common-errors skill. For Actor invocation and run lifecycle basics that pipelines build on, see apify-core-workflow-a.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill provides standard documentation and code references for managing Apify storage and multi-Actor pipelines using the official apify-client SDK. It correctly follows security best practices by advising the use of environment variables for API tokens rather than hardcoding credentials. While the skill facilitates the ingestion of external data from web sources, this functionality is central to its purpose as a scraping management tool.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: MEDIUM · 1 issue

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 20 hours ago.

Activeupdated 4 months ago
What it can do
Reads files Edits files Runs commands
version
1.5.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
[
  "saas",
  "scraping",
  "automation",
  "apify"
]
All 6 allowed tools
ReadWriteEditBash(npm:*)Bash(npx:*)Grep
Other metadata
compatibility
Designed for Claude Code

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/apify-core-workflow-b