Index of every page on this site: llms.txt
All skills
corey-alix avatar

/low-memory-deployment

@e4fbc1d
Run it yourself

Guides memory-safe implementation patterns for the DWP Hours Tracker deployed on a 512MB DigitalOcean droplet.

  • 1 file
  • 5.9 KB
  • Updated 9 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/corey-alix/dwp-hours/low-memory-deployment

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈33 tokens for metadata: the name and description. ≈1.5k when used: this file.

Description uses 1.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈134 characters, or 1.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Low-Memory Deployment Skill

Description

The DWP Hours Tracker production server runs on a DigitalOcean droplet with 457MB total RAM, no swap. Node.js gets ~200MB usable after OS and PM2 overhead. This skill captures hard-won patterns for avoiding OOM kills.

Trigger

Activate this skill when:

  • Implementing features that process large files (Excel, CSV, images)
  • Adding middleware or libraries that buffer request/response bodies
  • Changing PM2 or server configuration
  • Debugging SIGKILL crashes on the production server
  • Working on bulk data operations (imports, exports, report generation)

Server Environment

Resource Value
Total RAM 457 MB
Swap None
Available to Node.js ~200 MB
PM2 mode Cluster (single instance)
OS Ubuntu on DigitalOcean
Node.js heap default ~256 MB (V8 default for <2GB systems)

Response Pattern

1. Identify Memory Hotspots

Before implementing, estimate peak memory usage:

  • File uploads: multer memoryStorage() holds entire file in RAM
  • Library parsing: ExcelJS readFile() / load(buffer) expands compressed xlsx to full object model (~10–50x file size)
  • Database: sql.js holds entire SQLite DB in memory
  • Accumulated objects: Large result arrays, logging buffers

2. Apply Streaming Patterns

Process data incrementally instead of loading everything at once:

  • Use multer.diskStorage() instead of memoryStorage() — write uploads to /tmp, clean up in finally block
  • Use ExcelJS WorkbookReader streaming API instead of readFile() (see exceljs skill)
  • Process items one at a time, releasing references between iterations
  • For database bulk operations, disable autoSave and persist once at the end

3. Clean Up Aggressively

  • Delete temp files in finally blocks
  • Null out large objects after use
  • Release worksheet references after processing each sheet
  • Use workbook.removeWorksheet(ws.id) if using non-streaming ExcelJS

Proven Patterns

File Upload: Disk Instead of Memory

// BAD — holds entire file in Node.js memory
const upload = multer({ storage: multer.memoryStorage() });

// GOOD — writes to disk, stays off the heap
const upload = multer({
  storage: multer.diskStorage({
    destination: "/tmp",
    filename: (_req, file, cb) => {
      cb(null, `upload-${Date.now()}-${file.originalname}`);
    },
  }),
  limits: { fileSize: 10 * 1024 * 1024 },
});

// Always clean up temp files
try {
  await processFile(req.file.path);
  res.json({ success: true });
} finally {
  fs.unlink(req.file.path, () => {});
}

Excel: Stream One Sheet at a Time

// BAD — loads all 68 sheets into memory at once (~200MB+)
const workbook = new ExcelJS.Workbook();
await workbook.xlsx.readFile(filePath);

// GOOD — streams one sheet at a time (~5MB peak per sheet)
const reader = new ExcelJS.stream.xlsx.WorkbookReader(filePath, {
  worksheets: "emit",
  sharedStrings: "cache",
  styles: "cache",
  hyperlinks: "cache",
});

for await (const wsReader of reader) {
  const ws = await materialiseWorksheet(wsReader); // one sheet in memory
  processSheet(ws);
  // ws is GC'd on next iteration
}

Bulk DB Operations: Single Save

// BAD — sql.js autoSave writes entire DB to disk on every insert
for (const item of items) {
  await repo.save(item); // triggers disk write each time
}

// GOOD — disable autoSave, batch all writes, save once
const driver = dataSource.driver as any;
const originalAutoSave = driver.options?.autoSave;
driver.options.autoSave = false;

try {
  for (const item of items) {
    await repo.save(item); // in-memory only
  }
} finally {
  driver.options.autoSave = originalAutoSave;
  await driver.save(); // single disk write
}

Debugging OOM on Production

Symptoms

  • PM2 shows increasing restart count (↺ column)
  • Process exits with SIGKILL (not SIGINT or SIGTERM)
  • No error in application logs — process killed mid-operation
  • dmesg may show OOM killer messages (requires root access)

Diagnosis Steps

  1. ssh deploy@server "pm2 status" — check restart count and uptime
  2. ssh deploy@server "pm2 logs --nostream --lines 100" — find last operation before crash
  3. ssh deploy@server "free -m" — check available memory
  4. Look for the pattern: operation starts → no completion log → server restarts

Common Culprits

Operation Memory Risk Fix
Excel import (68 sheets) ~200MB+ with readFile() Use streaming WorkbookReader
File upload File size × 1 Use diskStorage()
Report generation Depends on data volume Stream output, limit batch sizes
Multiple concurrent requests Additive Queue heavy operations

PM2 Configuration Notes

  • The server runs in cluster mode with a single instance
  • Each cluster worker is a full Node.js process
  • On 512MB, never run more than 1 instance
  • Consider --max-memory-restart 200M as a safety net

Additional Context

  • This skill complements the exceljs skill (streaming patterns) and sql-js-database-assistant skill (bulk operations)
  • The production server has no swap partition — when memory is exhausted, the OOM killer acts immediately with SIGKILL
  • Always test memory-intensive operations with realistic data volumes before deploying

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at e4fbc1d. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub last month.

Dormantupdated 9 months ago

README badge

README badge for corey-alix/dwp-hours/low-memory-deployment