Index of every page on this site: llms.txt
All skills
jimliu avatar

/baoyu-url-to-markdown

@bf1ff07 official
by Jim Liu 宝玉jimliu/baoyu-skills28k stars
3,011
Run it yourself

Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.

Use this Skill: https://skilld.dev/gh/jimliu/baoyu-skills/baoyu-url-to-markdown

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈83 tokens for metadata: the name and description. ≈1.4k when used: this file. ≈1.8k more on demand in 4 files.

Description uses 4.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈333 characters, or 4.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

URL to Markdown

User Input Tools

When this skill prompts the user, follow this tool-selection rule (priority order):

  1. Prefer built-in user-input tools exposed by the current agent runtime — e.g., AskUserQuestion, request_user_input, clarify, ask_user, or any equivalent.
  2. Fallback: if no such tool exists, emit a numbered plain-text message and ask the user to reply with the chosen number/answer for each question.
  3. Batching: if the tool supports multiple questions per call, combine all applicable questions into a single call; if only single-question, ask them one at a time in priority order.

Concrete AskUserQuestion references below are examples — substitute the local equivalent in other runtimes.

CLI Setup

The CLI source is vendored in {baseDir}/scripts/lib; scripts/package.json installs only third-party runtime dependencies.

  1. {baseDir} = this SKILL.md's directory
  2. ${BUN} = bun if installed; otherwise suggest installing Bun
  3. If {baseDir}/scripts/node_modules is missing: ${BUN} install --cwd {baseDir}/scripts
  4. ${READER} = {baseDir}/scripts/baoyu-fetch (substitute everywhere below)

Preferences (EXTEND.md)

First one found wins:

Priority Path Scope
1 .baoyu-skills/baoyu-url-to-markdown/EXTEND.md Project
2 ${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-url-to-markdown/EXTEND.md XDG
3 $HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md User home

Found → read and apply. Not found → BLOCKING first-time setup: ask the questions in references/config/first-time-setup.md (Media, Output, Save — all in one call) before any conversion, write EXTEND.md, confirm "Preferences saved to [path]", then continue. Never create EXTEND.md with silent defaults.

Key Default Values CLI mapping
download_media ask ask (prompt each time) / 1 (always) / 0 (never) 1 → --download-media (requires --output)
default_output_dir empty (= ./url-to-markdown/) path or empty Agent builds --output <dir>/{domain}/{slug}.md; not a direct flag

Value priority: CLI arguments → EXTEND.md → skill defaults.

Usage

${READER} <url>                                          # headless, markdown to stdout
${READER} <url> --output article.md --download-media     # save with local media
${READER} <url> --wait-for interaction --output a.md     # login/CAPTCHA: auto-detect and continue
${READER} <url> --wait-for force --output a.md           # manual: Enter to continue
${READER} <url> --format json --output a.json
${READER} <url> --adapter youtube --output transcript.md

Options

Option Description
<url> URL to fetch
--output <path> Output file path (default: stdout)
--format <type> markdown (default) or json
--json Shorthand for --format json
--adapter <name> Force adapter: x, youtube, hn, or generic (default: auto-detect)
--headless Force headless Chrome
--wait-for <mode> none (default), interaction, or force
--wait-for-interaction Alias for --wait-for interaction
--wait-for-login Alias for --wait-for interaction
--timeout <ms> Page load timeout (default: 30000)
--interaction-timeout <ms> Login/CAPTCHA wait timeout (default: 600000 = 10 min)
--interaction-poll-interval <ms> Interaction poll interval (default: 1500)
--download-media Download images/videos to local imgs/ and videos/, rewrite links. Requires --output
--media-dir <dir> Base directory for downloaded media (default: --output directory)
--cdp-url <url> Reuse existing Chrome DevTools Protocol endpoint
--browser-path <path> Custom Chrome/Chromium binary path
--chrome-profile-dir <path> Chrome user data directory (default: BAOYU_CHROME_PROFILE_DIR env or ./baoyu-skills/chrome-profile)
--debug-dir <dir> Write debug artifacts (document.json, markdown.md, page.html, network.json)

Quality Gate

Headless capture is provisional: some sites silently return low-quality content with exit code 0. Inspect the saved markdown after every headless run, using references/quality-gate.md for the checklist, recovery steps, and capture modes.

Output Path

baoyu-fetch does not generate paths; the agent builds one and passes it to --output:

  1. Base dir: EXTEND.md default_output_dir or ./url-to-markdown/
  2. Domain from URL (e.g., example.com)
  3. Slug from URL path or page title (kebab-case, 2-6 words)
  4. Path: {base_dir}/{domain}/{slug}/{slug}.md — one directory per URL so media stays isolated
  5. On conflict: {slug}-YYYYMMDD-HHMMSS/{slug}-YYYYMMDD-HHMMSS.md

Downloaded media goes into subdirectories next to the markdown file.

Adapters & Media

references/adapters.md: adapter catalog (X, YouTube, Hacker News, generic), media download flow (ask / always / never), JSON output schema. Read it for adapter-specific questions or media prompts.

Environment Variables

Variable Description
BAOYU_CHROME_PROFILE_DIR Chrome user data directory (or --chrome-profile-dir)

Troubleshooting: Chrome not found → --browser-path. Timeout → raise --timeout. Login/CAPTCHA → --wait-for interaction. Debug → --debug-dir to inspect captured HTML and network logs.

Extension Support

Custom configurations via EXTEND.md — see Preferences above.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

2 warnings18d5 checks · Risk SAFE
  • Gen Agent Trust Hub18d

    This skill fetches web content from any URL and converts it into high-quality markdown, featuring specialized adapters for X (Twitter), YouTube, and Hacker News, while securely handling logins and media downloads.

  • Socket18d

    No alerts

  • Snyk18d

    Risk: MEDIUM · 1 issue

  • Runlayer8mo

    4/15 files flagged

  • ZeroLeaks7mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at bf1ff07. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 3 weeks ago
version
1.61.0
Other metadata
metadata
{
  "openclaw": {
    "homepage": "https://github.com/JimLiu/baoyu-skills#baoyu-url-to-markdown",
    "requires": {
      "anyBins": [
        "bun"
      ]
    }
  }
}
  • url-to-markdown
  • web-scraping
  • chrome-cdp
  • markdown
  • baoyu-fetch
  • x-twitter
  • youtube
  • hacker-news
  • login-captcha

README badge

README badge for jimliu/baoyu-skills/baoyu-url-to-markdown

Fetches any URL and converts it to markdown using Chrome DevTools Protocol with site-specific adapters for X, YouTube, Hacker News, and generic pages. Handles login and CAPTCHA interactions via wait modes, and optionally downloads images and videos to local directories.

Generated from the current SKILL.md.

Does this skill require Chrome or Chromium installed?
Yes. The skill uses Chrome DevTools Protocol to fetch and render pages. You can specify a custom Chrome binary with `--browser-path` if the default is not found.
What happens if a page requires login or CAPTCHA?
Use `--wait-for interaction` to pause and let you complete login or CAPTCHA manually, then continue. The skill will wait up to 10 minutes by default (`--interaction-timeout`).
Which websites have built-in adapters?
X (Twitter), YouTube, Hacker News, and generic pages. The skill auto-detects the adapter or you can force one with `--adapter`.
Can this skill download images and videos from a page?
Yes, with `--download-media`. Media files are saved to `imgs/` and `videos/` subdirectories next to the markdown file, and links are rewritten to point to local paths.
Does this skill require Bun?
Yes. Bun is the runtime for the vendored `baoyu-fetch` CLI. If not installed, the skill will prompt you to install it.

Generated from the current SKILL.md. These answers refresh after source changes.