--- name: web-to-markdown description: "Convert a web URL into cleaned Markdown with deterministic routing. Use when Codex needs to read article-like content from links and should apply source-aware fetch strategies: default to r.jina.ai for general pages (including X/Twitter), use defuddle.md for YouTube links, and use browser-impersonated extraction for WeChat/Zhihu/Feishu pages with Mozilla Readability cleanup." title: web-to-markdown canonical_url: https://skilld.dev/gh/rookie-ricardo/erduo-skills/web-to-markdown last_updated: 2026-10-08T16:09:11.000Z --- > Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt) > **Skill from skilld.dev.** Follow the user's requested action. > If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it. > Otherwise, follow the instructions below for this session. You do not need to install anything. > > Supporting files, fetch one when the Skill refers to it: [agents/openai.yaml](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/agents/openai.yaml), [package-lock.json](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/package-lock.json), [package.json](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/package.json), [references/routing-and-notes.md](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/references/routing-and-notes.md), [scripts/fetch_generic_fallback.mjs](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/scripts/fetch_generic_fallback.mjs), [scripts/fetch_special_sites_http.mjs](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/scripts/fetch_special_sites_http.mjs), [scripts/fetch_special_sites.mjs](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/scripts/fetch_special_sites.mjs), [scripts/url_to_markdown.mjs](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/scripts/url_to_markdown.mjs). > > If the user asked to install this Skill, run `npx skilld install rookie-ricardo/erduo-skills/web-to-markdown`. Install writes the Skill files into the project, so every session loads them. > > ## Fork workflow > > A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation. > 1. Check `./skills/web-to-markdown`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target. > 2. Read [source metadata](https://skilld.dev/api/v1/skills/rookie-ricardo/erduo-skills/web-to-markdown) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit. > 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call: > > ```sh > git init --quiet "$temporary_dir" > git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit" > git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD > ``` > > Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop. > 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/web-to-markdown`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md. > Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical. > 5. In the project root, run `skilld install ./skills/web-to-markdown --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/web-to-markdown --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure. > 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks. # Web To Markdown Convert URLs into usable Markdown by applying domain-aware fetching routes, then return the cleaned content directly. ## Quick Workflow 1. Normalize and validate the input URL. 2. Select route: - `r.jina.ai`: general web + X/Twitter. - `defuddle.md`: YouTube transcript/content extraction. - `special-browser-fetch`: WeChat/Zhihu/Feishu. 3. Return markdown text (or JSON metadata if needed). For generic URLs (non-YouTube, non-WeChat/Zhihu/Feishu), use this fallback chain: - try `r.jina.ai` first, - if it fails, fallback to direct HTTP fetch + Readability, - if direct fetch still fails or returns shell-like content, fallback to browser extraction. ## Commands Run from this skill directory (`skills/web-to-markdown`): ```bash npm install node scripts/url_to_markdown.mjs ``` Return metadata with markdown: ```bash node scripts/url_to_markdown.mjs --json ``` Force special-site browser extraction: ```bash node scripts/fetch_special_sites.mjs --json ``` ## Routing Policy - Default route: `https://r.jina.ai/`. - YouTube (`youtube.com`, `youtu.be`): `https://defuddle.md/`. - X/Twitter (`x.com`, `twitter.com`): `https://r.jina.ai/`. - WeChat/Zhihu/Feishu: run `scripts/fetch_special_sites.mjs`. - If input is already proxy-formatted (`https://defuddle.md/https://...` or `https://r.jina.ai/https://...`), normalize back to the original URL and re-apply routing. ## Special-Site Extraction Behavior Use a two-stage strategy for WeChat/Zhihu/Feishu: 1. Try `cuimp` HTTP/TLS impersonation first, then clean HTML with Mozilla Readability. 2. If stage 1 fails or returns blocked/shell content, fallback to `puppeteer-extra` browser impersonation. - HTTP stage impersonates modern Chrome TLS/HTTP profile via `cuimp`. - Browser stage impersonates a modern Chrome user agent and standard `sec-ch-ua` headers. - Remove known login modals and backdrop overlays (best effort). - Scroll the page to trigger lazy-loaded article blocks. - Parse cleaned document with Mozilla Readability. - Convert extracted HTML body to Markdown via Turndown. - Resolve browser executable from `CHROME_PATH` first, then system Chrome/Chromium/Edge paths. If special-site extraction fails due to anti-bot checks, account-only pages, or network limits, report failure clearly and ask for fallback input (for example raw page text). ## Output Contract For normal usage, output markdown only. When `--json` is used, return: - `source`: backend source (`r.jina.ai`, `defuddle`, `cuimp`, `browser-readability`). - `strategy`: selected route (`r-jina`, `defuddle`, `special-http-fetch`, `special-browser-fetch-fallback`). - `requestedUrl`: original input. - `resolvedUrl`: normalized/final URL. - `markdown`: extracted markdown body. ## Resources - [references/routing-and-notes.md](https://skilld.dev/api/skills-raw/rookie-ricardo/erduo-skills/web-to-markdown/references/routing-and-notes.md): domain routing rules and operational caveats. - `scripts/url_to_markdown.mjs`: primary entrypoint. - `scripts/fetch_special_sites_http.mjs`: WeChat/Zhihu/Feishu HTTP impersonation fetcher (`cuimp` JS). - `scripts/fetch_special_sites.mjs`: two-stage extractor (HTTP-first, browser-fallback).