dsh-fetch-data

dsh-fetch-data

🌐 English | 中文

Structured data extractor for DeepSeek Harness — intercepts a page's real data APIs (XHR/fetch JSON) and returns precise fields (comment like-counts, rankings, prices, tables) that text extractors lose when flattening HTML.

read_url answers "what does this page say?" — fetch_data answers "what are the exact numbers/data behind it?"

Why it exists

	read_url (text extractor)	fetch_data (this plugin)
Reads	cleaned page text / Markdown	the page's underlying JSON APIs
Output	prose the model reads	structured fields ({title, view} pairs)
Strength	reading articles/docs	exact field attribution — which number belongs to which row
Weakness	flattened text loses number↔item association	requires playwright (browser engine)

Real example: on a Xiaoheihe post, read_url returned "60125" with no way to know if it meant 60 likes + 125 favorites or anything else. fetch_data intercepts /bbs/app/link/tree and returns {user, up, content} — unambiguous.

Tool

fetch_data(url, api?, fields?, maxItems?) — capture the page's data APIs and extract fields

Param	Type	Default	Description
url	string	required	http(s) page whose data APIs to capture
api	string	auto	Pin a specific endpoint from the structure-mode menu (e.g. "/x/web-interface/ranking/v2"); auto-selected when omitted
fields	string	structure mode	Comma-separated field paths to extract; arrays via []: "data.list[].title,data.list[].view"
maxItems	number	20	Max array items per extracted field (1–100)

Two modes:

1. Structure mode (no fields) — returns the auto-picked endpoint's 2-level structure plus a menu of all JSON APIs (path · size · array?), so the model can re-call with api= to pin one:

页面 25 个 JSON 接口
选中: /x/web-interface/ranking/v2 (137763B)
结构: { code: number, message: string, ttl: number, data: { note: string, list: [100] { aid: number, ... } } }

接口清单（可传 api=<路径> 指定其中一个再提取字段）:
  /x/web-interface/nav · 249B
  /x/vip/ads/materials · 1140B · 含数组
  ...

2. Field mode (with fields) — extracts exact values, arrays truncated at maxItems:

data.list[].title (前5条):
  用MC还原《神的随波逐流》 【B萌应援】
  WasteTheFallen丨首曝PV＆实机演示：凝视深渊，人性渐泯
  ...
data.list[].stat.view (前5条):
  2428730
  10326863
  ...

Auto-pick logic

Ranks captured JSON responses: largest one containing an array wins (data endpoints are usually big and array-bearing; tracking/config endpoints are small). Falls back to the largest JSON. Manual api= overrides.

Real-world verification (2026-08-16, v0.1.3)

14-site sweep: 12 OK / 2 static-site expected errors / 0 crashes — driven by multi-site.mjs (committed). Auto-scroll (lazy-load capture) + JSONP parsing verified live.

Site	Result
Bilibili popular ranking	✅ auto-picked /x/web-interface/ranking/v2 (139KB, list[100]), extracted titles + play counts, 1:1 attributed
Juejin feed	✅ auto-picked /recommend_api/v1/article/recommend_all_feed, extracted article titles
Weibo	✅ captured /ajax/feed/hottimeline (240KB) + /ajax/statuses/config (745KB, oversized config edge case)
QQ news	✅ captured /getQNChannels (304KB, 36 APIs)
Douban	✅ captured /rexxar/api/v2/search/hots
Taobao	✅ captured 20 JSON APIs incl. mtop config (824KB, oversized-config edge case)
JD	✅ captured /wp-json/news/list + /category/get
Zhihu / Baidu / CSDN / Netease / Xiaoheihe	✅ captured their JSON endpoints (some are config/menu APIs — use structure mode to pick the data one)
example.com / ruanyifeng.com (static)	✅ clear "no JSON API captured" error, no crash

• 18 zero-dep assertions (field-path extractor, pick logic, structure summary, static-asset filter, truncation precision, description-length guard, JSONP parsing) + 22 live-interception assertions (incl. scroll-triggered lazy capture + JSONP extraction) all green.

Why it saves tokens

• Two-phase: structure mode returns a small "menu" (not the 137KB payload); field mode returns only the requested columns — never dumps the full JSON;
• Compact fixed cost: tool description trimmed to ~300 chars (the one thing sent on every call); static schema (KV-cache friendly);
• Sorted menu: array-bearing + biggest endpoints listed first, so the model finds the data API immediately; static-asset JSON (Bilibili /bfs/svg-next/...) filtered out and the count shown inline;
• Sizes rendered as 136KB not 139070B; arrays truncated at maxItems; values capped at 200 chars;
• Compact text render; clear one-line errors.

Architecture (DSH-aligned)

• Browser singleton launched once, closed via ctx.effect on unload (temporal composability);
• Fresh browser context per call — no cookie/state leakage across calls (avoids anti-bot flakiness);
• domcontentloaded + settle wait instead of networkidle (heartbeat-polling sites never idle);
• Auto-scroll triggers lazy-loaded data APIs (feeds / infinite lists); JSONP responses auto-unwrapped;
• Cooperative timeout: timeoutMs + exec.signal;
• Zero runtime deps beyond Node built-ins; playwright is the required engine (it's the interception layer, not an optional enhancement).

Install

# playwright is REQUIRED for this plugin (network interception core)
cd <DSH profile dir>
npm i playwright && npx playwright install chromium

# add the plugin
dsh plugin --profile web add github:2672243194/dsh-fetch-data

Boundaries

• Login-walled APIs are not accessible (same as read_url);
• Each site has its own endpoint structure — use the structure-mode menu to discover it;
• Auto-scroll + JSONP cover most lazy-loading/JS-delivered data; remaining gaps (login walls, SSR-only pages) return a clear error, never a guess.

Support

If dsh-fetch-data helps you, give it a ⭐ Star on GitHub. Free and open source (MIT); star count is how I decide where to keep investing.

License

MIT