dsh-arxiv-ml-search

dsh-arxiv-ml-search

A DeepSeek Harness plugin that searches arXiv for machine-learning papers, so
the agent checks claims against the published record instead of its own memory.

Two tools plus a skill that tells the model how to use them:

• arxiv_search — keywords, authors, categories, date ranges, sorting and
  paging. Returns truncated abstracts so a ten-result search stays cheap.
• arxiv_get — full metadata and complete abstracts for specific ids, with
  optional segmenting (max_chars + segment) for paging through long text.
• SKILL.md — the claim-checking procedure: restate the claim, run 2-3
  orthogonal queries, read the shortlist properly, report a verdict with ids,
  and never assert anything the returned text does not say.

What the skill actually changes

The two tools are always in the toolset, so the model can reach arXiv without
the skill. What the skill adds is knowing how to search — and that turns out to
be most of the value. Measured on one prompt, run twice in the same workspace,
once with the skill loaded and once without:

	skill not loaded	skill loaded
synonym sets (any_of)	0	11 distinct, across 36 calls
category narrowing (ml_only)	9	36
relaxed matches reported to the user	—	4

Without it the model still searches, just blindly: one phrasing per concept, no
narrowing, and no obligation to say when a match was approximate rather than
exact. With it, the answer names which abstracts were actually read and flags
where the evidence is thin.

It loads without being asked for papers

The skill is meant to fire on any ML question, not only on "find me papers".
Ask a plain engineering question —

My RL agent gets a set of state features but the reward never makes it rely
on some of them. How do I increase a specific feature's influence?

— and the model loads the skill and answers from real work: potential-based
reward shaping, curiosity, auxiliary tasks, each with an arXiv id it fetched
rather than recalled.

Getting that to happen took a specific fix. The skill catalog the model reads
carries name and description only — whenToUse never reaches it, and the
description is truncated past 500 characters — so that one string is the entire
routing signal. An earlier one-liner about "searching arXiv for papers" got
skipped on advice-shaped questions, with the model reasoning "no skill needed,
it's a general question", and then using the tools without any of the rules
above. The description now claims those questions explicitly. Routing is still
a model judgement rather than a guarantee: if the skill gets skipped, the
description is the thing to edit.

No Python and no build step. The arXiv API is a plain HTTP GET returning Atom
XML, so lib/atom.js parses exactly the fields we use, and the only thing the
plugin needs from the harness is @deepseek-ai/dsh-tools.

That must stay a peer dependency, never a regular one. dsh writes
autoInstallPeers: false into every profile's pnpm-workspace.yaml on purpose:
the harness owns dsh-tools, and a plugin that declares it under
dependencies makes pnpm install a second copy into the profile.
TOOL_RUNTIME_SCHEDULER inside that package is a module-local Symbol(), so
two copies mean two different symbols, and the whole tool registry breaks — not
just this plugin's tools. The symptom is every tool call in the session, built-in
ones included, failing with:

Cannot read properties of undefined (reading 'prepare')

Verified working on dsh 0.1.0-rc.8, in both the headless and web profiles.

Install

Two lines: add the package to the profile you use, then start it. Nothing to
download by hand and no build step.

dsh plugin --profile web add dsh-arxiv-ml-search

Without a global dsh install, the same through npx:

npx @deepseek-ai/dsh dsh plugin --profile web add dsh-arxiv-ml-search
npx @deepseek-ai/dsh web

Ask a question and the tools are simply there — no flags, no mention of the
plugin:

Do any papers show RLHF hurting calibration? Cite ids.

Any other profile works the same way; swap web for headless and it answers
one task on the command line and exits:

dsh plugin --profile headless add dsh-arxiv-ml-search
dsh --profile headless "Do any papers show RLHF hurting calibration? Cite ids."

Installing an unpublished build

Only needed when you are working on the plugin itself. dsh plugin add is a
thin wrapper over pnpm, so it takes a directory, a tarball or a git URL wherever
a package name would go:

dsh plugin --profile web add ../dsh-arxiv-ml-search          # a working copy
dsh plugin --profile web add ./dsh-arxiv-ml-search-0.1.0.tgz  # a local pack

Installing from the registry needs none of this.

If the plugin does not show up

Loading needs the package listed in dsh.profile.bundles
(~/.dsh/profiles/<name>/package.json), not just in dependencies. Normally
dsh plugin add appends it for you: after pnpm succeeds it walks the
dependencies and adds every package that declares dsh.bundle.

The catch is that the reconcile step only runs when pnpm exits zero. An
unrelated blocked build script elsewhere in the profile — ERR_PNPM_IGNORED_BUILDS
is the usual one — aborts the command first, so the package installs but never
joins the layer stack and the plugin stays silently inert. Check the manifest
after installing and add the entry by hand if it is missing:

"dsh": { "profile": { "bundles": [ "...", "dsh-arxiv-ml-search" ] } }

Confirm what actually composes, without booting anything:

dsh --profile web --dump-config

Develop

pnpm install                      # needed once: tests/plugin.test.js loads index.js
node --test                       # unit tests, offline, against saved feeds
node scripts/smoke.mjs "rlhf"     # live check against the real API
node scripts/smoke.mjs --refresh-fixtures

Dev-load without installing, via dev.cordis.yml in this directory:

- insert:
    - id: dsh-arxiv-ml-search
      name: file:///C:/absolute/path/to/dsh-arxiv-ml-search/index.js

dsh --profile headless --patch ./dev.cordis.yml "Do any papers show RLHF hurting calibration?"

Two things bite here:

• The path must be absolute; a relative one resolves against the harness
  installation, not your working directory.
• On Windows it must additionally be a file:// URL. A bare C:/... makes
  the ESM loader fail with ERR_UNSUPPORTED_ESM_URL_SCHEME (Received protocol 'c:'), because it reads the drive letter as a URL scheme.

Check the overlay composes into the tree before booting anything:

dsh --profile headless --patch ./dev.cordis.yml --dump-config

Use the overlay for iterating on the source; use a real install (above) for
anything you actually want to keep.

Layout

index.js          host plugin: tool + skill registration only
lib/query.js      builds search_query and request URLs
lib/atom.js       Atom feed -> paper records
lib/segment.js    truncation and segmentation (text budget)
lib/format.js     paper records -> tool output and chat rendering
lib/http.js       fetch: User-Agent, rate limit, timeout, one retry
lib/arxiv.js      orchestration, testable with an injected fetch
tests/            node:test suites; fixtures are real captured feeds

Every deterministic step lives in lib/ so the tests run without the harness
and without the network. lib/http.js is the only impure module, and it takes
fetchImpl as an option for exactly that reason.

Config

Set in the profile entry for this plugin:

key	default	meaning
categories	the ML set	categories used by ml_only
abstractChars	350	abstract budget per search result
limit	10	default page size
contact	—	contact address added to the User-Agent
timeoutMs	20000	per-request timeout

Notes on the arXiv API

• Requests are spaced at least three seconds apart and carry an identifying
  User-Agent, as arXiv asks of API clients. The limiter is process-wide.
• Search covers metadata and abstracts, not full text. A claim that turns on
  experimental detail needs the paper itself — the skill says so explicitly.
• arXiv is preprints. Absence of a result is not evidence of absence.

Query syntax

Plain text becomes a phrase clause. Anything containing a field prefix or a
boolean operator is passed through to the API unchanged:

abs:"reward hacking" ANDNOT abs:"reinforcement learning from human feedback"
(ti:"scaling laws" OR ti:"scaling law") AND cat:cs.LG

Generated with dsh-plugin-starter.