Skip to content

smart-search: mandatory SERP-first cost order wastes the budget when the query names a site #248

Description

@ankitranjan7

Problem

skills/smart-search/SKILL.md makes search-engine-first ordering non-negotiable:

Cost order is mandatory: webcmd web fetch first, webcmd web fetch-browser second, search adapters last.

For a search query without a direct URL, start with fetched search-engine result pages, not adapters.

Do not jump to adapters because one engine blocked.

Only after fetch-first search, target-page fetch, and allowed browser fetches fail or are insufficient, discover search adapters.

That ordering is right when the target site is unknown. It is backwards when the request names the sites. For "research browser automation on Hacker News and Reddit", the correct first move is HN Algolia and the reddit adapter — both purpose-built, structured, and unblocked.

Observed cost

A real session following the skill correctly:

Step Outcome
DuckDuckGo HTML empty body (302 + nginx)
Bing CAPTCHA wall
reddit.com/search.json WAF security page
old.reddit.com/search wrong content — unrelated article via readability failure
news.ycombinator.com/item?id=… "Sorry." block page
pullpush.io FETCH_BLOCKED → escalate to fetch-browser → not installed (#247)
HN Algolia API worked — primary evidence
webcmd reddit search --subreddit … worked — primary evidence

Everything that produced the final report was site-native. Everything the mandatory ordering tried first was blocked, and the skill explicitly forbids switching after one engine blocks.

Proposed change

Add a site-native fast path ahead of the cost order: when the request names a known community or site (Hacker News, Reddit, Stack Overflow, GitHub, …), go to that site's adapter or public API first, and use SERPs only for discovery when no site is named or the named site has no adapter.

Supporting tweaks from the same session:

  • Treat an empty extraction (0 usable result URLs) as a failure, not a success with no results.
  • Treat a recognised block/challenge page as a hard blocker for that engine — do not re-fetch variants of it.
  • Preflight adapter availability before reporting a site as unavailable; "Reddit is blocked" was reported while the reddit adapter simply was not installed.
  • Warn on query tokens that collide with everyday English. puppeteer returned puppet videos and politics.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions