cmoriarty/trog
1
0
Fork 0
Code Issues 11 Pull requests Projects 1 Releases Packages Wiki Activity Actions

03.2 · Web research: searxng + webfetch-mcp sidecar #13

New issue
Closed
opened 2026-07-23 16:38:03 -04:00 by cmoriarty · 3 comments
cmoriarty commented 2026-07-23 16:38:03 -04:00
Owner
Copy link

Follow-up from ticket 03.1's audit (docs/toolbelt-audit.md): web research was a keep.

Architecture decision (2026-07-23): searxng + webfetch-mcp are generic infra, not trog services — every hosted LLM on trogdor (ollama, llama_cpp_* stacks, OpenCode sessions) benefits. They live in a separate common-mcps repo with its own git-connected Portainer stack, on searxng's own upgrade cadence, decoupled from trog's deploy churn.

Scope — common-mcps repo (new):

  • searxng service (image-only compose, Portainer git stack on trogdor)
  • manooll/webfetch-mcp sidecar (searxng + Mozilla Readability, no API keys); fallback candidates in the audit doc
  • Published LAN ports so any consumer (trog, other stacks, laptop agents) can point at it

Scope — trog side (this ticket closes when both exist):

  • WEBSEARCH_MCP_URL env var on aegra (Portainer stack env, like FORGEJO_URL)
  • Brain toolbelt config gains the MCP endpoint — budget is 2 tools (search, fetch)
  • Acceptance: production graph phase can research (e.g. "current Phaser 4 particle API") and the extracted content is readable text, not raw HTML

Boundary rule (from the discussion): project-agnostic + multiple consumers → common-mcps; knows about trog's domain → trog compose (trog-mcp, playtest service stay here).

Why: precedent from agentic-game-dev — server-side readability extraction measurably reduced failed web reads vs letting the model fetch raw pages.

Follow-up from ticket 03.1's audit ([docs/toolbelt-audit.md](https://forgejo.underthere.xyz/cmoriarty/trog/src/branch/main/docs/toolbelt-audit.md)): web research was a **keep**. **Architecture decision (2026-07-23):** searxng + webfetch-mcp are generic infra, not trog services — every hosted LLM on trogdor (ollama, llama_cpp_* stacks, OpenCode sessions) benefits. They live in a separate **common-mcps** repo with its own git-connected Portainer stack, on searxng's own upgrade cadence, decoupled from trog's deploy churn. **Scope — common-mcps repo (new):** - `searxng` service (image-only compose, Portainer git stack on trogdor) - [manooll/webfetch-mcp](https://github.com/manooll/webfetch-mcp) sidecar (searxng + Mozilla Readability, no API keys); fallback candidates in the audit doc - Published LAN ports so any consumer (trog, other stacks, laptop agents) can point at it **Scope — trog side (this ticket closes when both exist):** - `WEBSEARCH_MCP_URL` env var on aegra (Portainer stack env, like FORGEJO_URL) - Brain toolbelt config gains the MCP endpoint — budget is 2 tools (`search`, `fetch`) - Acceptance: production graph phase can research (e.g. "current Phaser 4 particle API") and the extracted content is readable text, not raw HTML **Boundary rule (from the discussion):** project-agnostic + multiple consumers → common-mcps; knows about trog's domain → trog compose (trog-mcp, playtest service stay here). **Why:** precedent from agentic-game-dev — server-side readability extraction measurably reduced failed web reads vs letting the model fetch raw pages.
cmoriarty commented 2026-07-23 17:26:18 -04:00
Author
Owner
Copy link

Infra half done — common-mcps live on trogdor

Deployed and tested:

  • New repo common-mcps (public, image-only compose, same pattern as trog) — Portainer git stack 94 on trogdor, redeploy webhook enabled
  • searxng @ http://192.168.1.105:8181 — JSON API enabled via inline compose config, limiter off, LAN-only
  • mcp-searxng @ http://192.168.1.105:8182/mcp — streamable HTTP, isokoliuk/mcp-searxng v1.11.1

Candidate swap at deploy time: webfetch-mcp (audit's lead) has no prebuilt image and undocumented transport — replaced by ihor-sokoliuk/mcp-searxng: prebuilt image, native streamable HTTP, web_url_read does URL→markdown. Audit doc updated. One deploy gotcha worth remembering: it binds loopback by default — MCP_HTTP_HOST=0.0.0.0 required.

Acceptance test passed end-to-end (real MCP handshake, not curl-poking): initialize → tools/list (4 tools) → searxng_web_search("Phaser 4 particle emitter API") returned relevant extracted results → web_url_read on docs.phaser.io returned readable markdown, not raw HTML.

Remaining for this ticket (trog side):

  • WEBSEARCH_MCP_URL env on aegra stack
  • Brain toolbelt config gains the endpoint (tool budget now 4 from this server — search, url_read, suggestions, instance_info; consider exposing only search + url_read to the brain if the server/client supports filtering)
  • Acceptance: production graph phase does live research through it
## Infra half done — common-mcps live on trogdor **Deployed and tested:** - New repo [common-mcps](https://forgejo.underthere.xyz/cmoriarty/common-mcps) (public, image-only compose, same pattern as trog) — Portainer git stack **94** on trogdor, redeploy webhook enabled - `searxng` @ `http://192.168.1.105:8181` — JSON API enabled via inline compose config, limiter off, LAN-only - `mcp-searxng` @ `http://192.168.1.105:8182/mcp` — streamable HTTP, `isokoliuk/mcp-searxng` v1.11.1 **Candidate swap at deploy time:** webfetch-mcp (audit's lead) has no prebuilt image and undocumented transport — replaced by [ihor-sokoliuk/mcp-searxng](https://github.com/ihor-sokoliuk/mcp-searxng): prebuilt image, native streamable HTTP, `web_url_read` does URL→markdown. Audit doc updated. One deploy gotcha worth remembering: it binds loopback by default — `MCP_HTTP_HOST=0.0.0.0` required. **Acceptance test passed end-to-end** (real MCP handshake, not curl-poking): `initialize` → `tools/list` (4 tools) → `searxng_web_search("Phaser 4 particle emitter API")` returned relevant extracted results → `web_url_read` on docs.phaser.io returned readable markdown, not raw HTML. **Remaining for this ticket (trog side):** - `WEBSEARCH_MCP_URL` env on aegra stack - Brain toolbelt config gains the endpoint (tool budget now 4 from this server — search, url_read, suggestions, instance_info; consider exposing only search + url_read to the brain if the server/client supports filtering) - Acceptance: production graph phase does live research through it
cmoriarty commented 2026-07-23 17:43:22 -04:00
Author
Owner
Copy link

Done — trog side wired, acceptance passed live

What was done (commit 47a0812; infra half in the earlier comment)

  • server/trog_lib/toolbelt.py — the brain's standing toolbelt. Mounts searxng_web_search + web_url_read from WEBSEARCH_MCP_URL via langchain-mcp-adapters (streamable HTTP). Only those two ride along (schema tokens are paid every turn); suggestions/instance_info filtered out. Empty env or dead sidecar → [], research silently off — a missing MCP must never kill a run, and failures aren't cached so a sidecar coming up mid-run is picked up by the next phase.
  • production.py — phase nodes now go through _research_invoke: a bounded tool loop (4 rounds, then the document is forced out of a model that would rather keep searching). Tool errors return as tool messages, not exceptions. Gates stay tool-free — critics judge, they don't google. Phase prompts get a research hint only when the belt is live.
  • Compose/stack — WEBSEARCH_MCP_URL on aegra (empty default = disabled); Portainer stack env set to http://192.168.1.105:8182/mcp; image rebuilt + redeployed.
  • Tests — 5 new (tool loop happy path, disabled fallback, round cap, tool-error resilience, env gating); 20/20 green, CI green on 47a0812.

Acceptance — live, in the deployed container, exact phase code path: _research_invoke with the real brain (--jinja already on) + real MCP endpoint, task: "Phaser 4 particle emitters, CURRENT API." The brain called searxng_web_search on its own — searxng's engine log shows its query (Phaser 4 particles emitter example 2024) — and produced a correct v4-only answer (knew ParticleEmitterManager was removed in v4).

Both halves of the ticket exist and are verified. Full-run research behavior (a phase researching mid-production-run) will get exercised by the next produce-test.sh run.

## Done — trog side wired, acceptance passed live **What was done** (commit 47a0812; infra half in the earlier comment) - **`server/trog_lib/toolbelt.py`** — the brain's standing toolbelt. Mounts `searxng_web_search` + `web_url_read` from `WEBSEARCH_MCP_URL` via langchain-mcp-adapters (streamable HTTP). Only those two ride along (schema tokens are paid every turn); suggestions/instance_info filtered out. Empty env or dead sidecar → `[]`, research silently off — a missing MCP must never kill a run, and failures aren't cached so a sidecar coming up mid-run is picked up by the next phase. - **`production.py`** — phase nodes now go through `_research_invoke`: a bounded tool loop (4 rounds, then the document is forced out of a model that would rather keep searching). Tool errors return as tool messages, not exceptions. **Gates stay tool-free — critics judge, they don't google.** Phase prompts get a research hint only when the belt is live. - **Compose/stack** — `WEBSEARCH_MCP_URL` on aegra (empty default = disabled); Portainer stack env set to `http://192.168.1.105:8182/mcp`; image rebuilt + redeployed. - **Tests** — 5 new (tool loop happy path, disabled fallback, round cap, tool-error resilience, env gating); 20/20 green, CI green on 47a0812. **Acceptance — live, in the deployed container, exact phase code path:** `_research_invoke` with the real brain (`--jinja` already on) + real MCP endpoint, task: "Phaser 4 particle emitters, CURRENT API." The brain called `searxng_web_search` on its own — searxng's engine log shows its query (`Phaser 4 particles emitter example 2024`) — and produced a correct v4-only answer (knew `ParticleEmitterManager` was removed in v4). Both halves of the ticket exist and are verified. Full-run research behavior (a phase researching mid-production-run) will get exercised by the next `produce-test.sh` run.
cmoriarty commented 2026-07-23 17:58:38 -04:00
Author
Owner
Copy link

Closing — 03.2 complete

Both halves shipped and live-verified:

  • common-mcps repo + Portainer stack 94 (searxng :8181, mcp-searxng :8182) — generic infra, own deploy cadence, any LLM on the LAN can mount it
  • trog side (47a0812): trog_lib.toolbelt + bounded research loop in production phases, WEBSEARCH_MCP_URL in stack env; acceptance passed with the brain issuing its own search through the full chain

Final thoughts / watch items:

  • Extraction quality — web_url_read markdown was serviceable but nav-heavy on docusaurus pages. If it wastes context in real runs, the swap is mcp-searxng-enhanced (trafilatura + PDF) — one image build, same endpoint shape, zero trog changes.
  • Next produce-test run is the real exam — phases researching mid-run will stretch wall-clock (research rounds are capped at 4, so bounded). Watch whether the fun gate's rewrites start citing current Phaser APIs — that's the payoff signal.
  • OpenCode sessions (ticket 10) should mount the same endpoint in their config — one websearch infra, both harnesses (design note on #11).
  • Boundary rule held nicely — trog never learned how search works, only a URL. Keep it that way for future common services.

Follow-on threads: #11 carries the OpenCode session design; ticket 05 has the aseprite candidates; audit doc (docs/toolbelt-audit.md) reflects deployed reality.

## Closing — 03.2 complete Both halves shipped and live-verified: - **common-mcps** repo + Portainer stack 94 (searxng :8181, mcp-searxng :8182) — generic infra, own deploy cadence, any LLM on the LAN can mount it - **trog side** (47a0812): `trog_lib.toolbelt` + bounded research loop in production phases, `WEBSEARCH_MCP_URL` in stack env; acceptance passed with the brain issuing its own search through the full chain **Final thoughts / watch items:** - **Extraction quality** — `web_url_read` markdown was serviceable but nav-heavy on docusaurus pages. If it wastes context in real runs, the swap is mcp-searxng-enhanced (trafilatura + PDF) — one image build, same endpoint shape, zero trog changes. - **Next produce-test run is the real exam** — phases researching mid-run will stretch wall-clock (research rounds are capped at 4, so bounded). Watch whether the fun gate's rewrites start citing current Phaser APIs — that's the payoff signal. - **OpenCode sessions (ticket 10) should mount the same endpoint** in their config — one websearch infra, both harnesses (design note on #11). - **Boundary rule held nicely** — trog never learned how search works, only a URL. Keep it that way for future common services. Follow-on threads: #11 carries the OpenCode session design; ticket 05 has the aseprite candidates; audit doc (docs/toolbelt-audit.md) reflects deployed reality.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/11-walking-skeleton
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/trog#13
No description provided.