Search That Shows Its Work

search

A self-hosted MCP server for web search that reports how much of each answer to believe

A search tool fails in ways that look exactly like success. An engine answers with somebody else's subject. A page returns text that is an anti-bot shield. Sixteen sources turn out to be two engines counted eight times. None of that raises an error, and the model on the other end builds on it.

This server's job is to make those cases distinguishable, in fields you can branch on. It runs on your machine, over your own metasearch instance, with your own model key — or none at all.

🔧 Tools

Full argument reference, response shapes and failure modes: HOWTO-CALL.md.

📦 Install

From an open repository page to a working answer. Nothing is assumed to be on your disk already:

git clone https://github.com/AG-Bureau/mcp-search
cd mcp-search
cp .env.example .env
echo "SEARXNG_SECRET=$(openssl rand -hex 32)" >> .env
echo "BROWSER_WS_SECRET=$(openssl rand -hex 16)" >> .env
echo '{}' > door-tokens.json && mkdir -p tls
docker compose -f docker-compose.yml -f wiring/expose-localhost.yml up -d --build
curl -s http://127.0.0.1:8081/healthz

The fourth and fifth lines are not decoration. Without a value in SEARXNG_SECRET the up command refuses — and that refusal is deliberate: with no key of its own the metasearch does not fail, it comes up with a publicly known one from its image template, silently. BROWSER_WS_SECRET is the secret path of the browser sidecar's endpoint, and up refuses without it too: on a guessable path, anything that can open a WebSocket to the sidecar — any container on the compose network, and any page the browser itself opens — gets a browser of its own, outside every check the module applies to the pages it reads. The compose file hands the same value to the sidecar and, at the end of BROWSER_WS_URL, to the adapter; neither prints it.

The sixth line creates the two paths the compose file mounts for caller authentication and TLS — both off by default. A path that does not exist is created by Docker as a root-owned DIRECTORY, and a token "file" that is a directory reads as an unreadable list (see Configuration below). Both are in .gitignore: they hold secrets once you use them.

The overlay publishes the port on loopback only. A published container port does not go through the host firewall's usual chain, so exposing it more widely is a separate, deliberate step — see Deployment.

Two transports

MCP has two, and they answer different questions. HTTP — the commands above — is for a server that is already running somewhere. stdio is the protocol's default: the client starts the server as a process and talks to it through the pipes, which is how most desktop clients and wrappers work.

docker compose exec -T ag-search python /app/server.py --stdio   # or MCP_TRANSPORT=stdio
# outside compose: SEARXNG_URL=http://<a metasearch you can reach> python adapter/server.py --stdio

One JSON-RPC object per line in, one answer per line out. The mode is chosen explicitly and never guessed from whether a terminal is attached — that sign merely sits next to the subject, and one day it answers for a case nobody meant.

In stdio mode stdout is the protocol: answers and nothing else, with the log on stderr. One stray line of anything else breaks the client reading it.

Each process started this way is an instance of its own, even inside the running container: its browser sessions (BROWSER_MAX_SESSIONS), its engine and domain pacing and its /stats counters are separate from the HTTP server's. Several stdio clients on one sidecar add up in its pids_limit like several adapters do.

The sidecars do not depend on the choice; the metasearch does: SEARXNG_URL must reach one, or every search refuses with cause: upstream (the default, http://searxng:8080, is a name only the compose network knows). Started by a client with no compose project around it, the module names what else is missing instead of pretending: the browser path reports not_wired_up, and trouble carries pool_unmeasured — the engine pool was never computed from observation.

⬆️ Upgrading to 0.3.5 — no new variable

From 0.3.4: the same command as the install block, with the same -f files. Only the adapter image changes (ag-mod-search/adapter:0.3.5, used by the adapter and the prober); the sidecar stays ag-mod-search/browser:1.49.1-3. Two things you may notice: the door now logs one line per request (the caller's address, the path, the status and the caller's token name — never the token, the body or the query string), and a model call that runs out of the time its caller had now returns a refusal that says so instead of running on. From 0.3.3 or earlier, the 0.3.4 steps below apply too.

⬆️ Upgrading to 0.3.4 — no new variable, one stricter one

From 0.3.3: the same command as the install block, with the same -f files. Only the adapter image changed; the sidecar stays ag-mod-search/browser:1.49.1-3. Check AG_DOOR_AUTH first: a value that is not off, warn or on (in any case, spaces around it ignored) now stops the adapter from starting, where it used to mean off. From 0.3.2 or earlier, the 0.3.3 steps below apply too.

⬆️ Upgrading to 0.3.3 — one new required variable

BROWSER_WS_SECRET must be in .env before up: add it as the install block does, with openssl rand -hex 16. Then run the install block's own command, with the same -f files you installed with: docker compose -f docker-compose.yml -f wiring/expose-localhost.yml up -d --build. It rebuilds both images and recreates the adapter, the prober and the sidecar, since the image or the environment of each changed. Name no services on it: up … browser ag-search leaves the prober on the old image, and leaving out the -f overlay recreates the adapter without its published port. The sidecar image is ag-mod-search/browser:1.49.1-3; with the old sidecar, or with only one side recreated, the browser path reads "not responding".

A malformed secret stops the whole module. Compose checks only that the value is not empty. Characters outside A-Z a-z 0-9 _ - (base64 gives + / =) make the sidecar exit with 78 in a loop; the adapter waits for a healthy sidecar, so it never starts and up says "dependency failed to start".

If you run the door in warn or on, also read "Moving to on" under Configuration: the prober now takes a token, a token with no scopes (or with scopes written as a string) may call no tool, and /healthz?deep=1 needs a token that may search or read.

⬆️ Upgrading from 0.2.x — the answer changed shape

If you already run 0.2.1 or earlier, read this before updating. Nothing here is a new feature you may ignore; it is what your existing calls will return differently.

The search and image answers carry fewer fields by default, and they say so in their name: ag.search/3 and ag.images/3 instead of /2. A caller that branched on contract will break loudly, which is the intent — a field that simply vanished would read as "nothing was wrong" in most languages.

trouble is always present and empty when nothing went wrong, so if not trouble replaces the four separate checks. Nothing was deleted from the module — the accounting moved behind a request.

Booleans are now parsed rather than cast. In 0.2.x "read": "false" over MCP read the pages anyway — eight times the wall clock — while the plain door understood the same word correctly. Both doors now accept true/false, 1/0, yes/no, on/off, y/n, t/f and Python's True/False, case-blind. A value that cannot be read turns the flag off and is named in arguments_adjusted, and an EMPTY value counts as unreadable: read="" no longer buys the expensive default in silence.

read_top: 0 means "no preference", not "read nothing" — for nothing, use read: false. The schema used to declare minimum: 1 while accepting zero.

The field-by-field table is in the contracts: contracts/ag.search.v3.md and contracts/ag.images.v3.md, section "What changed against /2". Reading (ag.read/2), deep search (ag.deep/2) and screenshots (ag.shot/1) kept their numbers: they gained fields, and an addition breaks nobody.

⚙️ Configuration

A token file that cannot be READ is not an empty token file. Seen on a real deployment: the file was mounted into the container with mode 600 owned by root while the container runs as an unprivileged user. The door opened nothing, refused nothing, and looked correctly configured — the inability to look had been handed back as the result of looking. /stats therefore carries three separate fields: door_auth (the mode), tokens_configured (is a file named at all) and tokens_readable (did it yield at least one token — a readable {} reads false too). An unreadable list refuses everyone in on — the safe direction for a file we could not read — and in warn it serves everyone, which is exactly why the two cases must not share one field. A missing path is worse still: Docker creates it as a DIRECTORY, which reads like an unreadable file — so create the file before the first up (the install block above does) and check that it is a file.

Who owns the secret files. The adapter runs as uid 10001. Once the token file holds tokens, and for a TLS key: chown 10001:10001 door-tokens.json tls/key.pem && chmod 600 door-tokens.json tls/key.pem. The tempting fix for the failure above, chmod 644, makes the tokens and the private key readable to every user of the host.

Moving to on. Give the prober a token first (PROBE_DOOR_TOKEN), then go off → warn, and switch to on once unauthenticated in /stats has stopped growing — with the prober on a token, what is left there is callers you have not given one to yet. Since 0.3.5 the log says who they are: every request prints one line, [door] <address> <method> <path> -> <status> in <ms> ms, caller: <token name> (no token without one, unknown token for one the token list does not have — the refusal a caller gets does not tell the two apart, the log does; - where no door verdict was taken: a request refused before it could be read, a method the door does not serve such as HEAD or OPTIONS, or a POST anywhere but /mcp), with ?… where the request had a query string and the client's own X-Forwarded-For, unverified, when it sent one. The token, the other headers, the body and the query string are never logged.

The browser's session and watchdog limits are set in .env; see .env.example, which explains each one where you set it. Pacing, pool size and read limits have measured defaults in adapter/server.py, adapter/reader.py and adapter/pool.py; docker-compose.yml does not pass them through.

A model key is optional. Search, reading, image search and screenshots are HTTP requests and spend no model tokens. A model is called in exactly two places, and both are named in the answer: web_deep_search, and recognising a PDF with no text layer — which happens when you ask to read such a document, never behind your back in web_search. web_deep_search reads the pages it picks with recognition on, so a scan among them is recognised as part of that search's own model spend. With the door on, both need a token with the paid scope.

🎯 What a bundled search tool does not do

Cost you control. One argument changes the answer by an order of magnitude:

Measured on one machine, one query. Take the shape, not the digits. A consumer measured the same fork from outside and got 4.8-10.5 s against 0.7 s — the shape holds, the digits depend on the pages the query happens to find.

The choice is made before the call, not after the bill. read: false when you are mapping what exists or working under a narrow context ceiling; the default when you want the text of the top results and would otherwise fetch it yourself. read_top: 0 means "no preference", not "read nothing".

The engine list maintains itself. A hand-written list goes stale in silence: an engine that was the best returns nothing weeks later and says nothing about it. Ours was revised three times in a single day — each revision against the previous one, each correct on its own data. The problem was never the engines: a decision freezes while observation goes on.

So the list is not written here. A prober asks every known engine, continuously, with questions whose correct answer is known in advance, and the pool is the best few by reference hit share — recomputed on its own. Verified by falsification: a planted bad run took an engine out of the pool with no code change, and restoring the run brought it back by itself.

Until enough observation accumulates, the pool is a seed list and every answer says so: trouble.pool_unmeasured by default, pool_source itself under verbose.

Failure is distinguishable from success. Four ways an engine can fail, and what shows each:

The same applies to reading: seven distinct outcomes, and a page that returned a shield is stub, not empty text.

🩺 If the module stops and nothing looks wrong

One failure of this system does not show up in anything you would normally watch, so it is written down here rather than left to be rediscovered.

The symptom. Containers report unhealthy, docker compose up refuses to raise the adapter, and docker exec fails with runc: nsexec ... setns. Load, memory and disk are all fine.

The check, one command:

ps -eo comm | sort | uniq -c | sort -rn | head

If headless_shell is in the thousands, the browser sidecar has accumulated processes — a browser that was opened and never closed keeps its renderers alive, and past a few thousand the container runtime can no longer start a process at all. Since both health checks work by exec, everything then reports sick at once, and the cause looks like the last thing you changed.

The cure: docker compose up -d --force-recreate browser. Measured on our own instance: 4619 processes before, 126 after.

What 0.3.1 changed. The processes were never alive: chromium's children are orphaned when a visit ends, and a PID 1 that does not call wait() leaves each of them a zombie holding its slot — forty after ten visits, measured. The sidecar image now runs a real init that reaps them (and init: true in the compose file says the same for anyone who replaces the image), and the module closes every browser on every path including the ones that fail.

The second silent failure: every browser read hangs, then the disk fills

The symptom. web_read with the browser, and /healthz itself, stop answering while search still works; if several adapters share one sidecar, all of their browser reads fail; the machine's disk then fills with no log growing.

The check — count THREADS, not processes:

docker stats --no-stream --format '{{.Name}} {{.PIDs}}' | grep browser

pids_limit counts every thread of the container. An idle Chromium tree is about 42 threads, run-server's own base about 14; at the cap every new browser fails to create a thread and crashes, while the sidecar's own health check — a TCP connect — stays green. ps shows a handful of processes and nothing wrong.

What 0.3.2 changed, after exactly this on 2026-09-23 — a burst of concurrent reads filled the cap, eleven sessions hung for six hours in stages that have no timeout, the health probe hung behind a lock, and each crashing Chromium wrote a core dump into the container layer until the disk was full:

• every browser session runs in a child process of its own and is killed with its whole process group when it outlives its budget plus BROWSER_KILL_GRACE_S (default 10 s); the caller gets not_reached saying so, and saying whether the sidecar or the page was to blame — a page whose script never yields is the page's fault and does not mark the sidecar sick;
• one adapter instance holds at most BROWSER_MAX_SESSIONS sessions at once (default 2, health probes included); a read over that is told the browser is busy after at most BROWSER_SLOT_WAIT_S (default 5 s) instead of starting a browser the sidecar cannot run; the time spent waiting comes off the session, whose budget is cut to what the call has left — only a session that hangs anyway is killed BROWSER_KILL_GRACE_S (10 s) after that, and one that answered at the last moment gets up to 3 s more to exit, so a call can overrun its ceiling by up to 13 s with the defaults; and a slot whose session may have left a browser behind on the sidecar (killed during the launch, or a close that hung) is held back for BROWSER_COOLDOWN_S (default 150 s) while the sidecar's watchdog clears it;
• /healthz never launches a browser and never waits for one: the browser state is the outcome of the last real session, with its age, plus a port check, and a real probe runs in the background at most every BROWSER_PROBE_S (default 600 s) — or every health tick while the last verdict is "not responding", so a recovery shows within minutes rather than ten; until the first probe after a start it reads not checked yet: … and counts as degraded;
• the sidecar's main command is a watchdog (browser/supervise.py) that kills any Chromium browser older than BROWSER_MAX_AGE_S (default 120 s — the longest legitimate session is 42.5 s), whatever the client did;
• the adapter, the prober and the browser set ulimits: core: 0, so none of them writes core dumps, and the adapter image has an init of its own.

The cure, if it happens anyway: docker compose up -d --force-recreate browser. The adapters need nothing: since 0.3.2 their /healthz keeps answering, and their browser state turns back to alive by itself within a health tick of their session slots coming free. Slots whose sessions were killed while the sidecar was wedged are held back for BROWSER_COOLDOWN_S (150 s), so the whole recovery takes up to about three minutes — measured on a test instance: 167 s from recreating the sidecar to alive, with no read in between. GET /pages runs a real browser session there and then once a slot is free; docker compose restart ag-search skips the wait if it matters more. An adapter older than 0.3.2 is the exception: its /healthz can hang, and it needs docker compose up -d --force-recreate ag-search — and an upgrade.

⚖️ What it does with robots.txt, and why you must decide

The module REPORTS a site's rules and does not enforce them. Every read carries robots: allowed, disallowed_by_site, or not_checked when the file could not be read. A page a site forbids is still fetched, and the answer says so.

allowed means we read the rules and they permit this path — not merely that we saw no ban. A site that says the file is not there (404) has no rules, and that is allowed. A file that did not arrive at all — 5xx, a timeout, a torn connection, 401/403 — or arrived in an encoding we could not unpack is not_checked. Those are different events and the field keeps them apart: no data means "not checked", never "permitted".

That is a decision, not an omission, and it belongs to whoever runs this rather than to the module. Two reasons. Whether a tool called by a person obeys robots.txt is the operator's call — a rule written for crawlers indexing the web is not obviously a rule for fetching one page a user asked for. And the reference behaviour — treat 401/403 on robots.txt as a ban — produces false bans on ordinary sources, because the same sites answer 401 to everyone from behind an anti-bot service. We do not call that a ban. We do not call it permission either: the answer is not_checked, which says what actually happened — the rules exist or do not, and we did not get to see them.

So the gate is yours to add. If your use requires obeying robots, branch on the field: robots == "disallowed_by_site" means the site says no. If you obey it, treat not_checked as a stop too — it means we could not read the rules, not that there are none.

📖 How it works

• ALGORITHM.md — what happens, step by step, on each call.
• contracts/ — the call contracts, versioned separately from the code that implements them.
• measures/ — dated measurements: which engines were alive, what the load ladder gives, what the transport change bought. Numbers, with what was measured and when.
• contracts/ag.search.v3.md — and its neighbours: what each capability promises, plus the table of what changed against /1 and /2 for anyone with stored answers to read.

🔒 Deployment and exposure

wiring/expose-localhost.yml publishes the adapter on 127.0.0.1 only, and it is the only overlay that ships. Anything wider is an overlay you write yourself, and the header of wiring/expose-localhost.yml says what to check first: Docker passes traffic to published ports through FORWARD after DNAT, while the firewall's own chain sits before its hooks — so a firewall that says "closed" can be open to the internet on a published port.

A search server open to the outside is an open proxy that goes to the network in the machine owner's name.

Known limits of this release, named so that none reads as absent:

• The browser path keeps pages out of the perimeter with scripts and routing inside the browser, not with the network. Three cases stay open, and they differ. DNS rebinding is not closed and the page is not withheld: the inside/outside verdict is kept for the session, and the browser connects to whatever its own lookup returns. A redirect hop that went inside, and a prefetch from speculation rules the page's scripts cannot reach (inside a closed declarative shadow root, or an open one the parser made after pausing on its host), are seen and the page is withheld, but only after the request was made. A prefetch from a popup the page opens is not watched, and that page is not withheld. Since 0.3.5 the rules a page's scripts can reach are removed. The full list is in contracts/ag.read.v2.md; closing it needs a proxy or egress rules on the sidecar.
• The door log can name the wrong forwarded-for. On a keep-alive connection, a request refused before its headers are parsed (400, 414, 431, 505) is logged with the previous request's X-Forwarded-For. The field is marked unverified either way; this is open in 0.3.5 and fixed in the next release.
• Deep search words its own queue's refusal as the provider's. When no model slot comes free in time, the answer's cause is busy, as documented, but its error text says the model is rate limited rather than naming LLM_MAX_CONCURRENT. Only the text is wrong.
• BROWSER_WS_SECRET is visible to root inside the sidecar. tini and the sidecar's supervisor keep it in the environment they were started with, and root in that container reads it through /proc. The sidecar runs as root and its browsers with --no-sandbox, so a page that took over a renderer could read it; it would then already be root in the sidecar the secret protects.
• The call ceiling of web_read does not bound everything beneath it. A browser session that hangs, and one scan page's render that began just before its stop, can end past it. Since 0.3.5 the wait for a free model slot and the model's whole answer are bound by it. See the limits table in HOWTO-CALL.md.

✅ Tests

IMAGE=ag-mod-search/adapter:0.3.5 bash tests/in-image.sh

Four suites — the protocol and search against a fake metasearch, reading against a fake site, the computed pool against a database built in memory, and the parse of a model provider's answers against canned replies. Not one of them makes a single outbound request, and the runner holds that with --network none rather than on trust: for reading it matters more than for search, because a test that went to the internet would spend the very resource the tool protects — the reputation of the one address it calls from.

They run inside the built image rather than on the machine where the code is edited: the PDF parser lives in the image, and a suite run outside would skip everything that touches it. The skip is not silent — the check goes red with a note saying where to run it.

What these suites cannot check is written down in tests/README.md.

🤝 Contributing

A capability, engine or heuristic is not accepted until its reference attribute is declared — a property of the correct answer that the thing being tested could not have told us itself — and a pool of checked questions is attached. See CONTRIBUTING.

📄 License

GNU Affero General Public License v3.0. Run it, change it, build on it. If you make it available to others OVER A NETWORK, the changes you made go back out under the same licence — that is the one obligation, and running a service counts as making it available.

For whoever cannot live with that clause, a commercial licence is a question to ask rather than a fork to make.