> Source: https://vitna.ai/docs/search/serp
> Raw Markdown: https://vitna.ai/docs/search/serp.md
> Corpus index: https://vitna.ai/llms.txt
# Google and Bing SERP retrieval
Vitna's SERP API retrieves actual Google or Bing search result pages through
Vitna's own transport and parser. It returns parsed organic and rich SERP
features (or the fetched HTML) without calling Bright Data, supporting web,
news, images, videos, and shopping verticals alongside SerpApi-compatible
query aliases and output shapes.
## Request
```text
POST /v1/serp
GET /v1/serp
POST /v1/serp/bing
POST /request
```
`POST /v1/serp` and `POST /request` accept the same JSON request; `GET /v1/serp`
accepts the scalar fields below as query-string parameters for drop-in SerpApi /
Serper compatibility; `POST /v1/serp/bing` defaults `engine` to `bing` and
serves Bing SERP requests only. Send a dedicated `vitna_search_` bearer key, or
your org's Vitna key, `vitna_key_` (see [one key](../one-key.md)). `/request`
supports the URL-based request shape used by Bright Data's SERP API without
forwarding the request to Bright Data.
```bash
curl http://127.0.0.1:8789/v1/serp \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"engine":"google","query":"lunar science","country":"us","language":"en","page":1,"maxResults":5,"format":"json"}'
```
| Field | Default | Allowed values |
|---|---|---|
| `engine` | `google` (`bing` on `/v1/serp/bing`) | `google` or `bing` (`bing` only on `/v1/serp/bing`) |
| `query` (or `q`) | Required | 1 to 1,000 characters, with no control characters |
| `country` (or `gl`) | Unset; search-engine default | Two-letter country hint; normalized to lowercase |
| `language` (or `hl`) | Unset; search-engine default | Language tag, such as `en` or `en-US`, at most 32 characters |
| `page` | 1 when `offset` is omitted | Integer from 1 to 20; selects ten-result offsets; cannot accompany `offset` |
| `offset` (or `start`) | Derived from `page` | Integer from 0 to 190; zero-based engine result offset; cannot accompany `page` |
| `maxResults` (or `num`) | 5 (10 when `url` omits `num`/`count`) | Integer from 1 to 30 |
| `type` / `tbm` | `web` | `web`, `news` (`nws`), `images` (`isch`), `videos` (`vid`), or `shopping` (`shop`) |
| `timeRange` / `tbs` | Unset | `hour`, `day`, `week`, `month`, or `year` (or raw Google `tbs` filter such as `qdr:w`) |
| `location` / `uule` | Unset | Canonical location string (encoded deterministically to UULE) or raw Google `uule` token; mutually exclusive, Google only |
| `googleDomain` (or `google_domain`) | `google.com` | One of 25 supported regional Google domains (such as `google.co.uk`, `google.de`, `google.co.jp`), Google only |
| `lr`, `cr` | Unset | Google language (`lang_en`) and country (`countryUS`) restrictions, up to five pipe-separated values |
| `device` | `desktop` | `desktop` or `mobile` |
| `safe` | Unset | `active` or `off` (maps to `safe` on Google and `adlt` on Bing) |
| `nfpr` | Unset | Boolean, `"0"`, `"1"`, `0`, or `1`, Google only; disables query spell-correction when truthy |
| `udm` | Unset | `"2"`, `"7"`, `"12"`, `"14"`, or `"28"`, Google only |
| `linkResolution` | `complete` | `complete` or `fast` (skips interactive `/goto` browser unlocking for low-latency organic runs), Google only |
| `format` | `json` | `json` for parsed results or `html` (or `raw`) for the fetched HTML |
| `fresh` | `false` | Boolean; bypass a stored JSON response and retrieve a new engine page |
| `extras` | `false` | Boolean; when true with `format: "json"`, extracts People Also Ask (plus `people_also_ask_details`), related searches, ads, answer box, knowledge graph, AI overview, top stories, local results, inline videos/images, and forum discussions |
| `includeOmitted` | `false` | Boolean, Google only; when Google says it left out entries very similar to those shown, the search runs again with Google's `filter=0` and answers with the fuller page (see [Google's notices](#googles-notices)) |
| `filter` | Unset | `"0"`, Google only; asks Google for every entry, the very similar ones included, in the one capture |
| `keepDoorways` | `false` | Boolean; when true, `organic` keeps the doorway pages Vitna otherwise leaves out (see [Doorway pages](#doorway-pages)) |
| `googleShareMs` | Operator default | Integer from 1,000 to 60,000, Google only; caps how long owned Google may spend on this request before timing out or handing off to Bing, never exceeding the operator's `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` |
| `fallbackEngine` | Operator default | `"bing"`, Google only; opts this request into Bing fallback when `VITNA_SEARCH_SERP_FALLBACK_ENGINE` is unset |
| `shape` | `vitna` | `vitna` for the fields under [Parsed response](#parsed-response), or `serpapi` for SerpApi's names, built from the same result (see [SerpApi-shaped response](#serpapi-shaped-response)); JSON only, so not with `format: "html"`, and `serpapi` not with `data_format` or `brd_json` |
| `contents` | Unset | `{ "highlights": true }`, `{ "excerpts": true }` and/or `{ "text": true }`, each `true` or an object with `maxCharacters` (highlights and excerpts also take `query`): read each result page and attach verbatim highlights, excerpts, or its text (see [Page contents](#page-contents)); Vitna's JSON shape only, and only where the operator enables it |
The country and language settings become search-engine URL parameters.
`country` is a requested localization hint, not a verified proxy location.
The operator controls the outbound network and optional proxies. The caller
cannot select a proxy, upstream endpoint, headers, browser profile, or
credentials. Page numbers or offsets select one result page per request;
`maxResults` limits parsed organic entries, not the search engine's total
result count. When `extras: true` is requested, the parsed JSON response also
supplies `people_also_ask` (the questions listed under People Also Ask,
without their answers), `related_searches` (the related queries), and `ads`
(sponsored listings with `position`, `title` and `link`, and `snippet` and
`display_link` where shown), without altering organic ranking. Each is
omitted when the page has none. Google lists up to eight related searches,
fewer or none for some queries, and the list holds what the page showed.
Unknown fields are refused, and request bodies are limited to 8,192 bytes.
Whitespace around the query is removed before retrieval, and a run of
whitespace inside it becomes one space, so queries that differ only in spacing
share a cached answer and a capture in progress. Letter case is kept, since
Google reads `OR` and `AND` as operators only in capitals.
## Page contents
With `contents`, each organic result gains a `contents` object read from the
result's page, so an agent can check an answer without fetching every link.
`highlights` are up to three sentences from the page that hold the query's
terms, best first, quoted exactly, with their offsets in Unicode code points;
search operators (`site:`, `-term`, `OR`, quotes) never choose one, and
`contents.highlights.query` replaces the query for choosing them. `excerpts`
are passages of whole sentences, up to 500 characters each, that best answer
the query: as many as `maxCharacters` holds (1,000, two passages, by default;
at most 4,000), best first, quoted exactly with their offsets, and
`contents.excerpts.query` replaces the query for choosing them (see
[Excerpts](#excerpts)). `text` is the
page's main text, clipped to `maxCharacters`, with `characters`, `truncated` and
`totalCharacters`. Limits match [indexed search's inline contents](overview.md#inline-contents): highlights
default to 1,000 characters (at most 4,000), text to 20,000 (at most 100,000).
```json
{ "query": "\"Lake Lavarone\" elevation", "contents": { "highlights": true, "excerpts": true } }
```
```json
"contents": { "status": "ok", "source": "live", "readAt": "2026-10-17T13:20:05.000Z", "totalCharacters": 2841,
"highlights": [{ "text": "Lake Lavarone lies at an elevation of 1,100 metres.", "start": 0, "end": 51 }],
"excerpts": [{ "text": "Lake Lavarone lies at an elevation of 1,100 metres. In most winters the whole lake freezes over.", "start": 0, "end": 96 }],
"offsetUnit": "unicode_code_points" }
```
A Wikipedia result the search index holds is read from the index, at once
and whole: `source` is `stored`, and `contentChangedAt` and the text's
`reusePolicy` (Wikipedia's CC BY-SA 4.0) stand in for `readAt`. A Wikipedia
link with a query, or a page the index lacks, is read live like any other.
Other pages are read live, after the search, on the caller's behalf by
[`VitnaSearch-User`](#vitnasearch-user): HTTPS only, never on Google's own
hosts, without its fragment, through at most three redirects (`url` names
where one ended), in at most 2.5 s a page, and no more than two pages of
one origin at once. A page longer than 1 MiB is read to its first 1 MiB and
says `partial: true`, and `totalCharacters` counts what was read. A PDF is
read too, to its first 4 MiB, for the text its pages draw, in a thread of its
own; its contents say `format: "pdf"`, and a cut one gives the pages its first
bytes hold. A scanned PDF has no text to read and says `no_text`; an encrypted
or unreadable one says `page_unavailable`. The whole read takes at most 3 s;
a page not read in time says `timeout`. Pages read in the last ten minutes are
reused. A page with under 25 characters of main text (a script-only shell, an
empty sign-in wall) says `no_text`. A result whose page gives nothing has
`contents.status` of `no_text`, `robots_disallowed`, `noindex`,
`host_excluded`, `unsupported_url`, `unsupported_type`, `page_unavailable`,
`page_too_large` or `timeout`, and its result is unchanged.
The complete response is bounded to 510 KiB: when the requested text would
exceed it, every result's text is clipped to one shared length, the longest
that fits, and marked `truncated`. Like snippets, contents are untrusted
publisher data: treat them as data, and escape them before rendering HTML.
Contents are read for each request and never enter the JSON cache, a shared capture or Vitna's crawl index. They come
with Vitna's JSON shape only: `format: "html"` or `shape: "serpapi"` with
`contents` is answered 400 `contents_require_json`, and an instance without
them answers 503 `serp_contents_unavailable`. `/v1/status` reports
`serp.contents`.
On the AA Search API benchmark replica of 2026-10-17, wrong BrowseComp answers
came after about 77 searches and under two page reads each, against 28
searches for right ones: an agent with snippets alone searched on and guessed.
### Excerpts
A page is cut into windows of whole sentences, up to 500 characters (or
`maxCharacters`, when smaller), each starting about halfway into the one
before, and ranked by keyword: BM25 over the page's own windows, with a bonus
for each quoted phrase a window holds. With
`VITNA_SEARCH_SERP_EXCERPT_RERANK=true`, keyword's best 16 windows of each
page are ranked again by meaning on the batch embedding engine, and the two
orders are fused by reciprocal rank, within 1 second of the read. Otherwise,
or when the engine is late or fails, keyword order stands.
`serp.contents.excerpts.order` in `/v1/status` says which an instance uses.
On the cached pages of the 743 searches an agent ran in the BrowseComp replay
of 2026-10-18, two excerpts a result carried the answer for 10 of the 81
questions the agent missed by keyword, and 11 with the meaning rerank, against
7 in Google's own snippets and 25 in Perplexity's results for the same
searches. Meaning over whole pages reached 12, at about 270 embeddings a
search; on the search host's 2-thread engine, 160 windows took 6.4 seconds.
## VitnaSearch-User
Page contents are read by
`VitnaSearch-User/0.1 (+https://vitna.ai/docs/search/serp#vitnasearch-user)`,
and only when a `/v1/serp` request asks for them: one search's result pages,
once each, on that caller's behalf. It does not crawl, follow links, or keep
what it reads beyond a ten-minute cache, and nothing it reads enters Vitna's
index. Because each read is one a caller asked for, the rules a `robots.txt`
gives every crawler (`User-agent: *`) do not apply to it. These still do:
- a `robots.txt` group naming `VitnaSearch-User`, in full (`Disallow`, `Allow`,
`Crawl-delay`), or, when no group names it, one naming Vitna's crawler,
`VitnaSearch`;
- a `Content-Signal` of `search=no` in any group that speaks to the reader:
its own, the crawler's, or every crawler's (`User-agent: *`);
- a page's `noindex`, by `X-Robots-Tag` or by a meta tag named `robots`,
`VitnaSearch` or `VitnaSearch-User`.
The rules are checked again before a page read in the last ten minutes is
reused. A `robots.txt` that cannot be read shows no group naming the reader, so
only a page's own signals apply, and it is asked again after five minutes. One
that redirects is followed to any public HTTPS host, and its rules speak for the
origin asked. Google's own hosts are never read, at any hop, robots.txt
included: not by name, and not at an address Google serves its own services
from, which also covers a site on Google's front ends under its own domain. To
keep the reader off a site:
```text
User-agent: VitnaSearch-User
Disallow: /
```
## URL-based compatibility
A URL-based client can change its API base URL and bearer key while retaining
a supported search URL:
```bash
curl http://127.0.0.1:8789/request \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"url":"https://www.google.com/search?q=lunar+science&hl=en&gl=us","format":"json"}'
```
The URL must use a supported Google host (`https://www.google.com/search` or one
of the 25 supported regional `www.google.*` domains) or Bing vertical path
(`https://www.bing.com/search`, `/news/search`, `/images/search`,
`/videos/search`, or `/shop`), without credentials, a custom port, or a
fragment. It is limited to 4,096 characters. Vitna validates the parameters
and builds a normalized engine request itself. Arbitrary URLs and unsupported
search verticals are refused. A URL-based request must not also supply any
structured search field: `engine`, `query`, `country`, `language`, `page`,
`offset`, `maxResults`, or `googleDomain`.
| Engine | Supported URL parameters |
|---|---|
| Google | `q`, `hl`, `gl`, `start`, `num`, `udm`, `filter`, `tbs`, `tbm`, `uule`, `lr`, `cr`, `safe`, `nfpr`, `brd_json` |
| Bing | `q`, `setlang`, `cc`, `first`, `count`, `filters`, `adlt`, `brd_json` |
Unknown or repeated URL parameters are refused. Google `start` accepts
integers from 0 to 190; Bing `first` accepts integers from 1 to 191.
Google `udm` accepts `2`, `7`, `12`, `14`, or `28`, and `filter` accepts `0`.
Google `num` and Bing `count` must be from 1 to 30. Defaults select the
first page with at most ten parsed results.
The optional legacy `zone` field accepts 1 to 128 ASCII letters, numbers,
underscores, or hyphens and is ignored. It does not select a vendor zone or
proxy. `format: "raw"` selects HTML unless `data_format: "parsed_light"`
or the URL parameter `brd_json=1` requests parsed JSON. `parsed_light` is
the only supported `data_format`; `brd_json` accepts only `1`.
Explicit `format: "html"` cannot accompany either parsed-JSON selector.
For example, this supported migration request returns parsed organic JSON:
```json
{
"zone": "former_serp_zone",
"url": "https://www.google.com/search?q=lunar+science&hl=en&gl=us",
"format": "raw",
"data_format": "parsed_light"
}
```
This compatibility surface is a migration aid, not full compatibility with
every vendor request or output field. Bright Data documents its broader
`/request` shape and parsed response in its
[SERP API reference](https://docs.brightdata.com/api-reference/rest-api/serp/serp-api).
Clients that use additional fields or depend on ads, maps, shopping, news,
images, knowledge panels, or vendor-specific ranks must adapt to Vitna's
organic-only contract.
Google's Custom Search JSON API is closed to new customers and is scheduled
to end on January 1, 2027. Existing customers need a migration plan;
it is not an available new-customer provider for this replacement.
See Google's [Custom Search JSON API notice](https://developers.google.com/custom-search/v1/overview).
## Parsed response
An illustrative JSON response:
```json
{
"general": {
"search_engine": "google",
"query": "lunar science",
"page_title": "lunar science - Google Search",
"timestamp": "2026-10-04T12:00:00.000Z",
"requested_country": "us",
"requested_language": "en",
"search_type": "text",
"mobile": false
},
"input": {
"original_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5",
"final_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5"
},
"organic": [
{
"rank": 1,
"title": "An example lunar science page",
"link": "https://example.com/lunar-science",
"description": "An example excerpt from a search result.",
"display_link": "example.com"
}
],
"meta": {
"provider": "vitna_serp",
"engine": "google",
"requestId": "example-request",
"latencyMs": 500,
"transport": "direct",
"coverage": "organic_only",
"parserVersion": "1",
"provenance": "search_engine",
"offset": 0,
"page": 1,
"count": 1,
"retrieval": {
"route": "direct",
"attempts": [
{ "route": "direct", "transport": "direct", "outcome": "success", "latencyMs": 500 }
],
"cache": { "hit": false, "ageMs": 0 }
}
}
}
```
`organic` follows the recognized organic listing order on the fetched page.
`rank` is the entry's position among recognized organic listings on that page.
Rejected unsafe URLs and removed [doorway pages](#doorway-pages) can leave
rank gaps, and separate listings that link to the same URL retain their
observed positions.
Vitna does not invent a whole-page `global_rank`, total result count,
publication date, or verified location. `description` is search-engine
snippet text. It is not a quote retrieved from the linked page. Missing
descriptions and display links are omitted. `display_link` selects an
observed publisher hostname or breadcrumb. Where Google's citation for a
result holds only counts and an age, as on forum, video and community results
("20+ comments · 4 months ago"), it is the source line Google shows above the
title instead, such as `Reddit · r/buildapc` or `YouTube · Valve`. Dates, ages
and view or comment counts are never display links. A listing can include observed
`sitelinks`; they are associated with the parent result and do not receive
invented organic ranks.
`timestamp` is the transport's fetched-at time when available, otherwise the
response assembly time. `page_title` comes from the fetched page
or is null when unavailable. `requested_country` and `requested_language` appear only
when requested. `input.original_url` is the normalized outbound search URL,
including generated pagination and result-limit parameters. It is not a
verbatim echo of a supplied URL. `input.final_url` is the observed URL after
retrieval and permitted redirects. The observed final URL must retain the
requested engine, query, and offset. A changed query or offset returns an
explicit `serp_query_mismatch` error. `meta.transport` reports the transport
that captured the result; `meta.parserVersion` identifies the
parser implementation. `meta.coverage: "organic_only"` remains present
even when the page contains other features. `meta.offset` records the
zero-based requested offset. `meta.page` appears only when that offset is
a multiple of ten and identifies its ten-result window. `meta.count` is the
returned organic entry count. `meta.doorwaysRemoved` appears when Vitna left
doorway pages out, and counts them.
When the page contains a recognized forward navigation control,
`pagination.next_page_link` carries a normalized continuation URL derived
from that control. It preserves the observed Google `start` or Bing `first`
offset, validates supported search fields, and removes engine tracking
parameters. If the control omits `num` or `count`, the continuation defaults
to at most ten parsed results.
`pagination.current_page` appears only when the requested offset maps to a
ten-result window. Vitna does not invent a next-page URL when no control is
observed. Submit that continuation as `url` to `POST /v1/serp` or
`POST /request`. A structured client can instead use the observed Google
`start` as `offset`, or Bing `first` minus one.
A recognized no-results page may return an empty `organic` list. A challenge,
consent interstitial, unknown page layout, or invalid upstream response is
an explicit failure. It is never represented as a successful empty search.
A page with accepted organic results, or a recognized no-results page, is a
results page. Query text that the engine echoes in its title, suggestions,
related searches, or no-results notice does not make it a challenge.
Challenge elements, consent or sign-in forms, and consent, sign-in, or
`/sorry` landings still fail as blocks. Google's toolbar zero count is read
in any interface language, and its no-results notice is recognized in
English, German, and French. Bing's no-results notice is recognized in
English only.
Google descriptions support known snippet classes and the `data-sncf` or
`data-content-feature="1"` markers inside recognized organic wrappers.
These layout checks improve extraction after retrieval; they do not establish
that Google will accept an outbound route.
For `format: "html"`, the successful response body is the fetched HTML
instead of the parsed envelope, with `Content-Type: text/html` and an
`X-Request-Id` header. It uses the same page recognition and challenge
checks as JSON output. Treat HTML, titles, and descriptions as
untrusted external content. Escape text in a UI and keep raw HTML out of
an application document's trusted execution context.
### Google's notices
`search_information` appears when Google's page carries one of two notices
about its own result list, and is left out otherwise. Neither notice changes
`organic`.
```json
"search_information": {
"omitted_results": { "shown": 1, "included": true, "before": 1 },
"quotes_dropped": { "no_results_for": "\"acme widget\" site:example.com", "results_for": "acme widget site:example.com" }
}
```
`omitted_results` is Google's note under the results that it "omitted some
entries very similar to the 1 already displayed". It is recognized by its
`filter=0` link, so it reads alike in any interface language. `shown` is the
count Google states, or null when the notice holds no single number. Without
`includeOmitted` the notice is reported and nothing more is fetched. With
`includeOmitted: true`, Vitna fetches the page again with Google's `filter=0`
inside the same request's time and answers with it when it holds more
results, doorway pages aside: `included: true`, `before` the first page's count, and an
`input.final_url` that carries `filter=0` while `input.original_url` stays
the first request. Otherwise `included: false` and `reason` says why:
`no_more_results` when the unfiltered page held no more, `serp_timeout` when
the repeat ran out of time (it runs for at most 17 seconds, ends 2 seconds
before the attempt's deadline, and is skipped when that would leave it under
3 seconds), or the repeat's own error code, such as `serp_blocked`. The first
page is answered in each of these cases, at the latest when the repeat's time
is up; only the caller's cancellation or the attempt's own deadline ends the
search. The repeat is a second engine capture within one request,
and its answer is cached apart from a request without `includeOmitted`.
Google showed this notice on 3 of 30 one-result `site:` searches measured on
2026-10-17.
`quotes_dropped` is Google's note above the results that a quoted query
found nothing, so it searched the same words without their quotes:
`no_results_for` is the query as Google echoed it and `results_for` the
query it ran. The organic results are for `results_for`, not for the exact
phrase. It is read on English result pages only.
### Doorway pages
Some sites, most of them hacked small-business sites, publish a page for
each query searched, titled with the query's own words in lowercase at an
address that carries none of them. Google ranks them for long, specific
queries, the kind research agents write. Vitna leaves an entry out of
`organic` when its title has four or more words, every letter a lowercase
English letter, and no word of four letters or more that its link repeats.
A PDF, or a page on a `www.` host, is always kept. Entries are removed before
`maxResults` applies, the rest keep the engine's `rank`, and
`meta.doorwaysRemoved` counts the removed entries. Multi-page jobs leave them
out too. `keepDoorways: true` answers the page as the engine listed it, and
that answer is cached apart.
On 6,575 Google results for 743 searches a research agent ran on
2026-10-18, the rule removed 1,079 (16%). One was a real page, a journal
article with a lowercase title. Each of the 33 searches whose results
carried the answer the agent was after still had it in a result kept.
## SerpApi-shaped response
Code written against SerpApi's Google results can ask for SerpApi's field
names with `shape: "serpapi"`, on `POST /v1/serp` or `POST /request`, with
structured fields or a search URL:
```bash
curl http://127.0.0.1:8789/v1/serp \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"query":"lunar science","country":"us","language":"en","maxResults":5,"extras":true,"shape":"serpapi"}'
```
This shape is built from the same parsed result as the default one, after the
cache and any shared capture, so the two shapes share cached answers and
captures in progress, and authentication, quota, the deadline and `fresh`
work the same for both. It renames and regroups fields and adds no value the
parser did not observe:
```json
{
"search_metadata": {
"id": "6b0d4f0e-7d1c-4c55-9a43-2f1b1c3f9e01",
"status": "Success",
"processed_at": "2026-10-13 09:41:07 UTC",
"total_time_taken": 3.19,
"google_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5"
},
"search_parameters": {
"engine": "google",
"q": "lunar science",
"google_domain": "google.com",
"gl": "us",
"hl": "en",
"start": "0",
"num": "5",
"device": "desktop"
},
"related_questions": [{ "question": "What is lunar science?" }],
"organic_results": [
{
"position": 1,
"title": "An example lunar science page",
"link": "https://example.com/lunar-science",
"displayed_link": "https://example.com › lunar-science",
"snippet": "An example excerpt from a search result.",
"sitelinks": { "inline": [{ "title": "Missions", "link": "https://example.com/lunar-science/missions" }] }
}
],
"related_searches": [{ "query": "lunar science missions" }],
"pagination": {
"current": 1,
"next": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=10&num=10"
},
"meta": {
"provider": "vitna_serp",
"engine": "google",
"requestId": "6b0d4f0e-7d1c-4c55-9a43-2f1b1c3f9e01",
"latencyMs": 3187,
"transport": "browser",
"coverage": "full",
"parserVersion": "1",
"provenance": "search_engine",
"offset": 0,
"page": 1,
"count": 1
}
}
```
| SerpApi field | From | Notes |
|---|---|---|
| `search_metadata.id` | `meta.requestId` | This request's identifier; a cache hit has its own |
| `search_metadata.status` | | `Success`, or `Error` for a search that failed after admission |
| `search_metadata.processed_at` | `general.timestamp` | When the page was captured, written `YYYY-MM-DD HH:MM:SS UTC`; a cache hit keeps its capture's time |
| `search_metadata.total_time_taken` | `meta.latencyMs` | Seconds, to two places |
| `search_metadata.google_url` | `input.final_url` | `bing_url` instead when Bing answered |
| `search_parameters` | The request | `engine`, `q`, `google_domain` (always `google.com`), `gl`, `hl`, `start`, `num`, `device` (always `desktop`) and `udm` when sent, as strings; a Bing request gives `cc`, `setlang`, `first` and `count` |
| `organic_results[].position` | `rank` | The position on the page; a rejected listing leaves a gap |
| `organic_results[].title`, `.link` | `title`, `link` | |
| `organic_results[].displayed_link` | `display_link` | The breadcrumb, or on a forum, video or community result the source line (`Reddit · r/buildapc`); omitted when the page shows neither |
| `organic_results[].snippet` | `description` | Omitted when the page shows none |
| `organic_results[].sitelinks.inline` | `sitelinks` | `title` and `link` each; the parser reads no sitelink snippets and does not tell Google's expanded block from its inline row |
| `related_questions[].question` | `people_also_ask` | With `extras: true`; the question only |
| `related_searches[].query` | `related_searches` | With `extras: true` |
| `ads[]` | `ads` | With `extras: true`: `position`, `title`, `link`, `displayed_link`, and `description` from the ad's `snippet` |
| `pagination.current`, `.next` | `pagination.current_page`, `.next_page_link` | `current` only for an offset on a ten-result page |
| `meta` | `meta` | Vitna's own, unchanged |
SerpApi fields Vitna does not parse are left out rather than sent empty: an
organic result's `date`, `source`, `favicon`, `thumbnail`,
`snippet_highlighted_words`, `redirect_link` and `rich_snippet`; the answers
to People Also Ask and the links of related searches; an ad's `tracking_link`,
`block_position` and sitelinks; `search_information`, which carries result
counts (Vitna's own `search_information`, [Google's notices](#googles-notices),
is not mapped into this shape); `pagination.other_pages` and
`serpapi_pagination`; and page features
such as the AI Overview, perspectives, the knowledge graph, the answer box,
local results, top stories and images. A recognized page with no results has
an empty `organic_results` and status `Success`, where SerpApi adds an
`error` string.
Where the operator sets `VITNA_SEARCH_SERP_FALLBACK_ENGINE=bing`, a Google
search that Google does not answer can be answered by Bing. Both shapes say
so in `meta.engine`, which is then `bing`. In this shape `search_metadata`
gives `bing_url` in place of `google_url`, while `search_parameters` still
records the Google request.
A search that fails after admission keeps its HTTP status, `Retry-After` and
Vitna's error, and adds `search_metadata` and `search_parameters`:
```json
{
"search_metadata": { "id": "c2e5b8a1-4f3d-4e6b-9a7c-1d2e3f4a5b6c", "status": "Error", "total_time_taken": 0.01 },
"search_parameters": { "engine": "google", "q": "lunar science", "google_domain": "google.com", "start": "0", "num": "10", "device": "desktop" },
"error": { "code": "serp_routes_busy", "message": "all configured routes for this search engine are in use" },
"meta": { "requestId": "c2e5b8a1-4f3d-4e6b-9a7c-1d2e3f4a5b6c", "engine": "google", "latencyMs": 8, "retrieval": { "attempts": [] } }
}
```
`error` stays an object with `code` and `message` rather than SerpApi's
string, because the code says whether and when to retry. A request refused
before a search begins, for invalid input, authentication or quota, keeps the
shared error envelope in either shape.
The Web API's MCP server offers this route to agents as the `google_search`
tool, with SerpApi's argument names; see [MCP](../web/overview.md#mcp).
## Measured delivery
Three runs of live Google searches on the production search host, after its
Google retrieval moved to residential exits alone on the evening of
2026-11-03 (Eastern time). Each run was a separate process on that host
running the deployed SERP code as the service's own user, with the service's
configuration and browser daemon: `engine: "google"`, `maxResults` unset (so
5), `fresh: true` to skip the JSON cache, and one call a query. The first two
kept three searches in flight and the third twelve. The queries were
technical questions (configuration keys, flags, command lines and named
techniques), and none appears in more than one run. A search counts as
answered when it returned without an error. Every search, with its query,
result count and time, is in
[serp-runs-2026-11.json](https://vitna.ai/search/serp-runs-2026-11.json).
| Finished (ET) | In flight | Searches | Answered | 95% CI | All five results | Median | p90 | p95 | Slowest |
|---|---|---|---|---|---|---|---|---|---|
| 2026-11-03 21:57 | 3 | 100 | 97 | 91.5 to 99.0% | 89 | 4.01 s | 4.43 s | 9.30 s | 20.44 s |
| 2026-11-04 00:23 | 3 | 200 | 199 | 97.2 to 99.9% | 186 | 3.89 s | 4.23 s | 4.87 s | 20.86 s |
| 2026-11-04 15:42 | 12 | 2,500 | 2,273 | 89.7 to 92.0% | 2,256 | 4.19 s | 9.66 s | 15.96 s | 22.37 s |
Times are over every search, refusals included, from the call to the parsed
result on the search host. They leave out a caller's network round trip, the
HTTP layer, authentication and quotas, so a request from elsewhere takes
longer. The intervals are Wilson's; across the three runs 2,569 of 2,800
answered (90.7 to 92.7%), and across the two at three in flight 296 of 300
(96.6 to 99.5%). The first run followed 16 warm-up searches sent two at a
time straight after the change and restart, of which 11 answered; they are
not counted.
Every refusal was returned as the 503 the
[failure table](#authentication-quotas-and-failures) lists rather than as an
empty `organic` list. In the runs at three in flight each was `serp_blocked`
(8.99 to 17.97 s); in the 200-search run it was the first search sent. In
the run at twelve, 222 were `serp_blocked` (6.87 to 21.75 s) and 5
`serp_timeout` (18.78 to 22.37 s), and a refusal took a median of 14.89 s
against 4.16 s for an answer. They were not spread evenly: 184 of the first
250 searches sent were answered, and 2,089 of the 2,250 after them. The run
did not record why.
An answer can carry fewer than five results. In the 200-search run, 10
answers lost one or two to [doorway pages](#doorway-pages) (11 pages left
out), and three carried 3, 1 and 0 results for a reason the run did not
record. In the 100-search run, all 8 short answers were doorway removals. In
the 2,500-search run, 16 answers lost one each to doorway pages and one
carried 4 for a reason the run did not record. The runs recorded no attempt
counts, so how many answers needed a second try is not known; 94 of 100,
193 of 200 and 2,209 of 2,500 finished under 7 s.
Five is the default because Google unlocks a page's `/goto` result links in
turn and a page waits for its slowest link: on 2026-11-03, links 1 and 2
answered at once, links 3 to 5 within about 2.3 s of the page settling, link 6
after about 7.6 s and links 7 to 10 after about 24 s (see
[Run on owned infrastructure](#run-on-owned-infrastructure)). Earlier
100-search runs that week at `maxResults: 10` took 26.1 s at the median.
These are three runs on one host within a day, on one kind of query. They
describe those runs, not coverage of the wider web or a guarantee of
availability, and Bing was not measured. The runs called the retrieval in
the host's own process, so per-key quotas did not apply: twelve in flight is
what the host sustained, not what one key may send.
## Retrieval routes and retries
The operator chooses `auto`, `direct`, or `browser` transport. The following
describes ordinary routes; the experimental owned adapter below replaces
Google's routes when configured. With `auto` and no configured proxies,
Vitna prefers a JavaScript browser for Google
and direct HTTPS for Bing, with bounded fallback to the other available
transport. With configured proxies, `auto` uses only browser routes pinned
to those proxies. Explicit `browser` uses a browser route per proxy,
or a local browser route when no proxies are configured. Configured proxy
routes never fall back to local egress. Explicit `direct` uses direct HTTPS
and refuses proxy configuration. Proxy pools rotate their starting route
across requests while respecting route health and capacity.
SERP browser captures allow ordinary GET images and fonts through the same
public-address guard and request, byte, and deadline limits. This internal
option does not allow arbitrary POSTs, media, or WebSockets. The requested
Google `hl` or Bing `setlang` language is validated and forwarded to the
browser locale; without a language hint the browser keeps its English default.
These settings do not establish that Google will accept an egress route.
Pinning a proxy URL keeps its endpoint and effective credentials consistent
throughout one browser capture. Structured proxy entries can generate a new
provider session credential for each admitted attempt. For browser captures,
the provider must keep the same exit for that whole context, including
redirects and subresource connections. A gateway that rotates on each
connection can change exits within one capture even though its configured
URL stays the same. Vitna does not verify the provider's exit IP or session
stickiness. Route health and pacing track declared egress groups, not
measured public exit IPs.
The browser transport expects an HTTP(S) CONNECT tunnel to the checked
destination address and retains target TLS certificate verification. Managed
scraping proxies that intercept target TLS or return pages fetched by their
own browser need a separate integration. For example, Crawlbase's
[Smart AI Proxy documentation](https://crawlbase.com/docs/smart-proxy)
states that it intercepts TLS and asks clients to disable verification.
Vitna does not automatically disable verification for these endpoints.
The default request tries at most three distinct routes, bounded by the
available routes and the shared whole-request deadline. Each attempt has a
separate four-second default cap within the remaining deadline. Only blocks,
unreachable engines, attempt timeouts, and unrecognized pages permit another
route attempt. Requests keep the selected engine and query throughout.
Vitna returns the first validated result page; it does not combine rankings
from several routes or change Google requests to Bing requests.
Route concurrency and health are tracked in the current service process.
The experimental owned Google adapter below adds a persistent admission gate.
A route allows one active request per engine by default. Egress groups also
allow one active request per engine by default; direct HTTPS and the local
browser share that group. The first actual upstream block cools matching
routes for that engine/egress group for five minutes by default. A redirect
to the engine's own consent or sign-in page counts as a block. A browser
challenge flag counts only with HTTP 403 or 429; otherwise the attempt fails
as `serp_invalid_response` without a cooldown or another attempt. Repeated
blocks double that cooldown, up to a one-hour default cap. A valid upstream
capture on the same group resets its block streak; denied requests and cache
hits neither reset nor escalate it. Other transient failures use a
thirty-second default cooldown. A Google cooldown does not
block Bing. If an uncached request cannot begin an attempt because all
eligible routes are occupied or cooling, the service returns HTTP 503 with
`Retry-After`. When attempts have already failed, their final retrieval
failure is returned. Restarting the process resets ordinary route health.
Ordinary service replicas do not share these pacing, concurrency, or cooldown
limits.
If replicas use the same physical exit, coordinate retrieval starts across
them or run one retrieval process for that exit; adding replicas otherwise
multiplies its outbound traffic.
Independent retrieval starts are paced per engine and egress group. The
default interval is selected uniformly between five and fifteen seconds,
averaging roughly six starts per minute for each engine/egress group.
Direct HTTPS and the local browser share both pacing and the egress
concurrency slot. Fresh requests follow this pacing; cache hits can return
without starting a new retrieval. Internal fallback within one logical
request can reuse an egress only after its previous upstream attempt and
cleanup have settled, within the same request deadline.
The API does not queue requests until their next start time. If every
otherwise eligible route is paced, it returns HTTP 503
`serp_routes_throttled` with `Retry-After` in seconds and
`meta.retrieval.attempts: []`.
Clients can retry after that interval. Busy routes take precedence.
Otherwise, when waiting routes differ, the code and `Retry-After` describe
the route that can start soonest, whether it is paced, cooling, or awaiting
its runtime retry; equal start times report pacing. A paced route that is
due sooner is never reported as another egress's longer cooldown. The shared
thirty-requests-per-minute client limit caps admission; it does not promise
that many upstream starts. Ordinary route pacing and egress slots are
process-local;
horizontally scaled instances do not coordinate them.
Browser dependencies load lazily when a browser attempt is needed. Starting
the service or reading its status does not launch a browser or probe a
search engine. Missing browser dependencies can therefore produce an
explicit runtime-unavailable response on a browser attempt.
That response pauses the affected route, for every engine it serves, for
sixty seconds.
Requests in that window return HTTP 503 `serp_runtime_unavailable` with
`Retry-After` and no new attempts. The first request after it makes one
fresh attempt, so a slow browser start does not disable the route until
restart. A request never retries the same route. A missing dependency keeps
failing; repair the runtime setup and restart before verifying that route
again.
## JSON cache and freshness
Successful parsed JSON responses can be reused by the same client for
thirty seconds by default, and for up to one hour when the operator sets
`VITNA_SEARCH_SERP_CACHE_TTL_MS`. The cache is process-local and bounded to 128
entries and 8 MiB of serialized responses. Entries are kept per API
client, including pages its multi-page jobs capture, so a cache hit, its
age, and its capture timestamp never reveal another client's search.
HTML and failed retrievals are not cached. Responses still carry HTTP
`Cache-Control: no-store`; the short-lived cache belongs to the service,
not an intermediary or the caller's browser. When a larger `maxResults` capture
for the same client and query parameters is already cached, smaller `maxResults`
requests slice from that superset entry (`cacheEvent: "superset"`). Operators
can also configure stale-while-revalidate (`VITNA_SEARCH_SERP_CACHE_SWR_MS`),
stale-if-error (`VITNA_SEARCH_SERP_CACHE_STALE_IF_ERROR_MS`), and an optional
WAL-mode SQLite L2 store (`VITNA_SEARCH_SERP_CACHE_SQLITE_PATH`) so warm JSON
entries survive process restarts. Live delivery and latency gates can be
evaluated against a running host with `search:serp:benchmark`:
```bash
npm run search:serp:benchmark -- --url https://search.vitna.ai --fresh
```
Set `fresh: true` to bypass a cached response. A successful new capture
refreshes that cache entry. Cached results retain their
original engine capture timestamp; a cache hit does not imply the search
engine was contacted for that request. Cached requests still use the shared
API authentication, quota, and concurrency controls. Restarting the process
clears the cache. Its transient in-memory query/result data is separate
from receipts, which omit queries and result content.
When the cache is full, its least recently used entry leaves first, even
before its TTL ends. Captures made with `fresh: true` are stored too, so a load
test that searches fresh can push other clients' entries out. The status
`cache.evictions` count shows how often that happens. Operator metrics count
each answered request by where its page came from (`capture`, `cache`, or
`shared` for a joined capture, below), split by test and real traffic in
`vitna_api_serp_answer_sources_total`. Together they show how often searches
repeat within the TTL before a longer one or a durable store is worth having.
Matching in-flight JSON requests from the same client can share one capture
in the current process. The canonical engine, trimmed query, effective
offset, country, language, result limit, JSON format, and `fresh` flag must
match. Fresh requests share only with matching fresh requests. Another
client's identical request never joins; it needs its own route capacity
like any other request. Each caller still uses
its own API authentication, quota slot, request deadline, cancellation,
request identifier, and latency. Cancelling one caller leaves the capture
available to the others; when the last caller leaves, the producer is
aborted. A shared capture runs on the route's whole timeout so later callers
can join it. A route that rotates its own exits spends every second it is
handed on more tries, so it gets no more than the latest waiting caller
still wants. A lone Google search beside a Bing fallback hands the owned
route 49 of the 55 s, so its tries come back before it leaves. Until
2026-10-12 the route got all 55, and a search that ran to its deadline lost
its tries when its abandoned capture was aborted. HTML captures are
separate, and failures never become cached responses.
The sharing pool is bounded to 128 active captures and 32 pending callers
per shared capture, including its first caller. An abandoned producer keeps
its capacity reservation until it settles. Capacity refusals use HTTP 503
`serp_routes_busy` with `Retry-After: 1` and no new attempts.
`meta.retrieval.route` identifies the route that captured the returned page.
`meta.retrieval.attempts` lists this request's attempts with `route`,
`transport`, `outcome: "success"` or `"failure"`, `latencyMs`, and an error
`code` for failures. A cache hit has no new attempts. The sibling `cache`
object reports `hit` and the cached entry's `ageMs`; a new capture reports
`hit: false, ageMs: 0`. Route identifiers such as `direct`,
`browser-direct`, and `proxy-1` are safe labels, not proxy addresses.
Joined callers receive `meta.retrieval.coalesced: true` with `attempts: []`;
the first caller receives `coalesced: false`.
Authenticated `GET /v1/status` exposes `serp.retrieval` with configured
attempt limits, cooldowns, cache counts, bytes and evictions, `minIntervalMs`,
`maxIntervalMs`, `maxConcurrentPerEgress`, `maxBlockedCooldownMs`,
`runtimeRetryMs`, and the safe engine transport preference map
`preferredTransports`. Its
`coalescing` object reports `active`, `waiters`, `maxEntries`, `maxWaiters`,
and `hits`, without queries or key hashes. Its `routes` list includes
safe route identifiers, transports, and per-engine state. Each engine state
reports `ready`, `busy`, `cooldown`, `throttled`, or `unavailable`, active
route count `inFlight`, shared egress count `egressInFlight`,
attempt/success/failure counters, `consecutiveBlocks`, `cooldownUntil`,
`unavailableUntil`, `nextRequestAt`, and `lastSuccessAt`.
The latter timestamps use Unix milliseconds, with no success recorded as
null. These are process-local operational observations; a `ready` route
has capacity and is neither cooling nor paced, without an upstream
availability probe.
## Multi-page Google jobs
Use an asynchronous job to request up to 100 distinct organic Google links
across at most ten observed result pages. The existing `POST /v1/serp`
and `POST /request` remain single-page requests with `maxResults` at most 10.
Multi-page jobs return parsed JSON and support Google only.
```bash
curl http://127.0.0.1:8789/v1/serp/jobs \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"engine":"google","query":"lunar science","maxResults":100,"maxPages":10,"timeoutMs":300000}'
```
| Field | Default | Allowed values |
|---|---|---|
| `engine` | `google` | `google` only |
| `query` | Required | Same validated query string as single-page SERP retrieval |
| `country`, `language` | Unset | Same localization hints as single-page requests |
| `maxResults` | 100 | Integer 1 to 100; target distinct result count |
| `maxPages` | 10 | Integer 1 to 10; maximum captured result pages |
| `timeoutMs` | 300,000 | Integer 1,000 to 300,000; whole-job deadline, including queue time |
| `fresh` | `false` | Boolean; `true` bypasses completed-job reuse and requires a new collection unless an identical fresh job is already active |
New collection jobs begin at offset zero and make fresh, sequential, isolated captures.
They follow the recognized `pagination.next_page_link` from each fetched
page, rather than asking Google for `num=100` or inventing offsets. A page
with fewer than ten results can still have an observed continuation. Each
attempt has a deadline of at most sixty seconds within the remaining job
time and the instance's configured shared and SERP deadlines.
Blocks and cooling stop the job instead of switching engine or bypassing
the connection's penalty. Busy or paced routes use the exact private
millisecond wait when available, subtracting cleanup time already spent;
otherwise they use the public `Retry-After` seconds. The durable admission
gate is rechecked before capture. Waits stay within the job deadline, with
at most twenty tries for a logical page. The private timing hint is not an
API field. A page refused admission because every service slot is in use
(`service_busy`) or the per-minute limit is reached (`rate_limited`) also
waits its `Retry-After` within the job deadline, with at most twenty
admission tries. A daily limit or unavailable request accounting stops
the job.
Creation returns HTTP 202 with a `job` envelope, a `Location` status path,
and `Retry-After: 1`. Its `id` identifies the job; a new job begins `queued`,
while a reused job retains its existing status. `progress` reports `pagesCompleted`,
`returnedResults`, `requestedResults`, and `maxPages`. `result` appears when
captured results are available, and an optional `error` contains a safe code.
Jobs expose creation, start, finish, and expiry timestamps when applicable.
An admitted job becomes `running` when its worker starts.
With `fresh: false`, an exact same-client request can reuse a successful
`completed` job for thirty seconds after its finish time. Only jobs that
reached their requested target without an error qualify; partial, failed,
cancelled, and interrupted results are not reused. Engine, trimmed query,
localization hints, result target, page limit, and job deadline must match.
The original job ID, results, and capture timestamps remain intact.
Identical same-client `queued` or `running` requests with the same `fresh`
flag join that one job. This is one shared cancellable job: cancelling its
ID stops it for every same-client caller using it. `fresh: true` bypasses a
completed result and joins only an identical active fresh job. Reuse does
not start more page captures or consume more page quota, but each POST still
uses the job-control allowance.
The creation response adds `job.reuse` with `kind: "none"`, `"active"`, or
`"completed"`, `requestedFresh`, `ageMs`, and `ttlMs`. Read responses omit
that request-specific reuse marker. Stable `job.freshness` records the
original job's `requestedFresh` policy and latest page `fetchedAt`, when
available; reused results retain those capture times.
Use the same client bearer authentication to read or cancel that job:
```text
GET /v1/serp/jobs/{id}
DELETE /v1/serp/jobs/{id}
```
Creating and reading jobs share a separate limit of sixty requests per
minute per client. Cancellation has its own ten-per-minute allowance so
exhausted polling does not prevent cancellation. These controls do not
consume query quota. Reads return
HTTP 200 and cancellation returns HTTP 202, both with `{job}`. Unknown,
expired, evicted, or another client's jobs return HTTP 404 `job_not_found`.
Each admitted logical page consumes one shared SERP quota slot, including
bounded route waits, with its concurrency slot held through cleanup. Job
reads and cancellation remain available while page slots are occupied. Requesting
100 links does not purchase or guarantee 100 results. Cancellation preserves
captured partial results. A running job can report `cancelling` while its
worker remains occupied until the actual browser and network cleanup settles.
The aggregate `result` contains `general`, `organic`, `pages`, and `meta`.
Duplicate detection strips URL fragments. For exact `/watch` URLs on
`youtube.com`, `www.youtube.com`, or `m.youtube.com` with one valid
11-character `v` value and no nondefault port, it also ignores `xstg`.
Other query parameters, their encoding and ordering, and host aliases
remain distinct. The first observed entry wins with its original link and
discovery order; recovery preserves saved legacy results.
Single-page SERP output preserves its observed listing rows.
Each organic entry retains its page-local `rank`
and adds `position` for the aggregate discovery order. Its `source` records
the sequential capture `page`, actual observed `offset`, original `pageRank`,
`fetchedAt`, and `serpUrl`.
That aggregate position is not an invented Google global rank. `pages`
retains each capture's URL, time, result counts, and safe metadata.
Aggregate `meta.timingMs` separates observed `admission`, `retrieval`,
`pacingWait`, and `checkpoint` time. Admission includes waits after a busy
or per-minute refusal. Retrieval includes backend capture and
physical cleanup; these are measured phases, not search-quality scores.
Compare a fresh job separately from completed-result reuse. Google listings
and snippets are a different output from native index matches or the
research API's verbatim cited passages. An owned-route provider fee of zero
excludes infrastructure, browser, network, storage, crawling, and operating
costs.
Only reaching the requested distinct-result target produces `completed`,
`meta.complete: true`, and `meta.stopReason: "target_reached"`. Fewer results
produce `partial` when usable links are retained, including `no_next_page`,
`page_limit`, or a retrieval error; no usable results finish `failed`.
Other stop reasons include `blocked`, `cooling`, `deadline` when the
whole-job deadline ends collection, `timeout` when a page capture times out
with job time remaining, both with error code `serp_timeout`, and
`cancelled`. Failed, cancelled, or interrupted jobs retain usable partial
results when available. The limits and observed continuation do not guarantee
that Google will deliver 100 links.
One authenticated live job on 2026-10-05 at 12:37:07 UTC was accepted with
HTTP 202 and requested 100 links across at most ten pages. It returned
`partial` with 94 unique publisher URLs in 107,047 ms, ending with
`meta.stopReason: "page_limit"` and `meta.complete: false`. The ten captured
offsets were 0 through 90 in steps of ten. Their counts were 7, 10, 10, 10,
10, 10, 10, 9, 10, and 10: 96 raw results with two duplicate links removed.
All captures used `owned-google`; result source references matched their
captured pages, with no reported error or challenge. The local stock browser
used the owned OVHcloud exit under the same durable admission gate.
This verifies one multi-page API run. The new multi-page console controls
were not part of that live check; an authored real-Edge fixture separately
verified creation, sequential polling, result cards, and page provenance.
It does not establish sustained production
reliability, geographic targeting, or guaranteed delivery of 100 links.
The manager allows one active worker, eight queued jobs, at most 64 retained
records, and two nonterminal jobs per client. Records expire one hour
after creation. Under load, finished results can be evicted sooner: when
all 64 records are in use, a new job removes a finished one, the submitting
client's oldest first, otherwise the oldest of any client. Queued and
running jobs are never evicted.
Service startup defaults to memory. Opt into private JSON
snapshots with `VITNA_SEARCH_SERP_JOBS_DIR=.search-data/serp-jobs` locally,
or `/var/lib/vitna-search/serp-jobs` on a Linux service host. These snapshots
contain queries and results; protect their directory as private operator
data. An exclusive writer lease permits only one service per jobs directory.
Its holder refreshes the lease every thirty seconds. A restart reclaims a
same-host lease whose process is gone, whose PID now names a later process
on Linux, or whose PID is the restarting process itself, as PID 1 finds
after a container restart. A lease left unrefreshed for two minutes is stale
from any host, so a replacement container can start once its predecessor's
lease is that old. A service that finds its lease replaced or removed stops
admitting jobs and writing snapshots.
A restart marks recovered
nonterminal jobs `interrupted`, preserving saved partial results without
automatically resuming retrieval. Recovery deletes an expired snapshot once
its ID and expiry are readable, without validating the rest, and accepts
timestamps that a wall-clock step or host sleep left out of order.
Authenticated status exposes these availability, persistence, queue, record,
and limit counts under `serp.collection`.
## Run without an index
SERP retrieval does not require a SQLite search index or a crawler worker.
Mint client credentials as described in the
[search overview](overview.md#starting-an-alpha-instance), then enable the
SERP capability in the service's private environment:
```dotenv
VITNA_SEARCH_CLIENTS={"pilot":"REPLACE_WITH_CLIENT_KEY_SHA256"}
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=auto
VITNA_SEARCH_SERP_TIMEOUT_MS=10000
```
Enabling retrieval does not authorize redistribution.
`VITNA_SERP_RESALE_APPROVED=true` records the operator's decision that
collecting Google and Bing results and redistributing them through Vitna's
API, including fetched HTML, is permitted. Keep both settings unset until
that decision is made. Startup and the live checker refuse
`VITNA_SEARCH_SERP_ENABLED=true` without this confirmation, rather than
running with SERP retrieval silently off.
Start the service:
```bash
node --env-file=.env.search platform/search/start.mjs
```
The direct transport uses the built-in Node runtime and makes bounded HTTPS
requests to supported search engines. It requires no Bright Data API key,
zone, search adapter, or provider account. Starting the process makes no
search-engine probe.
| Environment variable | Default | Meaning |
|---|---|---|
| `VITNA_SEARCH_SERP_ENABLED` | `false` | Enable independent Google/Bing SERP retrieval; also requires `VITNA_SERP_RESALE_APPROVED=true` |
| `VITNA_SERP_RESALE_APPROVED` | `false` | Operator confirmation that collecting Google/Bing results and redistributing them through the API is permitted; `true` or `false` |
| `VITNA_SEARCH_SERP_TRANSPORT` | `auto` | Bounded direct/browser route selection; explicit `direct` or `browser` also supported |
| `VITNA_SEARCH_SERP_TIMEOUT_MS` | 10,000 | Integer 100 to 60,000; retrieval deadline also bounded by the shared request deadline |
| `VITNA_SEARCH_SERP_PROXIES` | Unset | Operator JSON array of at most 20 HTTP(S) proxy URL strings or structured route entries; requires `auto` or `browser` transport |
| `VITNA_SEARCH_SERP_ATTEMPTS` | 3 | At most 1 to 5 distinct attempts per request, within one deadline |
| `VITNA_SEARCH_SERP_ATTEMPT_TIMEOUT_MS` | 4,000 | Integer 100 to 60,000; per-attempt cap within the remaining deadline |
| `VITNA_SEARCH_SERP_ROUTE_CONCURRENCY` | 1 | Integer 1 to 4; concurrent requests per route/engine pair |
| `VITNA_SEARCH_SERP_EGRESS_CONCURRENCY` | 1 | Integer 1 to 4; concurrent requests per engine/egress group; shared by direct HTTPS and local browser |
| `VITNA_SEARCH_SERP_MIN_INTERVAL_MS` | 5,000 | Integer 0 to 60,000; minimum jittered interval between independent starts |
| `VITNA_SEARCH_SERP_MAX_INTERVAL_MS` | 15,000 | Integer 0 to 60,000; maximum interval; must be at least the minimum |
| `VITNA_SEARCH_SERP_COOLDOWN_MS` | 30,000 | Integer 1,000 to 300,000; transient-failure route cooldown |
| `VITNA_SEARCH_SERP_BLOCKED_COOLDOWN_MS` | 300,000 | Integer 1,000 to 3,600,000; engine/egress cooldown after a block |
| `VITNA_SEARCH_SERP_MAX_BLOCKED_COOLDOWN_MS` | 3,600,000 | Integer 1,000 to 3,600,000; repeated-block cooldown cap; must be at least the base blocked cooldown |
| `VITNA_SEARCH_SERP_CACHE_TTL_MS` | 30,000 | Integer 0 to 3,600,000; successful JSON reuse window, at most one hour; 0 disables reuse |
| `VITNA_SEARCH_SERP_CACHE_ENTRIES` | 128 | Integer 0 to 1,024; maximum cached responses, also bounded by 8 MiB; 0 disables caching |
| `VITNA_SEARCH_SERP_JOBS_DIR` | Unset; in-memory jobs | Optional private JSON snapshot directory, such as `.search-data/serp-jobs` locally or `/var/lib/vitna-search/serp-jobs` on Linux; one service writer per directory |
| `VITNA_SEARCH_SERP_BROWSER_PROFILE` | Inherited Unblocker profile, otherwise `native` | `native` or `cloakbrowser`; the latter requires a browser executable path |
| `VITNA_SEARCH_SERP_BROWSER_CHANNEL` | Inherited Unblocker channel, otherwise bundled Chromium | Optional installed browser channel, such as `msedge` |
| `VITNA_SEARCH_SERP_BROWSER_PATH` | Inherited Unblocker path, otherwise Playwright default | Optional operator-selected browser executable path |
| `VITNA_SEARCH_SERP_OWNED_SSH_HOST` | Unset | Experimental Google route: operator-configured SSH host alias, such as `owned-host`; requires `auto` or `browser` |
| `VITNA_SEARCH_SERP_OWNED_EGRESS_ID` | Unset | Required declared physical exit label, such as `owned-exit`; follows the proxy egress-label rules |
| `VITNA_SEARCH_SERP_OWNED_BROWSER` | `stock` | Experimental owned browser: `stock` uses the local installed browser; `camoufox` uses a separately provisioned remote Linux worker |
| `VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT` | Unset | Required for `camoufox` only: private remote installation root `/tmp/vitna-firefox.` followed by 8–32 letters, digits, underscores or hyphens |
| `VITNA_SEARCH_SERP_OWNED_STATE_PATH` | `.search-data/serp-browser-check-state.json` | Private persistent gate shared with matching diagnostics; relative paths resolve from the service working directory |
| `VITNA_SEARCH_SERP_OWNED_PROFILE_ROOT` | `.search-data/browser-controls` | Private root for fresh local stock-browser profiles; relative paths resolve from the service working directory; unused by `camoufox` |
| `VITNA_SEARCH_SERP_OWNED_MIN_INTERVAL_MS` | 5,000 | Integer 1,000 to 60,000; minimum interval for the owned route's persistent admission gate |
| `VITNA_SEARCH_SERP_OWNED_MAX_INTERVAL_MS` | 15,000 | Integer 1,000 to 60,000; maximum interval; must be at least the owned minimum |
| `VITNA_SEARCH_SERP_OWNED_ROTATING_EXITS` | `false` | `camoufox` only: the remote daemon rotates, spaces and cools its own residential exits, so the route neither paces nor rests (its interval settings are unused) and a failed search records a strike only |
| `VITNA_SEARCH_SERP_OWNED_CONCURRENCY` | 1 | Integer 1 to 32; captures the route runs at once; above 1 requires rotating exits. Slot N holds its lease in `<state path>.slot-N` |
| `VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON` | `false` | `camoufox` only: send each search to the daemon's own socket (`nc -U` over the SSH host alias, or a local Unix socket when `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET` is also set), instead of starting the private worker, which saves about 150 ms a search. Without `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET`, needs a daemon running in the root and OpenBSD `nc` on the remote host. Route N sets its own `_N_` value |
| `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET` | Unset | `camoufox` with `VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON=true` only: safe absolute `.sock` path (at most 100 characters, such as `/run/vitna-camoufox/daemon.sock`) when Search and the Camoufox daemon run on the same host, connecting over a local Unix domain socket with no `ssh` or `nc` process. When set, `VITNA_SEARCH_SERP_OWNED_SSH_HOST` and `VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT` are optional. Route N sets its own `_N_` value |
| `VITNA_SEARCH_SERP_FALLBACK_ENGINE` | Unset | `bing` only: a Google search the owned route does not answer, for any reason but a cancelled request (blocked, cooling, throttled, busy, unreachable, an invalid answer, or out of its time), goes to Bing with the rest of the deadline. Until 2026-10-12 only the first four and running out of time did, and over 4,500 load-test searches that night 43 failed outright without trying Bing: 25 on busy routes, 15 unreachable, 3 invalid answers. Owned Google then stops 6 s before the deadline (at most a quarter of it) so Bing can still answer, and a Bing route still in its pacing interval is waited for whenever its next turn leaves 1.5 s to answer, however many turns other searches take first; until 2026-10-18 a search that lost one turn to another failed, about 5% of the AA benchmark's searches at a 30 s Google share. A Bing route busy with another search's hand-off is polled every 100 ms until it frees, while 1.5 s would still be left; until 2026-10-13 the second of two searches handed over together failed at once as `serp_routes_busy`. On 2026-10-10/11, 41 of 500 long-tail searches spent the whole 55 s on Google and failed without trying Bing, while 10 Google deliveries came in the last 6 s; a free Bing route answered in about 0.4 s |
| `VITNA_SEARCH_SERP_CONTENTS` | `false` | `true` lets a request ask for [page contents](#page-contents), read live from each result page; `false` answers such a request 503 `serp_contents_unavailable` |
| `VITNA_SEARCH_SERP_EXCERPT_RERANK` | `false` | `true` ranks [excerpts](#excerpts) by meaning as well as keyword. It needs the embedding settings and `VITNA_SEARCH_EMBED_BATCH_URL`, since the engine answers one request at a time and a search's query embedding would otherwise wait behind excerpt windows |
| `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` | Unset | 1000 to 60000: the most of a search's time owned Google gets before timing out or handing off to Bing, never more than the deadline less any Bing reserve. Unset, Google gets all of that. The answered p90 is the searches Google answers late or never. By day on 2026-10-12, 102 of 457 Google answers came after 10 s, and the 8.6% Bing answered came at 37 to 49 s. Replayed with a 20 s share, the p90 fell from 32 to 21 s and Google's share from 91 to 84%; with 15 s, to 16 s and 81%. Overnight, when far more searches miss Google, a 20 s share cut Google from 66 to 49%. Bing's route takes a start every 5 to 15 s, so check it keeps up with what the share sends it |
| `VITNA_SEARCH_SERP_GOOGLE_SHORT_SHARE_MS` | Unset | 1000 to 60000: overrides `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` when `maxResults` is 5 or fewer, never more than the deadline less any Bing reserve. Under `--goto-respect-num 1 --goto-pace-ms 1000`, `maxResults <= 5` stays inside Google's 5-link zero-hold `/goto` regime (~3.8 to 4.1 s delivered p50) while `maxResults = 6` hits a ~7.6 s 6th-link hold (~9.6 s delivered p50) and `maxResults` 7 to 10 waits out the 22 to 24 s `/goto` hold ceiling (~26 to 27 s delivered p50), so operators can set a ~20 s share for short requests and a ~42 s share for 6-to-10-result requests |
The SERP-specific browser profile, channel, and path override the
corresponding `VITNA_UNBLOCKER_BROWSER_PROFILE`,
`VITNA_UNBLOCKER_BROWSER_CHANNEL`, and `VITNA_UNBLOCKER_BROWSER_PATH`
settings. Omitted SERP-specific settings inherit those operator values.
The stock owned Google route instead requires an explicit absolute
`VITNA_SEARCH_SERP_BROWSER_PATH`; its stock browser does not inherit a
channel or Unblocker executable path. Camoufox requires its remote
installation root and does not require a local browser executable.
An owned route requires the SSH host, egress label, and the selected
browser's configuration. `OWNED_REMOTE_ROOT` is refused with
`OWNED_BROWSER=stock`.
The opt-in browser integration test uses authored local response fixtures and
the installed runtime; it does not query search engines:
```bash
VITNA_SEARCH_SERP_TEST_BROWSER=1 node --test tests/search-serp-browser.test.mjs
```
Set the Unblocker browser path, channel, and profile when using an installed
browser instead of bundled Chromium.
The ordinary local browser transport reuses Vitna Unblocker's browser runtime and requires
its Playwright dependency and Chromium installation. Install those in
`platform/unblocker` as documented in the
[Unblocker overview](../unblocker/overview.md). Proxy URLs contain no path
beyond `/`, query, or fragment. Effective routes must be unique, as described
below. Proxy configuration belongs in the private operator environment;
URLs, credentials, and templates are not exposed through responses,
authenticated status, receipts, or diagnostic reports. Browser retrieval
does not run a CAPTCHA solver or wait for a challenge to be solved.
Browser requests keep the installed runtime's coherent browser headers and
user agent. The service does not rotate random user-agent strings to disguise
one browser as another. Direct retrieval sends a bounded `Accept-Language`
matching the initial engine URL's `hl` or `setlang`, with
`en-US,en;q=0.9` when no hint is supplied. It requests
`Accept-Encoding: identity` so its HTML size bound applies to the received
body. Each browser attempt uses an isolated context and its
configured pinned proxy, when present.
On Linux, the host or container runtime must support Chromium's sandbox and
user namespaces. Run the browser service as an unprivileged user with the
sandbox enabled. A sandbox launch failure requires fixing that runtime's
sandbox support.
## Run on owned infrastructure
Vitna can run the API, browser captures, organic parser, route scheduler, and
CONNECT gateways on operator-controlled machines. This path requires no
managed scraping service or proxy provider API. Run the
[owned egress gateway](../unblocker/overview.md#run-an-owned-egress-gateway)
on each chosen machine, then register its authenticated endpoint in the
search service's private environment:
```dotenv
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=browser
VITNA_SEARCH_SERP_PROXIES=[{"url":"https://serp:[email protected]:8792","egressId":"owned-cloud"}]
```
The endpoint is a placeholder for an operator-owned listener with TLS.
Use a separate gateway port when Search already occupies that machine's
port 8791. Keep the gateway's credential hash on the gateway and its
plaintext connection credential only in the private browser configuration.
The search release includes the gateway modules and the three
`deploy/vitna-egress` templates. Adapt the service template's working
directory, Node executable, and environment path to the host installation.
Packaging these files does not install a service or publish a listener.
Declare the same `egressId` for listeners or machines sharing a physical
exit. Adding containers or gateways on that exit adds capacity without
establishing a new network route. Owning the stack does not establish that
Google accepts its requests; validate each available route with the
diagnostics below and retain its block cooldowns.
### Experimental stock Edge over owned SSH
An opt-in Google adapter runs the installed desktop Edge browser through
an operator-owned SSH exit with public IPv6 connectivity. Configure a
noninteractive SSH alias privately, with host verification and TCP forwarding
available. Browser-to-Google TLS retains target certificate verification.
This SSH transport selects checked public IPv6 targets, with no IPv4 fallback.
The desktop must remain online. A cloud API process cannot use this Windows
browser without a connected worker on that desktop.
```dotenv
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=auto
VITNA_SEARCH_TIMEOUT_MS=30000
VITNA_SEARCH_SERP_TIMEOUT_MS=30000
VITNA_SEARCH_SERP_OWNED_BROWSER=stock
VITNA_SEARCH_SERP_BROWSER_PATH=C:/Program Files (x86)/Microsoft/Edge/Application/msedge.exe
VITNA_SEARCH_SERP_OWNED_SSH_HOST=owned-host
VITNA_SEARCH_SERP_OWNED_EGRESS_ID=owned-exit
VITNA_SEARCH_SERP_OWNED_STATE_PATH=.search-data/serp-browser-check-state.json
VITNA_SEARCH_SERP_OWNED_PROFILE_ROOT=.search-data/browser-controls
```
These settings select `owned-google` exclusively for Google, with one
attempt and no fallback to ordinary routes. Bing keeps its configured
transport and proxies. Allow twenty to thirty seconds for both the shared
request deadline and SERP deadline; the smaller deadline still bounds the
capture. Existing route defaults remain unchanged until explicitly configured.
Each capture uses a fresh profile and headed stock browser, without headless
or browser automation flags. It submits the actual Google GET search form.
Before accepting a capture, it checks that result rows and the observed
next-page URL agree across two observations 150 milliseconds apart.
Owned-route starts retain a random five-to-fifteen-second interval by
default. Operators can set `VITNA_SEARCH_SERP_OWNED_MIN_INTERVAL_MS` and
`VITNA_SEARCH_SERP_OWNED_MAX_INTERVAL_MS` within the bounds above. These
settings change the same durable admission gate; they neither reset a
cooldown nor change the declared egress identity.
On Windows, the stock launcher assigns the suspended browser to its owned
Job Object before resuming it. Cleanup terminates that job and requires
confirmation that its assigned process tree is empty before releasing the
lease or deleting the profile. This containment applies to the Windows
helper. Transient profile locks permit bounded removal retries, with the
allocated directory, root identity, and real paths rechecked before every
retry. A replaced directory is preserved; missing drain confirmation keeps
the lease held and prevents runtime reuse. The renderer accepts delayed
drain confirmation only within the original request deadline. Exact cleanup
proof releases a temporary launcher fence; unverified physical cleanup
keeps the fence in place.
The renderer resolves recognized opaque Google `/goto` links with bounded
native-browser GETs through fresh subpages in the same context, using
`maxRedirects: 0`. Native interception reads status and `Location` headers,
then aborts every response, including HTTP 200, before following a redirect,
downloading its body, or executing scripts. Validated publisher destinations
are recorded for the parser without fetching them or rewriting the captured
HTML. An unrecognized wrapper response is an explicit failure.
At most two link-resolution workers share this capture's context, request
budget, and deadline. Mappings retain the observed listing order, and failure
or cancellation retains both workers' cleanup before releasing the lease.
SERP pages themselves remain sequential.
Non-navigation POSTs are restricted to the main
frame at exactly `/gen_204`, `/log`, `/httpservice/retry/enablejs`, and
`/httpservice/retry/jserror` on `https://www.google.com`: at most 32 per
capture, with at most 64 KiB of reported body data per request. A response-stage
gate aborts POST redirects before another hop. Public-address, request, byte, and deadline
guards remain in force.
Native proxy authentication supplies the capture's private credential only
for a Basic challenge from its exact loopback gateway with the `Vitna` realm.
Server authentication and mismatched or repeated challenges are refused.
Credentials stay out of general HTTP authentication and target headers.
A gateway 407 is a runtime/setup failure, without a Google block penalty.
The dedicated Google capture admits at most 128 browser requests, while
SSH remains capped at 80 tunnels. HTTP request interception does not prevent
every speculative TLS preconnection. Those connections still pass the
gateway's DNS, public-address, transfer-byte, and tunnel limits.
The shared state file reserves starts five to fifteen seconds apart and
persists block cooldowns from five minutes up to one hour. Matching declared
egress identities and state paths share this gate across API and diagnostic
invocations; the exclusive lease remains held through actual cleanup, even
after caller timeout. Keep the physical exit label stable. It is an operator
declaration, not verification of the exit's public IP.
Ordinary SERP pacing and cooldown environment settings do not reconfigure
this durable gate. The owned route's process-local retrieval still follows
the operator's cooldown and cache settings, as Bing's routes do:
`VITNA_SEARCH_SERP_COOLDOWN_MS`, `VITNA_SEARCH_SERP_BLOCKED_COOLDOWN_MS`,
`VITNA_SEARCH_SERP_MAX_BLOCKED_COOLDOWN_MS`, `VITNA_SEARCH_SERP_CACHE_TTL_MS`,
and `VITNA_SEARCH_SERP_CACHE_ENTRIES`. A capture still needs the gate's
admission.
Authenticated status joins the gate deadlines this process last observed to
`owned-google`'s process-local state. A reserved start reads `throttled`
until its `nextRequestAt`, and a durable block cooldown reads `cooldown`
until its `cooldownUntil`. Each capture opens the gate afresh, so a start
reserved by another process sharing the state file appears only after this
process observes it, usually through a refusal.
The implementation and live retrieval evidence remain distinct:
| Check | Verified evidence and limit |
|---|---|
| Windows process lifecycle | Native Edge authored capture and early-abort fixtures passed 2/2, confirming process drain and profile cleanup |
| Native response interception | Authored POST redirect and `/goto` HTTP 307, 308, and 200 fixtures passed; this verifies guards, not live Google access |
| Gateway authentication | Authored proxy-auth fixture suite passed 5/5, including native Edge CONNECT and server-challenge checks |
| Earlier Google prototype | One capture produced nine listings, eight with opaque `/goto` links; one separate GET returned a publisher destination with HTTP 302 |
| Full authenticated owned API | Three distinct fresh Google requests returned HTTP 200 through `owned-google`/`browser` on 2026-10-05: “example domain” returned nine publisher URLs in 5,477 ms, “NASA solar system” five in 4,989 ms, and “Python documentation” seven in 4,588 ms; all 21 URLs resolved without a challenge |
| Multi-page owned API | One authenticated job captured ten observed Google pages on 2026-10-05, returning 94 unique publisher URLs from 96 listings in 107,047 ms; source references matched, with no error or challenge; the 100-link target ended partial at the page limit |
| Local console browser flow | An isolated browser connected with the existing local key, selected Google SERP, submitted “example domain,” and rendered five publisher links after HTTP 200 in 3,959 ms |
These samples used the online PC's stock Edge browser and an owned cloud
exit. The Python sample also verified full service startup at the local
Search console. Health, readiness, and authenticated status returned HTTP 200, with the existing
65-page pilot index loaded. Browser and SSH cleanup completed. This small
sample does not establish production reliability, geographic targeting,
broad Google coverage, or a full replacement for Bright Data's retrieval
infrastructure. The route remains experimental and can still face challenges.
### Experimental Camoufox on an owned Linux host
Set `VITNA_SEARCH_SERP_OWNED_BROWSER=camoufox` to use an operator-provisioned
remote Firefox worker through the same private SSH alias, declared exit,
state file, and owned pacing. This mode needs no local stock browser or
desktop worker. Provision `platform/search/serp-camoufox-worker.py` in the
private remote installation alongside Camoufox 0.5.7, Playwright 1.62.0,
the pinned Firefox 156 beta 34 build, private setup metadata, and permissions
for its authenticated cgroup v2 parent. The repository has no supported public
provisioner. The worker validates private paths and declared pin metadata;
operators must verify installed binaries and packages during provisioning.
```dotenv
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=auto
VITNA_SEARCH_TIMEOUT_MS=60000
VITNA_SEARCH_SERP_TIMEOUT_MS=60000
VITNA_SEARCH_SERP_OWNED_BROWSER=camoufox
VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT=/tmp/vitna-firefox.PRIVATE_INSTALLATION
VITNA_SEARCH_SERP_OWNED_SSH_HOST=owned-host
VITNA_SEARCH_SERP_OWNED_EGRESS_ID=owned-exit
VITNA_SEARCH_SERP_OWNED_STATE_PATH=.search-data/serp-browser-check-state.json
```
Replace the installation placeholder privately. Stock remains the default;
the remote option changes no ordinary routes or deadlines unless configured.
Each search on such a route runs the private worker over SSH, and the worker
starts a second Python process for the capture. On the box on 2026-10-17 each
start cost about 85 ms (17 ms for the interpreter, 19 ms to compile the
worker, 39 ms for its imports), so about 170 ms of every search went to
starting processes before the daemon was asked. With
`VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON=true` the service instead writes the
search to the daemon's socket, `<root>/daemon.sock`, with `nc -U` over the same
SSH host alias, and reads the daemon's one-line answer: about 20 ms outside the
daemon, measured the same night. When Search and the daemon run on the same
Linux host, `--socket-path` and `--socket-group` on `serp-camoufox-daemon.py`
expose an additional Unix socket in a non-world-writable runtime directory
(such as `/run/vitna-camoufox/daemon.sock`, mode `0660` when `--socket-group`
is set and `0600` otherwise), and `VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET`
connects to it directly from Node with no `ssh` or `nc` process at all (under
1 ms per capture vs ~150 to 300 ms to spawn `ssh ... nc -U`). The answer is checked exactly as the worker's
is (the page's identity, its redirect mappings and a parse that is neither a
block nor a failure), and the daemon's budget keeps the worker's margin of 2 s
or a tenth of the request. The daemon owns its browsers and their cleanup, so
no receipt is awaited: a cancelled search closes its SSH client, and the
daemon finishes or abandons that search on its own, as it does when a worker
is stopped. Keep the SSH alias's `ControlMaster` connection open
(`ControlPersist`): a search that has to open a new one pays its handshake,
about 240 ms on the box, on either path.
A daemon with a residential pool tries up to 20 exits per search and spaces
and cools each exit itself. The route's own rests were built for one IP and
work against that pool: a search the daemon could not land locked the whole
route for minutes, and one search at a time turned a second caller away. Set
`VITNA_SEARCH_SERP_OWNED_ROTATING_EXITS=true` on such a route and it neither
paces nor rests: failures count strikes only, and its interval settings are
unused. `VITNA_SEARCH_SERP_OWNED_CONCURRENCY` (up to 32) lets that many searches
run at once, each holding its own durable lease slot. When every slot is
taken, a new search waits for one to free instead of being refused, as long as
its deadline would still leave 8 s to capture (for a shared capture, the
deadline of the latest caller still waiting on it). A search with less time
left, or one whose try already failed, is still answered `serp_routes_busy` at
once. A waiting search holds one of the instance's `VITNA_SEARCH_CONCURRENCY`
request slots, so set that well above the route's concurrency: at the default
of 4, a route serving 3 leaves room for one waiter and the next request is
refused as 429 `service_busy` instead. Give the daemon more warm
browsers than that (`--max-slots`, one upstream session line each): a search
fails over browser by browser and a blocked one relaunches, so three searches
on three browsers kept waiting on each other. Such a route also keeps all but
15 s of the request deadline per fallback route, rather than an even share. A
route searching from a server's own address must keep both unset. Numbered
routes never inherit them from route 1.
The worker creates a fresh headless context and checks stable result and
continuation evidence while preserving the canonical search identity.
Its authenticated CONNECT guard validates every DNS answer before pinning
a public target address. Browser traffic is GET-only, bounded to 128 routed
requests and 128 CONNECTs, 20 MiB of aggregate bidirectional proxy TCP payload,
and 2 MiB of captured HTML. Unknown background destinations remain denied
without invalidating an otherwise valid capture; forbidden main navigation
still fails. A reCAPTCHA frame or widget marks a challenge on any page.
Challenge wording counts only on a page without recognized organic results,
and never in the title, search form, result cards or scripts, which echo
the query.
Up to ten observed opaque `/goto` links use Python TLS header-only GETs
through that same guard, with at most two workers. This is a hybrid
browser/Python retrieval path, not browser-native redirect mapping. It does
not share Firefox cookies, follow publisher destinations, or rewrite
captured HTML. Each attempt has isolated private request files.
Stdin EOF, exact-attempt cancellation,
and deadlines stop its owned cgroup; a matching durable receipt must prove
child exit and an empty group before the API releases its gate, within the
bound below. Losing SSH does not itself prove browser cleanup. A capture
whose SSH run ends without that receipt fails as `serp_unreachable`, never
as the caller's cancellation, unless a receipt recovered afterwards proves
its page or a different failure. Recovery resends exact-attempt
cancellation until the worker answers it, then polls for the receipt,
backing off from 5 to 60 seconds. A cancellation that finds no directory
for its attempt, because SSH failed before the run created it or the host
was unreachable, records a terminal `request_cancelled` receipt in that
directory's place, so recovery ends with proof. A run that arrives later
for the same attempt then refuses before starting a browser. If the
installation itself is gone, for example after a reboot cleared `/tmp`,
nothing can answer and the bound applies: recovery holds the gate for at
most 180 seconds, longer than any worker supervisor for the attempt can
run, then releases it without proof. The capture's private drain then
reports `serp_camoufox_cleanup_unconfirmed`, and service shutdown, which
waits no longer than that bound, logs that a resource could not finish
closing. Neither names the host or the query.
Once its receipt proves cleanup, an attempt keeps only that receipt: the
worker deletes its request, progress, capture and DOM diagnostic files and
its private home and temporary directories, so captured pages are not kept
for later inspection. Each new run first removes at most eight attempt
directories: any whose cleanup receipt is more than 15 minutes old, and any
other attempt more than two hours old for which no cgroup exists. Removal
happens as runs start, so an idle host keeps its latest receipts until its
next run.
On 2026-10-05, this optional route was verified through authenticated local
API requests with `fresh: true` and the actual Search console, using the owned OVH Camoufox
worker without a paid retrieval service, proxy provider, or solver:
| Integrated live check | Observed result |
|---|---|
| Fresh API, “Python asyncio TaskGroup documentation,” page 1 | HTTP 200; nine verified organic results in 36,202 ms |
| Fresh API, “What are the latest US AI chip export restrictions?” | HTTP 200; nine organic results in 23,252 ms, including Congress and Reuters listings |
| Fresh API, the same Python query at page 2 / offset 10 | HTTP 200; ten organic results in 7,843 ms; the two Python pages contained 19 distinct publisher URLs |
| Local console, “site:bis.gov H200 January 2026 license review policy” | HTTP 200; four BIS links rendered in 6,875 ms |
Further controlled fresh samples returned nine Rust results in 13,935 ms,
nine headphone results in 11,136 ms, nine for an accented French query in
9,506 ms, and three results on matching hosts for a quoted MDN query with
`maxResults: 3` in 9,697 ms. Python page 3 at offset 20 returned ten results
in 9,398 ms, bringing the three pages to 29 distinct publisher URLs. A fresh
Rust collection targeting 18 links across two pages finished `completed`
with 18 distinct links: 26,069 ms wall time and 23,162 ms reported job time.
The fresh PostgreSQL 18 MERGE baseline returned nine results with nine
snippets in 8,463 ms, including 8,354 ms of owned capture, without cache
reuse or coalescing. These additional checks included six successful
single-page captures and two Rust collection pages before the empty-page
change. After that change, a verified empty response brought the accepted
checks to nine captures: seven standalone pages, including the empty result,
and two Rust collection pages.
No CAPTCHA, HTTP 403, or HTTP 429 was recorded in this additional sample.
Limits appeared alongside those successes. A NASA `site:` query retained
its address-bar identity but yielded mostly unrelated or off-site listings;
the scope guard refused it with `serp_query_mismatch`. The first
French call returned HTTP 503 with `Retry-After: 2` before capture. An
improbable quoted query intended to test empty results twice reached HTTP 504
near sixty seconds;
the stored DOM later showed the genuine Google title, query and canonical
URL, an explicit “About 0 results” count, an AI-only overview, and no organic
listings. The initial shell completed with three main-document HTTP 200
responses. Narrow parser and worker support passed an offline regression
with that stored AI-only DOM. A later fresh retry of the original quoted
query returned HTTP 200 with nine organic listings in 8,122 ms, without
cache reuse or coalescing. Google had changed its response; this failed the
expected empty-result check and is excluded from the accepted captures.
A separate nonexistent scoped query returned HTTP 200 with `organic: []`
in 6,884 ms through one fresh owned attempt, followed by clean idle state.
This verifies genuine empty API output, but does not establish that the
original AI-only layout was reproduced live. Forced-cancellation receipts
can retain
stale initial counters, so zero counters alone do not prove zero traffic.
A separate no-Google cgroup self-test confirmed child and grandchild
cleanup. The earlier four-result diagnostic below retains its separate
browser and header-mapping timings; those timings are not API latency.
The integrated route remains experimental. This small controlled sample
does not establish sustained reliability, soak or load performance,
unchallenged future access, or public production deployment.
#### Warm standby browser daemon and latency optimizations
On 2026-10-07, the Camoufox worker was augmented with an optional standby
daemon (`serp-camoufox-daemon.py`) to reduce single-query latency from the
3.6s to 4.7s cold baseline down to sub-second and low-second times.
Cold browser process startup accounts for approximately 1.1s to 1.3s of each
independent search attempt. When active, `vitna-camoufox-daemon.service`
maintains a pre-warmed headless Camoufox browser instance listening on a
private Unix domain socket at `/tmp/vitna-firefox.PRIVATE_ROOT/daemon.sock`
(permissions `0600`, owned by the unprivileged worker user).
Key architectural properties of the standby daemon include:
- **Isolated ephemeral contexts**: Each incoming search query creates a fresh
`BrowserContext` via `browser.new_context()`. The context navigates the query,
settles the DOM, unwraps redirects, and is closed immediately upon completion.
No cookies, cache, storage, or history leak across queries.
- **Parallel redirect unwrapping**: Google `/goto` opaque redirect links are
unwrapped concurrently through the local guarded CONNECT proxy with
`GOTO_WORKERS = 4`, reducing redirect resolution overhead from ~800 ms to
~250 ms.
- **Resource filtering**: Image requests (`resource_type == "image"`) are
aborted at the route level, eliminating unnecessary asset downloads (~170 KB
payload reduction per query) without affecting search result layout or text.
- **Gecko subsystem tuning**: Firefox preferences disable hardware video
decoding, media autoplay, background animations, and session history to
minimize memory consumption and thread churn.
- **Resource bounds and idle auto-shutdown**: The daemon automatically closes
the browser process after 10 minutes of inactivity (`IDLE_TIMEOUT = 600`),
guaranteeing zero idle browser RAM footprint. It recycles the browser process
after every 50 requests (`MAX_REQUESTS = 50`) to prevent memory accumulation.
The production unit's cgroup allowed `MemoryMax=6G` and `TasksMax=2048` on
2026-10-13, when its ten browsers used about 2.6 GB and 982 tasks.
- **Profiles a stop leaves**: Playwright removes each browser's temporary
profile when the browser closes, but stopping the service kills its driver
first, so every restart left each browser's profile in the unit's `TMPDIR`
(`ROOT/work`): 1,185 of about 29 MB, 34 GB, by 2026-10-17. At start, before
it launches a browser, the daemon lists the Playwright and stock Chrome
directories there (`playwright_firefoxdev_profile-*`,
`playwright-artifacts-*`, `vitna_chrome_slot-*`), and once it
serves it removes them, sparing any a running process names on its command
line. It does this only when `TMPDIR` is `ROOT/work`, never in a shared one.
- **Seamless cold fallback**: The supervisor worker checks for the daemon socket
first. If the daemon is stopped, restarting, or unresponsive, the worker
falls back immediately to cold in-process browser execution.
Live benchmarks on the primary `vitna-api` host on 2026-10-07 confirmed:
| Benchmark Query | Cold Execution Latency | Warm Daemon Server Latency | Total Round-Trip (Remote Client) |
|---|---|---|---|
| Single-result query (“apple investor relations”) | 3,658 ms | 745 ms | 1.57 s |
| 9-result query with 9 unwrapped redirects (“nvidia quarterly earnings report”) | 3,820 ms | 1,512 ms | 2.17 s |
All supervisor receipt checks, attempt isolation, and cgroup cleanup guarantees
remain fully enforced.
`--fast-relaunch 1` (off by default) keeps each browser slot's Playwright
driver running across relaunches, replacing it only after a launch on it fails,
and lets an old browser close while its replacement starts. A relaunch after a
refusal took about 2.5 s on the API box on 2026-10-13 (medians of 5): closing
the old browser 306 ms, the exit check 721 ms, starting the new browser 828 ms
and its first context and page 619 ms. A slot launched, relaunched five times
and closed there with the real browser (no exit check, no search traffic) took
1,751 ms a relaunch without the setting and 1,145 ms with it: the close it
waited for went from 336 ms to none, and the browser start from 835 to 567 ms.
A slot's driver stops when the slot closes, after the old browsers on it. The
first try on each browser records `browserStartMs` and `browserCloseMs`, the
close its relaunch waited for (null when no relaunch came before it).
#### Failover across verified exits
With upstream residential proxies, each search fails over across up to 20
distinct exits (`DEFAULT_MAX_ATTEMPTS`) within the request's SERP budget. Every
attempt has 8 s and runs on one browser slot whose exit was checked as a US
address that is not cooling; a challenged exit cools for ten minutes and its
slot rotates to a new session.
Failed tries got faster (1.7 to 1.9 s each by 2026-10-12), so the cap, not the
deadline, became what ends Google: 384 of 427 searches Google missed stopped
at exactly 20 tries, with 38.9 s used at the median of a budget near 48 s,
while tries after the 13th still delivered 6.4%. `--max-attempts` (1 to 40,
default 20 in `legacy` profile, 2 in `--profile standard`) sets the total try
cap, and `--max-block-attempts` (0 to 40, default 0 off in `legacy` profile, 2
in `--profile standard`) stops a search immediately once Google returns that
many `serp_blocked` responses so a 120-second query hold never cascades across
additional warm residential exits; a 40-try history is about 100 KB. The worker reads the history back through a progress
file capped at 1 MiB, which holds the longest one a search can make: 42 tries
(the cap plus a race's overrun, below) with every field at its bound, about
750 KB, built by the daemon tests. Every answer says
what ended it in `endedBy`: `delivered`, `attempts` (the cap), `exits` (no
browser could give it an exit, below), `deadline`, or `stopped` (a race
another chain won).
A search the deadline ends has to report before the launcher stops waiting,
and the launcher counts its budget from before the worker started. So the
worker hands the daemon its own remaining time less 2 s, or a tenth of it on
short budgets, as room to write its files and exit. Until 2026-10-12 it handed
over all of it, and such searches were cut off with their answer unread: 48 of
the 49 that Google missed at `--max-attempts 32`, and 3 of 3 at 40, left
receipts without the daemon's tries.
A search with no exit to be had ends at once as `serp_unreachable`, which
Search hands to Bing, instead of waiting out its deadline. That is so in two
cases. Every browser has already failed to find this search an exit, or the
provider has stopped giving exits: 20 exit probes in a row have failed across
the pool, the last within 30 s, and no browser is still live. On 2026-10-12,
from 16:00:38 to about 16:13, every DataImpulse line answered
`407 TRAFFIC_EXHAUSTED`, no browser could launch, and each search waited its
whole 49 s before Bing answered it. A probe that answers ends the run of
failures, and after 30 s without a failed probe the next search probes again,
so the pool resumes by itself once the provider is back.
An exit that just delivered (proven) delivered 79.8% of the first tries it
served and 25.9% of the later ones, against 29.5% and 7.3% for other exits
(22,708 tries, 2026-10-10 to 12), yet 72% of its tries went to searches that
had already missed. `--proven-first-wait-ms` (0 to 5000, unset by default)
keeps proven exits for first tries: a later try takes one only when no other
warm browser is free, and a first try waits up to that long for one that is
busy or coming off its spacing. Each try records `firstTry`, `provenExit` and,
on a first try under the setting, `provenWaitMs`.
A 100-search run on 2026-10-08 delivered 56. In its last 43 searches, 246 exit
attempts went as follows: 177 met Google's reCAPTCHA, 31 stalled, 21 delivered.
A stall is a results page whose header Google flushes (title and search box,
about 84 KB of DOM) and whose body never arrives; each was waited out to the
attempt deadline. Two changes spend the budget on more exits instead:
- **Stall cut-off**: an unrecognized page that stops changing for 2.5 s
(`STALLED_PAGE_GRACE`) ends the attempt as `serp_timeout` with
`unrecognizedReason: "google_page_stalled"`. Healthy captures settle within
about a second; no stall recovered in production.
- **Exit freshness**: an exit proved by a launch probe or a post-capture check
within 15 s (`EGRESS_FRESH_SECONDS`) skips the before probe, about a second
per attempt. A cooling exit is still refused, the post-capture check still
runs on every delivered page, and the attempt history marks the skip with
`egressCheckSkipped` and `egressVerifiedAgeMs`.
- **A page that changes while its links resolve**: Google can keep rendering
after a results page has held still. When the final check finds the page
changed but still a results page for the same search, the daemon resolves
only the links that are new and checks once more, instead of discarding a
delivered page (about 280 KB through the exit) and starting a fresh try. It
fails as before if the page changes again. The attempt history marks this
with `pageChangedDuringRedirects`. About 2% of searches hit it on 2026-10-09.
`--fingerprint-os` (`linux`, `windows` or `macos`, default `linux`) sets the
operating system the browser fingerprint presents.
Each line of the upstream proxy file (`--upstream-proxies-file`) is one warm
browser's session: `host:port:user:pass`, `user:pass@host:port` (DataImpulse's
dashboard export) or a full `http://user:pass@host:port` URL. After a block the
browser rotates to a new exit before relaunching, in the provider's own way.
Decodo keeps the session in the username (`...-session-<id>-...`), so the
daemon swaps the id. DataImpulse keeps it in the port: each port from 10000 to
20000 holds one exit for the dashboard's rotation interval (measured
2026-10-10), so the daemon moves the browser to a random new port on the same
login. Set DataImpulse to Sticky; with Rotating, the check page, the results
and the exit checks would each leave from a different IP. Either way the exit
is verified through `ip.decodo.com/json`, which answers through both.
`--direct-exits` (`4`, `6` or `4,6`, unset by default) adds one browser per
family that leaves from the host's own IPv4 or IPv6 address, through the same
guard with no upstream line. Direct browsers come on top of `--max-slots` and
first: a direct address is tried whenever its spacing allows, and the
residential browsers take the rest. `--direct-interval-ms` (4000 to 60000,
default 6000) spaces searches on each direct address, proven or not. On the API
box on 2026-10-13, one browser a family at 10 searches a minute delivered 593 of
601 first tries over IPv4 and 297 of 301 over IPv6 (98.7% each, median 0.84 and
0.80 s), against 61 to 75% through the residential lines beside them. An
address's standing moves with use, though: by 2026-10-14 the IPv6 delivered
78%, steadily, with `udm=14` or the full page alike, and the IPv4 was refused
4 times in 5. Google refused single E2B addresses within 8 to 10 minutes at 27
or more searches a minute, and within about 40 at 15.
Nothing can rotate a direct address, so a refusal relaunches its browser on it,
and the address cools only when 7 of Google's last 10 answers to it were
refusals, for 10 minutes; one more refusal after that cools it again until
deliveries push the refusals out. With `--keep-proven-exits 1`, a refusal
after a delivery keeps the browser instead, unless it is the one that cools
the address. Simulated at 10 searches a minute, a steady
22% refusal rate keeps an address in use 91% of the time and 80% keeps it in
use 1%. A residential exit's cooldown after every refusal idled both addresses
for most of a 30-minute run beside production, and since a first version
counted direct browsers within `--max-slots`, their cooling left that arm two
browsers. A browser closed while its address cools is relaunched in the
background once it has cooled, at most every 30 s, so a family whose route is
down costs one probe a half-minute. A direct browser stays warm whatever
`--idle-timeout` says, is launched without geoip as it was measured, and has
its address probed at launch only: nothing can move it under a page, and
probing each search would load the echo from one address. Its probes leave the
provider-outage watch alone, and its record stays out of the network
judgments, which its rate would skew. Each try records `directFamily` (4, 6 or
null).
`--direct-sources` (public addresses of this host, comma-separated, unset by
default) adds one direct browser per address, bound to it: its connections and
its launch probe leave from that address, and the address echo must report that
same address back or the browser is not launched. It is for addresses added to
the host for searching, such as OVH Additional IPs on `ens3`, each spaced
`--direct-interval-ms` apart (60000 for one search a minute) and cooled by the
same rule. By 2026-10-14 the box's own IPv6, after a day of tests at ten a
minute, was refused on all 30 searches at one a minute; whether a fresh address
used gently stays healthy is what a day at one a minute on new addresses is to
show.
`--static-proxies-file` (a file, unset by default) adds one browser per line
whose exit never changes, such as a dedicated ISP proxy's own port, written as
the upstream lines are; blank lines and `#` comments are skipped. Static
browsers come after any direct ones and ahead of the residential lines, on top
of `--max-slots`, and are kept as a direct address is: never rotated, spaced
`--direct-interval-ms` apart, cooled only when 7 of Google's last 10 answers to
the address were refusals, never closed for idleness, and relaunched in the
background once cooled. Unlike a direct browser, a static one launches with
geoip from its exit, as a residential one does, and its pages are checked after
they deliver, since its provider could move it. Its launch probe, through its
own line, proves the exit until such a check fails, so its tries skip the
probe before Google. Its probes leave the provider-outage watch, which is for
the rotating lines, alone. With `--keep-proven-exits 1`, a refusal after a
delivery keeps its browser, but never one that cools the address. A fixed
exit, static or direct, takes a search's first try only: a later try runs
while Google holds the search, when any exit is refused, and each refusal
costs an address that cannot be swapped (after one, a proven exit's next
first try failed 24.6% of 658 times against 15.3% of 2,850 after a delivery,
2026-10-14 and 15), so later tries go to the residential browsers, and a
daemon with none ends a refused search at once. `--fixed-retry-exits 1` (off
by default) relaxes that exclusion for at most two failed static proxy tries per
search (`static_failures < 2`) while still keeping direct host addresses
(`--direct-exits`, `--direct-sources`) first-try-only and preferring free proven
exits ahead of unproven static lines in retry windows, so a first try that lands
on a flagged static ISP port can fail over onto one more healthy static ISP port
before falling through to residential exits. Once a warm fixed exit has
delivered at least once, or during its first two cold tries (`consecutiveBlocks
< 2`), it takes every first try it can, whether or not its last try delivered:
in the ISP trial of 2026-10-15 a line whose last try was a refusal it had not
kept delivered 32 of its next 36 first tries, against 247 of 327 for the
residential exits that took first tries while it waited; a never-proven fixed
exit (`successes == 0`) with 2 or more consecutive refusals yields priority to
warm rotating and proven exits so externally hot static lines on a fresh health
ledger cannot starve rotating exits across 70 cold refusals before the 7-of-10
window cools them. A fixed exit also rests 3
hours when its first tries over the last hour delivered under
`--fixed-decay-floor-pct` (0 to 99, default 85, 0 never), once it has 20 and
at least 8 of them were refused: the box's own IPv4 delivered 98.7% of first
tries on 2026-10-13 and was refused 4 times in 5 a day later, a slide the
7-of-10 window never sees. The 8 refusals keep a short streak from resting a
good line, since 20 first tries is two minutes at 10 searches a minute: on
2026-10-16 a line rested for 4 refusals in its first 20 and delivered 168 of
its next 172. Replayed over every ISP line's first tries from 2026-10-15 to 17,
the 8 turn 10 rests into 3, and a line refused 1 time in 5 still rests, by
about its 40th first try. A line with peers, the other fixed exits with 20
first tries in the last hour, rests only when it delivers 8 points or more
under their median, or under 70% whatever they do. When every line dips at
once, as all five did from 05:30 to 07:40 UTC on 2026-10-16 (84% to 91%), a
rested line's first tries go to the residential exits, which delivered 73% of
theirs that night, while the two left running delivered 92% and 94% of their
first tries over the next two hours. Over the same replay this keeps the one rest that was a line's
own (84% beside peers at 98%) and drops the two in that dip (84% and 85%
beside 91% and 88%). A fixed exit refused on 8 first tries in a row, with no
delivery between them, rests 1 hour, however long they took and whatever its
peers do, and back from that rest 2 more refusals in a row rest it again: the
7-of-10 window cools a refusing line 10 minutes at a time, one first try a
cooldown, so a line Google has flagged never reaches 20 first tries in an
hour. On 2026-10-17 four ISP lines refused every first try (0 of 9 each) for an
hour after the AA benchmark's agent queries and were never rested, each try
holding its search until Google's window. A line delivering 85% refuses any 8
first tries in a row about once in 4 million, and 2 in a row 2.25% of the time.
The rest is an hour, not three, because a burst of re-sent queries, which
Google refuses from any exit, gave every working line its 8 within minutes the
same night, and 3-hour rests left no line at all while plain queries delivered
20 of 20. A flagged line costs 2 refused first tries an hour.
`--fixed-long-words` (3 to 32, default 0, off) makes a search of that many words
or more a long one, as an agent's are, and a fixed exit takes a long search's
first try at most once every `--fixed-long-interval-ms` (1000 to 3600000,
default 60000). A long first try that finds every fixed exit paced goes to the
residential browsers, and never waits for a proven exit it may not take; short
searches keep the lines' own spacing. Ten ISP lines took short queries at 8 to
10 a minute each for 6 to 10 hours at 97% delivered (2026-10-16), while the AA
benchmark's agent queries, long and often quoted, at about 3 to 5 a minute an
address flagged five of them within minutes (2026-10-17). Each try records
`longQuery`. Each try
records `staticLine`, the line's number in the file from 1. The daemon refuses
a file with a line it cannot read, a repeated line, or a line on a port that
rotates its exit: a DataImpulse port, which is a session, or port 8000 of
Oxylabs's self-service dedicated ISP gateway, `disp.oxylabs.io`, which gives a
random address of the plan on each request where each port from 8001 holds
one (Oxylabs's documentation, retrieved 2026-10-14), or the same port of
`isp.oxylabs.io`, the gateway an account's dashboard gives for the plan. Its errors name the line's
number, never its text, which carries a password.
`--proven-interval-ms` (1000 to 10000, default 5000) spaces a proven exit, one
whose last outcome was a delivery within the last five minutes, this far apart;
every other exit keeps 10 s. At 10 s for all, a browser on a good exit sat idle
while searches queued for one. An A/B on 2026-10-10 (four alternating runs of
100 searches): proven exits reused under 7 s apart delivered 86% of tries (84%
at 10 s), the median wait for a browser before a try fell from 513 ms to 0 (p90
3.0 s to 1.1 s) and the median search from 6.3 s to 5.4 s, with blocked tries
and bandwidth unchanged. Each attempt records the spacing its admission set as
`egressIntervalMs`.
`--proven-seconds` (60 to 3600, unset by default, which keeps 300) sets how
long an exit stays proven after it delivers, for the slot preference, the
spacing above and each try's `provenExit` alike. Replayed over the load tests'
31,636 tries to 2026-10-12, a first try delivered 76.6% of the time when its
exit had delivered under a minute before, 74.2% at 5 to 15 minutes (31 tries),
43.8% at 15 to 60 (16) and 33.3% past an hour (33), against 29.4% for an exit
never seen. Under load no first try met a gap of 1 to 5 minutes, so five
minutes ends the preference just when sparse traffic could use it. The
samples past five minutes are small: the setting is for a trickle test of 900
to 1800 s, not yet a default.
`--race-width` (1 to 3, default 1) runs up to that many chains of tries for
one search at once. A search fails only when every try that fits before its
deadline is refused: a failed try costs about 2.3 s, so about 22 fit, and with
7.6% of long-tail tries delivering that predicted 17.6% failures against 16%
measured (2026-10-11); across eight runs of the same 100 queries, misses per
query followed chance, not the query. The extra chains take only a warm
browser while no other search waits for one, so a busy daemon serves exactly
as before. They share the exits already tried, the first delivery answers,
and the others finish the try they are in and stop. They share the cap too:
no chain starts a try once the chains have made `--max-attempts` between them,
so a raced search ends at most two tries over it, the ones still running.
Capped chain by chain, three chains at 40 could make 120 tries, more than the
progress limit holds, and a history past that limit fails the search with its
delivered page. Every chain's tries are in the answer's `attemptHistory`,
tagged `raceChain`, with `raceWidth` and `raceChainsTried` beside them.
`--retry-window-ms` (1000 to 30000, unset by default) changes what a search
does once Google has refused it. Google lets a refused search through again
about every 10 s: over 5,710 tries at production's setting on 2026-10-11 and
12, a later try admitted 10 to 11.5 s, 20 to 21.5 s or 30 to 31.5 s after the
search's first delivered 21.9% of the time, and one admitted between those
windows 4.6%. The gaps held 68% of tries and 64% of bytes but 28% of Google's
answers. Live receipts the same night agreed, 9.2% against 2.8%, on browsers
of the same age inside and outside the windows. With the flag, a refused
search's later tries wait, holding no browser, for windows at its first
admission plus the period, twice the period and so on, each 1.5 s long, and
are admitted on the mark. `--retry-window-tries` (1 or 2, default 2) lets a
window take a second try, no earlier than 0.8 s past the mark, when the first
ends inside it; a window's tries never run together, which is how racing
lost. `--retry-window-after-refusals` (1 or 2, default 1) lets a refusal on the
first try or an unproven exit try one more exit immediately before windows
begin, while a later refusal on a proven exit still starts windows at once.
`--retry-window-from-refusal` (0 or 1, default 0) anchors each retry window a
full `--retry-window-ms` period after the latest Google refusal finishes rather
than from the first try's pre-navigation admission, so `probeBefore`, proxy TLS
and immediate strikes do not shorten the 10 s quiet gap before `Window 1` or
fire a second turn 0.8 s after a refusal inside a window; window turns are
counted only after `probeBefore` succeeds, so an idle sticky session whose exit
rotated (`ip_changed`) before Google is contacted yields immediately to the next
warm slot without consuming `Window 1`. A failure that is not
a refusal does not push the search to the next window, and a window that cannot
fit before the deadline ends the search, leaving Bing the time. Each try records
`sinceFirstAdmissionMs` and, when it aimed at a window, `windowTargetMs`. The
flag needs `--race-width 1`. What an A/B of the two has to show is whether the
answers found between windows today come back in the next window.
`--window-proven-exits 1` (off by default; needs `--proven-first-wait-ms`)
lets a later try that starts in one of those windows, every 10 s after the
search's first admission and 1.5 s long, take a proven exit that is free, as a
first try would, without waiting for one. In the windows a proven exit doubles
a later try's chance: over the production-setting runs of 2026-10-12/13, later
tries admitted in a window delivered 36.5% on a proven exit (52 tries) against
17.0% on another (802). Only 6% of window tries had one, since the proven
first-try wait keeps proven exits for first tries, where they are worth more
still (75.0% against 40.7%), so a window try never waits for one. The window is
judged when the pool hands the try its browser, not when the try starts
waiting for one, so a try that waits into a window takes a proven exit there
and one whose wait outlasts a window does not. A try aimed at a window by
`--retry-window-ms` takes its browser 0.5 s before the mark, before Google's
window opens, and counts as that window's try; without that, the two flags
together never gave a window try a proven exit. Each try records
`windowProven`.
`--keep-proven-exits 1` (off by default) keeps an exit that has delivered
through its first failure in a row. After a refusal the exit and its browser
stay, and the refusal's cooldown is cleared; after a timeout or an unreachable
check the exit stays and the browser is relaunched on it. A second failure in
a row moves the browser to a new exit, as without the flag. Without it, a
refusal sends an exit that has been delivering to the back of the queue for at
least 10 minutes and its browser hunts through fresh exits, each one a browser
relaunch. In A/Bs beside production on 2026-10-13 (one browser per arm, 10
searches a minute, one try a search), a proven exit kept through one refusal
delivered the next search 16 times in 25, where a fresh exit delivered 8 to 13%,
and keeping proven exits took first tries from 60.7% to 82.6% (about 448
searches an arm). Each try records `keptExit`: `browser`, `exit`, or null when
the slot moved on.
`--spare-gap-refusals 1` (off by default) treats a later try's refusal between
Google's windows (above), on an exit that has delivered, as the query's and not
the exit's: it neither cools the exit nor counts toward `--keep-proven-exits`'
second failure, the browser stays, and the exit's record changes only in when
it was seen. Google holds a refused query whatever the exit: with both levers
on beside production on 2026-10-14 (1,200 searches), later tries between
windows delivered 7.6% on exits that had just delivered and 8.4% on the rest,
and an exit kept through such a refusal delivered the next first try on it 31
times in 42. Yet two of them in a row were what swapped out 133 of the 179
exits that run dropped after delivering. An exit that never delivered is
swapped and cooled as before, which keeps fresh exits coming for refused
searches, and a search may take an exit it spared once more, in one of Google's
windows, where later tries on proven exits delivered 17 of 26 in the same run.
A first version spared every exit, and beside production its refused searches
ran out of exits: 18 of 211 ended at their deadline after 5 to 7 tries, against
4 of 214 for the control. Nor does a refusal between windows give any exit its
first verdict, which `--network-yield-floor-pct` judges networks by: in the same
run 915 of the 1,145 fresh exits Google first answered met it between windows,
where 79 delivered (8.6%), against 34 of 76 on a first try, so the floor was
judging how often a network's exits were handed held queries. A refusal on a
first try or in a window, and any failure that is not a refusal, counts as
before. Such a try records `keptExit` as `spared`, and a try on an exit the
search spared records `sparedRetry`.
`--after-check` (`wait` or `background`, default `wait`) decides whether a
delivered page waits for its post-capture exit check, the fresh probe through
the same line that starts when the page settles. On 2026-10-11 that wait was
about 0.5 s at the median, and the check passed 132 times out of 132. With
`background` the validated page goes back at once (`probeAfterWaitMs: 0`,
`exitCheckAfterReply: true`) and the check finishes on its own, keeping its
browser until it settles, so the next try there still starts from a proven
exit. A check that fails withdraws the exit's proof, so the next try probes it
before Google, and the page already returned stands: what `background` gives
up is the guarantee that that page came from its stated exit. Each try reports
the daemon's running `afterReplyChecksPassed` and `afterReplyChecksFailed`.
Measured the same night over 400 long-tail searches in alternating arms,
`background` made the delivering try 0.50 s shorter at the median and 0.61 s
at p90. The wait for a browser, tries, blocks and bytes did not change, and 1
of 162 background checks failed. Production runs `background` from 2026-10-11
23:46 UTC.
Every try that gives up as `google_page_stalled` records `mainBodyFinished`:
false when Google sent a page's header and never finished its document, true
when a complete page simply showed no results. On long-tail queries on
2026-10-10, a stalled try held a page of 74 to 75 KB every time (121 tries) and
cost 3.6 s before the 2.5 s grace ended it. `--stall-samples` (0 to 32, default
0) keeps the last N such pages for an hour in `stall-observations/` under the
daemon root, beside a JSON report of the status, page size, hold time and
`mainBodyFinished`, so the page can be recognized at once. The directory is
private (mode 0700, files 0600, generated names) and is never mixed with
`challenge-observations/`. A page can carry the query, so it stays off unless
an operator is diagnosing stalls; each try that kept one names it as
`stallSampleId`.
Every try also records `requestKinds`: each request the page made and what the
guard did with it (`allowed`, `cached`, `skipped`, or `refused:` for its host,
method or type), as method, request type, a Google host name (any other host
reads `other`) and at most two path segments, never a query, twelve kinds at
most. `failedRequestKinds` counts allowed requests the browser then failed, by
browser error code, and a stalled try records `pendingRequestKinds`, the
allowed requests still unanswered when the grace ran out. Serving a stalled
check page the one Google file it asked for left it stalled (2026-10-11), so
these name what else it waits on.
They showed what most stalls were: Google's quiet refusal of the results
request. The check page passes, and the try's second document is a complete
page titled for the query, with the search form and its empty containers but
none of the results column (`#search`, `#rso`, `#rcnt`, `#center_col`,
`#botstuff`); it asks only for a logging beacon and a font, as every delivered
page does, and never moves on. Once that second document has finished arriving
and the latest response is a success, the daemon ends the try after 1 s
unchanged as `serp_blocked`, with `unrecognizedReason` `google_page_held` and
`heldMs`, instead of waiting out the 2.5 s stall as a timeout. The exit then
cools down like any other refusal (a timeout records nothing against it), and
the operator monitor counts the try as a challenge. A first document, one still
arriving, a page moving on, a page with a results column, or one titled only
"Google Search" still stalls as before.
`--held-grace-ms` (100 to 5000, default 1000) sets how long that page must
stay unchanged. About one long-tail try in four ends this way, so the grace is
paid often. A page that looked held and then turned into results is what a
shorter grace would lose, so a try that recovers records `heldRecoveredMs`, how
long its page looked held, and every try reports the daemon's running
`heldPagesEnded` and `heldPagesRecovered`.
Before a browser launches on a new exit, the daemon may pass over the exit's
network and probe another (at most twice a pick, about 0.8 KB a probe). By
default a network is passed over when, over 30 or more of its tries, it
delivered under half as often as the pool, except for one exit in five.
`--network-yield-floor-pct` (0 to 50, default 0, off) judges a network by
Google's first verdict on each of its fresh exits instead, a delivery or a
block, so one proven exit delivering again and again cannot lift it: a network
whose fresh exits delivered under the floor, over 5 or more of the exits the
daemon remembers, is passed over except for one exit in ten. Replayed over
17,773 fresh long-tail exits from 2026-10-10 to 12, a 5% floor would have
passed over 24% of them, which delivered 4.6% of the time against 12.0% for
the rest.
A try Google redirects away from its own search ends as `serp_redirect_blocked`
(7.6% of long-tail tries in one run on 2026-10-11). It records
`redirectDrift`, which says what moved and never a
value: its `source` (the navigation request, its response, the page URL, or
the proxy's CONNECT guard), the target's host class and first path segment,
whether a fragment appeared, how many `q` parameters there are and whether the
query matches (exactly, ignoring case, or containing ours, with the change in
word count), whether `start` matches, and the names of query parameters added
or removed against ours.
When the redirect leaves `www.google.com` for another host, the slot moves to
a new exit at once and the exit cools for an hour, as after a block, though it
counts as no challenge. Every redirect read on 2026-10-11 (103 tries in two
runs) came from one exit that Google sent to `www.google.com.hk/url`, its
country-domain redirect, which follows the exit rather than the query; the
slot that kept that exit failed every search that reached it. A redirect that
stays on `www.google.com` keeps its exit, since another query or offset may be
the query's own.
`--goto-resolver` (`http2` or `browser`, default `browser`) chooses who asks
Google where each result link points. `http2` is the Python HTTP/2 client
described above, from the same exit, and sends no cookies or Referer. `browser`
uses the page's own `fetch` with a manual redirect, `credentials: "same-origin"`
and no Referer, so the lookups carry Firefox's TLS and reuse the non-anonymous
HTTP/2 connection that loaded `/search` (rather than partitioning into
`LOAD_ANONYMOUS` and opening extra upstream proxy `CONNECT` tunnels: 11 vs 13
`connects` per capture and `326 ms` vs `586 ms` 2-link redirect median in a
16-query Latin-square on one warm residential exit on 2026-11-03, `16/16`
delivered with 0 holds); the daemon reads `Location` from the response the
browser reports, and the publisher is never requested. Each attempt records
`gotoResolver`. An A/B on 2026-10-09 found no difference in how often Google
blocked an exit's next search after either, so TLS on the lookups is not a
block signal; `browser` is the default because it skips the lookup client's own
tunnel and handshake, about 10 KB of residential traffic per delivered page
(296 KB against 306 KB for a delivering attempt). When `--goto-respect-num 1`
targets 1 or 2 initial links under `--goto-refresh-max-ms` or `--goto-pace-ms`,
`render_search` also starts that zero-pace burst once during DOM stability
settling (never cancelling or restarting in-flight tokens, cutting 2-link
`redirects` phase median to `251 ms`).
`--goto-refresh-max-ms` (0 to 60000, default 0, off) lets the `browser` and
`http2` resolvers wait out Google's hold on a result link. On 2026-11-02 the
first two or three links on a page answered a 302 and each further one a 200
"Verifying your request" page whose meta refresh names a new `/goto` link and a
wait that grew link by link: 3, 4, 5, 7, 12, 22 and 24 s. Once its wait was up,
a plain fetch of the new link answered the 302, with no cookies and no script
(34 of 34 across six pages). Off, such a link fails the search as
`serp_invalid_response`, which made every served page fail. On, navigation and
page settling keep their 8 s attempt ceiling, link lookups wait until the page
has settled so streaming cards do not cancel in-flight holds, and a settled
page extends its redirect budget by `--goto-refresh-max-ms` within the search's
deadline. Each lookup reads the 200's body, waits the named seconds plus half a
second, and fetches the link it names, up to two holds per link; a hold whose
waits add up past the cap, or that cannot end before the search's deadline, is
refused at once and fails the search as before. A page's links wait side by
side, so its slowest hold sets the cost: 22 to 24 s on a page of ten links,
longer than the 18 s `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS` production ran on
2026-11-02, so turning this on means setting at least `30000` and raising that
share and the caller's budget past 35 s. Each try records `gotoHolds`,
`gotoHoldsRefused` and `gotoHoldWaitMs`, the longest wait one link was given.
`--goto-max-links` (1 to 10, default 10), `--goto-respect-num` (0 or 1, default
0) and `--goto-pace-ms` (0 to 2000, default 0) shape how many discovered
`/goto` links a capture unwraps and how 3-to-5-link lookups are spaced. On
2026-11-03 a 49-page, 7-arm Latin-square sweep (`n = 7` per arm across 4 warm
residential Camoufox slots, `294` `/goto` lookups, `49/49` delivered) mapped
Google's per-impression `/goto` unlock curve (shared across exits, browser
contexts and document navigations for a single SERP impression; `94 to 120`-byte
AEAD protobuf tokens, zero publisher URLs in HTML, `<script>` blocks or XHRs):
links 1 and 2 answer instant 302s (`~0.64 to 0.80 s` total SERP latency on warm
slots), links 3, 4 and 5 unlock non-uniformly at `~0.50 s`, `~1.05 s` and
`~2.30 s` after page settle (`14/14` zero-hold 302s on links 3 and 4 at `550 ms`
step, `12/14` zero-hold on link 5 at `750 to 850 ms` step where the `14%` held
tail pushed 5-link `p90` to `6.25 to 6.54 s`, and `7/7` zero-hold on link 5 past
`2.65 s` at `3.15 s` unwrap p50 / `3.23 s` p90), link 6 hits a `~7.6 s`
server-side hold (`49/49 = 100.0%` held across every arm from `0` to `1000 ms`,
`8.03 to 8.79 s` unwrap / `~9.63 s` capture p50), and links 7 through 10 hit a
`~24 s` wall-clock unlock ceiling regardless of spacing. `--goto-max-links`
caps unwrapped `/goto` links per capture; `--goto-respect-num 1` bounds that cap
by the search URL's `num` parameter (unwrapping only `num` links upfront when
`num < 5` and pulling spare links up to the 5-link ceiling on demand only when
an earlier result resolves to a doorway page, while `num=5` caps strictly at 5
links rather than spilling into link 6's `7.6 s` hold, clamped to
`--goto-max-links`); and `--goto-pace-ms` scales the non-uniform
per-index unlock schedule `(0, 0, 550, 1100, 2650) ms` by `pace_ms / 1000` when
3 to 5 links are unwrapped, while leaving 1-to-2-link and 6-to-10-link captures
unpaced so neither pays extra delay. Each try records `gotoDiscovered`,
`gotoSelected`, `gotoSpareUsed` and `gotoPaced`. When `--goto-refresh-max-ms` or `--goto-pace-ms`
is active, non-first-try attempts and `--retry-window-ms` window sleeps reserve
the expected post-settle `/goto` unwrap time (`minimum_unwrap_seconds`: 24.75 s
for 7-to-10-link pages under a 25 s+ hold cap, 8.75 s for 6-link pages, 2.90 s
for 5-link pages, 1.35 s for 4-link pages, and 0.80 s for 3-link pages at
`--goto-pace-ms 1000`, 5.75 s for unpaced 5-link pages, and 0 s for 1-to-2-link
pages) on top of the 2.5 s try minimum (27.25 s total for `num=10`, 5.40 s for
paced `num=5`), breaking immediately instead of sleeping 10 s into a retry
window that cannot finish unwrapping `/goto` links before the deadline. That
reserve lets a 20 s `maxResults=5` budget and a 42 s
`maxResults=10` budget each fit one post-hold 10 s retry window and end failed
searches at ~12.3 to 12.8 s rather than burning 32 to 57 s across doomed late
windows.
`--ab-arm-b` and `--ab-salt` run an A/B search by search rather than in blocks
of runs. On 2026-10-12, runs of 100 searches on one unchanged setting swung
from 68 to 93 Google answers, as much as any setting tested, so an A/B that
alternates runs measures the hour as much as the setting. With both flags,
each search falls in arm a, the daemon's own settings, or arm b, those given
to `--ab-arm-b` on top, by a hash of the salt and its query with each run of
whitespace as one space. Both arms then share every minute, the pool and its
exits; a report can tell any search's arm from its query alone; and a new salt
each run moves about half the queries to the other arm, so over several runs
each query is seen in both. Arm b takes only settings a search reads for
itself: `--max-attempts`, `--max-block-attempts`, `--proven-first-wait-ms`, `--held-grace-ms`,
`--retry-window-ms`, `--retry-window-tries`, `--retry-window-after-refusals`,
`--retry-window-from-refusal`, `--window-proven-exits`, `--keep-botguard-cookies`,
`--warm-google-session`, `--goto-refresh-max-ms`, `--goto-max-links`,
`--goto-respect-num`, `--goto-pace-ms` and `--fixed-retry-exits`,
with the same bounds as the daemon's own; windows in arm b need `--race-width 1`.
The arms still share one pool, so the proven first-try wait, which changes which browser a search
takes, and retry windows, which free a browser between windows, are measured
beside the other arm's searches rather than in a pool of their own. Each
answer records its arm as `abArm`.
`--browser-engine` (`camoufox` or `chrome`, default `camoufox`) and optional
`--chrome-path` let the daemon run stock Chrome or Chromium instead of Camoufox
behind the same socket protocol, guarded local proxy, exit-health record and
supervisor checks. When `DISPLAY` is unset on Linux, the pool starts one
private `Xvfb` display (`1366x768x24`, `-nolisten tcp`, discovered through
`-displayfd`) shared across its slots and restarts it if it dies. Each slot
spawns Chrome in its own process group with a private `vitna_chrome_slot-*`
profile and `HOME` directory under `TMPDIR`, `--remote-debugging-port=0` on
`127.0.0.1`, `--disable-blink-features=AutomationControlled`, QUIC and
non-proxied WebRTC UDP disabled, and `--host-resolver-rules` mapping every
non-loopback hostname off-host so DNS and TCP stay inside the slot's
`StandingGuardedProxy` (`127.0.0.1:0` with `require_auth=False`, since Chrome
does not take proxy credentials in `--proxy-server`). Playwright attaches over
loopback CDP (`connect_over_cdp`) and reuses the slot's driver across relaunches
when `--fast-relaunch 1` is set. On 2026-11-02 Google's BotGuard refused CDP
`browser.new_context()` (incognito context with `1280x720` device-metrics
overrides) and `context.route_web_socket()`'s injected `WebSocket` wrapper
script, so `ChromePageContext` opens a fresh tab on `contexts[0]` with
page-level request routing and clears cookies and Google origin storage
(`Storage.clearDataForOrigin`) when the tab closes, preserving Google's
challenge-pass cookies (`__Secure-STRP`, `AEC`, `NID`, `SEARCH_SAMESITE` and
legacy `SG_SS`; on 2026-11-03 Google retired `SG_SS` in favor of
`__Secure-STRP` + `NID` + `SEARCH_SAMESITE`, and retaining the full set per
verified exit IP collapsed cold 2-hop BotGuard redirects from `1.03 s` to a
single `200 /search` hop in `0.64 to 0.72 s` across `49/49` warm Camoufox
captures) when `--keep-botguard-cookies 1` is enabled. With `--warm-google-session 1` (off by
default), each stock Chrome slot primes `https://www.google.com/?hl=en` through
its guarded exit on launch and retains Google cookies (`NID`, `AEC`,
`__Secure-STRP`, `DV`, `SEARCH_SAMESITE`) and origin storage across delivered
captures on that same exit IP, wiping cookies and `Storage.clearDataForOrigin`
immediately on any block, capture failure, cold A/B arm or session rotation;
each try records `warmSessionApplied`, `warmSessionRetained`,
`googleSessionWarmup` and `googleSessionWarmupMs`. Because Chromium over CDP suppresses
Playwright `page.on("response")` events on manual opaque redirects,
`BrowserRedirectClient` reads `3xx` `Location` headers from a page CDP session
(`Network.responseReceivedExtraInfo` and `Network.requestWillBeSent`
`redirectResponse`) while waiting out 200 refresh holds under
`--goto-refresh-max-ms`. On the OVH host on 2026-11-02, a 26-slot warm Chrome
pool (10 static ISP lines plus 16 sticky residential lines) plus `Xvfb` warmed
with `--warm-google-session 1` in 13.0 to 15.6 s using 7.98 to 8.05 GB of
physical memory (`Pss`, about 296 to 298 MB per slot), `--fast-relaunch 1`
started replacement Chrome browsers in 274 to 519 ms, and each try records
`browserEngine` (`camoufox` or `chrome`). In sequential 50-query `maxResults=10`
runs at concurrency 3 (`--goto-refresh-max-ms 30000`, `timeoutMs=58000`), cold
Chrome delivered 30/50 (60.0%, 13/50 first-try, 17/37 retry recovery, 290 holds
waited and 12 refused; delivered-only latency 36.4 s p50 and 56.6 s p90, 25.9 s
first-try p50), whereas two runs adding `--warm-google-session 1` and
`--fixed-retry-exits 1` delivered 33/50 (66.0%, 28/50 first-try) and 42/50
(84.0%, 35/50 first-try, 7/15 retry recovery, 345 holds waited and 2 refused;
delivered-only latency 25.9 s p50 and 35.4 s p90, 25.6 s first-try p50), or
75/100 (75.0%, 63/100 first-try) pooled across two 50-query runs of the same
query list. In standalone 100-distinct-query runs on separate fresh 26-slot
pools at concurrency 3 (`--warm-google-session 1 --fixed-retry-exits 1
--goto-respect-num 1 --goto-pace-ms 1000 --goto-refresh-max-ms 30000`),
`maxResults=10` (`timeoutMs=58000`) delivered 79/100 (79.0%, 95% Wilson CI 70.0
to 85.8%; 66/100 first-try, 13/34 retry recovery; quartiles Q1 21/25, Q2 20/25,
Q3 15/25, Q4 23/25; unconditional all-100-query latency 26.1 s p50, 47.7 s p75,
52.9 s p90; delivered-only latency 25.7 s p50, 37.3 s p90; 651 holds waited and
6 refused), and `maxResults=5` (`timeoutMs=35000`, started immediately after on
an empty health ledger while the 10 static lines were hot: 0/9 static ISP vs
72/91 = 79.1% residential first-try) delivered 76/100 (76.0%, 95% CI 66.8 to
83.3%; 72/100 first-try; quartiles Q1 6/25 while cooling the 10 hot static
lines, then Q2 23/25, Q3 23/25, Q4 24/25 on warm residential slots; 0 `/goto`
holds waited or refused across all 76 delivered pages; unconditional
all-100-query latency 5.74 s p50, 28.2 s p75, 32.3 s p90; delivered-only latency
4.06 s p50, 6.18 s p90, 4.03 s first-try p50). Failed queries run to deadline,
so unconditional all-query p90 remains deadline-bound whenever 10% or more of
queries fail. To remove hour-to-hour and pool-draw confounding, a
simultaneous two-period AB/BA crossover across two 13-slot daemons (40 paired
`maxResults=10` queries dispatched in lockstep and swapped across both pools)
delivered 17/40 (42.5%, 10/40 first-try, 25.8 s delivered p50) with
`--warm-google-session 1 --fixed-retry-exits 1` against 4/40 (10.0%, 0/40
first-try, 37.4 s delivered p50) on cold Chrome (13 vs 0 discordant delivered
pairs, two-sided McNemar `p = 0.00024`; 10 vs 0 discordant first-try pairs,
`p = 0.0020`). On a fresh 26-slot pool running 60 `maxResults=5` queries
(`timeoutMs=45000`) split query-by-query via `--ab-arm-b="--goto-pace-ms 1000"`,
overall delivery was 50/60 (83.3%, 36/60 first-try): most of the latency drop
relative to `maxResults=10` comes from Google rendering 5 `/goto` links instead
of 10 (avoiding the ~24 s hold ceiling on links 7 through 10), while
`--goto-pace-ms 1000` cut first-try p50 from 6.80 s (29/32 delivered, 88 holds
waited, median 3 holds per page) to 4.09 s (21/28 delivered, `p = 0.17` vs
unpaced; 0 holds waited across all 21 delivered pages; 4.30 s vs 7.12 s
delivered p50, and 5.02 s vs 7.14 s unconditional all-query p50). Follow-on
batches on an already-pruned pool delivered 19/20 (`maxResults=5`, 4.39 s
delivered p50) and 14/15 (`maxResults=2`, 2.76 s delivered p50, 0.88 s min, where
Google already renders only 2 `/goto` links so pacing and `num` capping are
no-ops). Follow-up 100-query fresh-pool runs with `minimum_unwrap_seconds` and
`_fixed_ready` measured tighter deadlines across all intermediate trials (each
wiping `egress-health.json` while the 10 static lines were already cooling):
cutting to zero retry windows (`12 s` `maxResults=5` and `35 s` `maxResults=10`
on re-sent query sets) ended failures in 2.8 to 3.0 s p50 (5.2 s and 26.9 s
unconditional p90) but dropped delivery to 12/100 and 28/100 (rising to 17/20 in
the final 20 `maxResults=10` queries once proven residential exits stabilized),
while retrying between windows (`15 s` / `35 s`) recovered only 5/86 (5.8%) and
1/62 (1.6%) inside Google's 10 s query hold (19/100 and 39/100 overall, Q4
25/25 on `maxResults=10`; reverted), and an asymmetric `20 s` (1-window) / `35 s`
(0-window) re-run on reused sticky ports delivered 35/100 (`maxResults=5`, 12.1 s
unconditional p50, 13.8 s p90, 4.39 s delivered p50, 12.8 s failed p50) and
3/100 (`maxResults=10` at 44.3 queries/min closed-loop retry flood). Sizing both
budgets for one post-hold 10 s retry window plus unwrap reserve (`20 s`
`maxResults=5` via `VITNA_SEARCH_SERP_GOOGLE_SHORT_SHARE_MS` and `42 s`
`maxResults=10` via `VITNA_SEARCH_SERP_GOOGLE_SHARE_MS`, disjoint sticky ports)
delivered 69/100 on `maxResults=10` (69.0%, 95% CI 59.4 to 77.2%; 67/100
first-try, 0/10 hot static ISP cleared in 10 tries, 67/90 = 74.4% residential
first-try; Q1 1/25, Q2 18/25, Q3 25/25, Q4 25/25, or 60/60 first-try across
deciles D5 to D10; 0 holds refused; unconditional all-100 latency 26.1 s p50,
27.6 s p75 vs 47.7 s at 58 s, 27.8 s p90 vs 52.9 s at 58 s; delivered-only
27.4 s p50, 28.0 s p90; failed-only 12.8 s p50, 14.4 s p90 vs ~53 s at 58 s)
while the repeatedly re-sent `maxResults=5` set delivered 7/100 (`2.5 s` proven
spacing) and 2/100 (`10 s` proven spacing; failed-only 12.3 to 12.5 s p50). On
previously unseen 100-query sets (`bench_query(1000..1099)` and
`bench_query(1100..1199)`), `maxResults=5` at `20 s` delivered 48/100 overall
(48.0%, 95% CI 38.5 to 57.7%; Q1 0/25 and Q2 5/25 during cold residential
warmup, then Q3 21/25 = 84% and Q4 22/25 = 88%, or 43/50 = 86.0% first-try
across the warmed second half; 0 `/goto` holds waited or refused; unconditional
all-100 latency 11.7 s p50, 12.4 s p75 vs 28.2 s at 35 s, 13.6 s p90 vs 32.3 s
at 35 s; delivered-only 4.49 s p50, 5.11 s p90; failed-only 12.3 s p50, 14.9 s
p90) and `maxResults=10` at `42 s` (started 30 s later on a fresh cold pool)
delivered 6/100 (6.0%, 95% CI 2.8 to 12.5%; Q1 0/25, Q2 0/25, Q3 1/25, Q4 5/25;
25.9 s delivered p50; 12.5 s failed p50, 15.5 s unconditional p90). Two follow-up
unseen fresh-pool pairs with `--retry-window-from-refusal 1` and
`--retry-window-after-refusals 1` first exposed `probeBefore` (`ip_changed` on
idle sticky sessions) consuming `--retry-window-tries 1` window turns before
Google was contacted (`bench_query(1200..1299)` at `22 s`: 28/100, Q1 0/25, Q2
0/25, Q3 12/25, Q4 16/25, 4.01 s delivered p50, 13.3 s failed p50, 22
`serp_unreachable`; `bench_query(1300..1399)` at `44 s`: 0/100, 13.1 s failed
p50, 42 `serp_unreachable`), and after moving window-turn accounting past
`probeBefore` (`bench_query(1400..1499)` at `25 s` and `bench_query(1500..1599)`
at `46 s`) reduced `serp_unreachable` to 0/100 in both runs while delivering
2/100 (`D10` 2/10, 4.34 s first-try, 12.9 s failed p50, 16.0 s unconditional
p90) and 0/100 (13.6 s failed p50, 16.3 s unconditional p90), confirming that
one-window budgets bound failed-query tail latency to ~12.3 to 13.6 s p50
(~13.6 to 16.3 s p90) with zero `/goto` holds refused, while cold-pool runs with
`egress-health.json` wiped and all 10 static ISP lines externally hot (~1 to 2%
fresh residential discovery rate per cold IP vs 64 to 100% once proven)
bifurcate sharply by how early 2 to 3 clean residential exits are discovered.
A simultaneous paired head-to-head benchmark across two symmetric 13-slot cold
pools (`5` randomly split static ISP lines plus `8` disjoint sticky residential
ports per daemon, `egress-health.json` empty on both, concurrency 3 per daemon,
`200` previously unseen distinct queries `bench_query(1600..1799)` paired by
topic and coin-flipped per pair so zero query strings overlapped across engines;
even pairs `maxResults=5` at `25 s`, odd pairs `maxResults=10` at `46 s`; shared
`--goto-refresh-max-ms 30000 --goto-respect-num 1 --goto-pace-ms 1000
--fixed-retry-exits 1 --retry-window-ms 10000 --retry-window-from-refusal 1
--retry-window-after-refusals 2 --retry-window-tries 2 --window-proven-exits 1
--keep-proven-exits 1 --spare-gap-refusals 1 --proven-first-wait-ms 1000
--proven-interval-ms 2500 --fast-relaunch 1`, plus `--warm-google-session 1` on
stock Chrome) isolated cold-pool exit discovery from hour-to-hour variance:
Camoufox delivered 68/100 (68.0%, 95% Wilson CI 58.3 to 76.3%; 54/100 first-try,
4/15 static ISP and 50/85 = 58.8% residential first-try; 14/46 = 30.4% retry
recovery across 190 tries and 40 window sleeps; quartiles Q1 14/25, Q2 14/25, Q3
21/25, Q4 19/25, or 40/50 = 80.0% with 95% CI 67.0 to 88.8% and 37/50 = 74.0%
residential first-try across the warmed second half; 29 `serp_blocked` and 3
`serp_timeout`; unconditional all-100 latency 16.6 s p50, 26.1 s p75, 27.5 s
p90; delivered-only 25.7 s p50, 28.0 s p90; failed-only 15.4 s p50, 17.9 s p90)
against 0/100 (0.0%, 95% CI 0.0 to 3.7%; 0/6 static ISP and 0/94 residential
first-try across 305 tries and 90 window sleeps; 99 `serp_blocked` and 1
`serp_timeout`; failed-only 14.3 s p50, 17.2 s p90) on stock Chrome (68 vs 0
discordant topic pairs, two-sided exact McNemar `p = 6.8e-21`). Within
Camoufox's 100 queries, `maxResults=5` (`n = 50`, `25 s`) delivered 33/50
(66.0%, 95% CI 52.2 to 77.6%; 25/50 first-try at 4.85 s p50 and 5.60 s p90,
8/25 = 32.0% retry recovery; 0 `/goto` holds waited and 0 refused; unconditional
latency 6.63 s p50, 16.5 s p75, 18.7 s p90; delivered-only 5.17 s p50, 18.6 s
p90; failed-only 15.6 s p50, 19.7 s p90) and `maxResults=10` (`n = 50`, `46 s`)
delivered 35/50 (70.0%, 95% CI 56.2 to 80.9%; 29/50 first-try at 26.3 s p50 and
27.8 s p90, 6/21 = 28.6% retry recovery; 280 `/goto` holds waited and 0 refused;
unconditional latency 26.1 s p50, 26.8 s p75, 39.2 s p90; delivered-only 26.4 s
p50, 39.8 s p90; failed-only 14.2 s p50, 16.7 s p90). Both engines exited with
zero surviving processes or leftover directories (Camoufox 4.65 GB PSS across 13
slots, 625 ms fast-relaunch p50; stock Chrome 4.24 GB peak PSS across 13 slots,
306 ms fast-relaunch p50); `camoufox` remains the default engine because its
cold residential discovery rate held up where stock Chrome's collapsed, while
`--browser-engine chrome` stays opt-in.
#### Proxy traffic per capture
Residential exits are billed per GB. On 2026-10-08 a delivering attempt moved
about 1.1 MB through its exit, 952 KB of it Google's script bundles
(`www.google.com/xjs/`), OneGoogle bar (`www.gstatic.com/og/`) and fonts, all
cacheable for a year and downloaded again by every fresh context. Organic
results are in the HTML, so by default the daemon does not load those assets;
BotGuard (`/js/bg/`) and everything else still load. A delivering attempt then
moves 165 to 187 KB, and 23 such captures each delivered ten organic results.
`--load-static-assets` restores them. Whatever static assets still load are
kept in a cache shared by the slots (`--static-cache-mb`, default 96, 0 to
disable): public, long-lived, cookie-free responses only, keyed by exact URL.
A challenged attempt moves about 62 KB.
## Alternative retrieval experiments
On 2026-10-05, an isolated Python environment on the owned OVH host ran
Google-only SearXNG at [revision
`d48c4b555421e824342c51d68482dd0898e54d0f`](https://github.com/searxng/searxng/tree/d48c4b555421e824342c51d68482dd0898e54d0f),
bound to loopback. Its `/wml/search` request used `curl_cffi` with a Nokia
user agent. All three measured paths received Google HTTP 403; SearXNG
returned API HTTP 200 with zero results and an access-denied report:
| Tested outbound path | Observed latency |
|---|---|
| Host default connection | About 421 ms |
| Fixed owned IPv6 | About 371 ms |
| Private SSH CONNECT relay through the PC's own network | About 645 ms; one relay connection transferred 8,641 bytes |
The host worked, but these samples did not establish unchallenged Google
retrieval. They used no third-party search service, Custom Search fallback,
or production provider integration.
4get's pinned [Google implementation](https://git.lolcat.ca/lolcat/4get/src/commit/03ba5d7b5ed3dc91b3e7d8b6278d8f52818a99ae/scraper/google.php)
and [configuration guide](https://git.lolcat.ca/lolcat/4get/src/commit/03ba5d7b5ed3dc91b3e7d8b6278d8f52818a99ae/docs/configure.md)
at revision `03ba5d7b5ed3dc91b3e7d8b6278d8f52818a99ae` require a separate
4play Firefox service for Google; the 4get Docker container alone is
insufficient. This alternative was reviewed, not installed or live-tested.
Whoogle's [official end-of-life announcement](https://github.com/benbusby/whoogle-search#whoogle-has-reached-the-end-of-the-road--24-jul-2026)
dated July 24, 2026 says it no longer returns results: Google blocked its
last working non-JavaScript user agent, and the Custom Search fallback was
unworkable. The repository was archived on GitHub on August 14, 2026. Whoogle was
reviewed only, not installed or live-tested.
The reviewed GitHub topic entries largely required paid-provider accounts.
An independent candidate, [omkarcloud/google-scraper at revision
`11a5562251a8c330475a1fd312e5f572db85d7e8`](https://github.com/omkarcloud/google-scraper/tree/11a5562251a8c330475a1fd312e5f572db85d7e8),
uses a Camoufox Firefox session to obtain cookie identity, then replays
requests with `curl_cffi`. Its optional paid solver and proxy services were
excluded from this investigation.
An earlier separate private experiment installed Camoufox 0.5.7, Playwright 1.62,
and browser build 156 beta 34 on the owned OVH host, with verified cgroup
process cleanup. A fresh direct headless capture for
`site:bis.gov H200 January 2026 license review policy` returned HTTP 200
without a detected challenge in 36.237 seconds, including browser startup.
Four BIS organic results were verified with the existing `parseSerpHtml`
parser, canonical search identity, and single-site guard.
After an admission gap, a separate direct Python SSL diagnostic made
header-only GETs to the four observed opaque Google `/goto` links. All
returned HTTP 302 destinations on `www.bis.gov` or `media.bis.gov`, with
423 ms of actual network time. It followed no publisher destination and
closed every connection before recording proof. This step used no Firefox
cookies, user-agent spoofing, or third-party provider.
These were separate admitted diagnostics, not an end-to-end API latency
measurement or production integration, and do not establish sustained
reliability. The browser diagnostic checks identity only at the final
search URL and accounts for bytes after response bodies complete; these
limitations apply only to the private experiment.
## Configure proxy routes
Existing URL strings retain their fixed credentials. A structured entry
accepts `url`, optional `egressId`, and optional `session` with `username`
and/or `password` credential templates. For example, this placeholder
configuration requires the operator's provider to support the illustrated
username syntax:
```json
[
{
"url": "http://proxy.operator.example:8000",
"egressId": "shared-exit",
"session": {
"username": "ACCOUNT-session-{session}",
"password": "OPERATOR_PASSWORD"
}
}
]
```
Store the array in private `VITNA_SEARCH_SERP_PROXIES`. Credential templates
are literal, unencoded strings; Vitna encodes the resulting URL credentials.
At least one supplied template must contain `{session}`; other placeholders
are refused. Each admitted attempt generates one cryptographically random
32-character lowercase hexadecimal token and substitutes that same token
everywhere `{session}` occurs. That attempt uses one generated proxy URL
throughout its isolated browser context. A later attempt generates a new
token. A supplied template replaces its entire credential field; fields
without a template keep the URL's static credential. Template syntax and
the provider's session stickiness remain operator responsibilities.
`egressId` is an operator declaration that routes share a physical exit.
It accepts 1 to 32 lowercase letters, digits, and hyphens, starting with a
letter. `direct`, `browser-direct`, and `proxy-N` names are reserved.
Different proxy URLs with the same label share per-engine pacing,
concurrency, and block cooldown in the service process. Declare the shared
label when different endpoints use the same exit; separate route labels do
not establish separate physical exits. Replicas still have no shared
coordinator.
Routes receive safe `proxy-1` through `proxy-20` identifiers in array order.
Duplicate detection compares the endpoint and effective fixed credentials
or templates. Changing a credential that its template overrides, or
assigning another egress label, does not make the same effective route
distinct. Every attempt uses an isolated context without ordinary browser
state, and a configured proxy pool has no implicit local-egress fallback.
## Validate an operator route
Run the explicit live checker after enabling SERP and configuring the
intended runtime and egress routes:
```bash
node --env-file=.env.search platform/search/serp-check.mjs --engines google,bing --rounds 1 --timeout-ms 10000
```
This command performs actual engine HTTPS/browser requests. It is separate
from startup, readiness, and the local fixture integration test. By default
it checks both engines once, using three fixed public queries per engine:
six fresh JSON retrieval requests. Each request still has its configured route
attempt budget. `--engines` can select `google`, `bing`, or both;
`--rounds` accepts 1 to 3, bounding the run to at most eighteen samples.
`--timeout-ms` accepts 100 to 60,000 and is clamped to the configured SERP
timeout. Omitting it uses that configured timeout.
Before another fresh sample, the checker respects the observed
`nextRequestAt` and can wait up to sixty seconds for pacing. A sample refused
with `serp_routes_busy` or `serp_routes_throttled` started no capture; the
checker waits that refusal's interval and retries the same sample, within
sixty seconds of such waits and sixty retries per sample. A refusal needing
a longer wait remains a failure. These client-side waits are separate from
the sample's latency measurement, and each retry has its own per-probe
deadline. The checker does not sleep through cooldowns or retry failed
samples; those outcomes remain failures in its report.
The command writes a JSON report to stdout with `ok`, `samples`, `delivered`,
`successRate`, latency percentiles `p50Ms` and `p95Ms`, actual route-attempt
count `attempts`, safe per-route counts in `routeAttempts`, the configured
`transport`, and numeric route count `configuredRoutes`.
Its `engines` entries report those sample/delivery/latency counts and
failure-code totals per engine. `successRate` is a fraction from 0 to 1.
`samples` counts logical retrieval requests;
`attempts` includes the actual backend attempts and retries. Queries, search URLs,
result content, HTML, credentials, and proxy addresses are omitted. Save
stdout to `.search-data/serp-check.json` if an operational record is useful.
Configuration or invocation failures return a safe `ok: false, error` report.
Exit code 0 means every requested response was delivered; exit code 1 means
a request or the setup failed. A successful bounded run establishes outcomes for its
sampled requests and configured routes, without establishing whole-engine
coverage or future availability.
## Diagnose a browser route
The optional standalone browser diagnostic submits one fixed public Google
query through the current connection, using an isolated browser context.
Choose one configuration for each invocation:
```bash
node platform/search/serp-browser-check.mjs --channel=msedge
node platform/search/serp-browser-check.mjs --channel=msedge --headed --navigation=form
```
The default is the `native` profile, headless mode, direct search navigation,
and a 12,000 ms timeout. `--channel=msedge` or `--channel=chrome` selects an
installed browser; otherwise the configured SERP or Unblocker channel is
used, then bundled Chromium. `--headed` opens an automated browser window.
`--navigation=direct` opens the search URL; `--navigation=form` opens the
Google homepage and submits its visible search form. Both use one deadline.
`--timeout-ms` accepts 1,000 to 20,000, using the equals form, such as
`--timeout-ms=20000`.
`--profile=cloakbrowser` uses an already installed runtime selected by
`VITNA_SEARCH_SERP_BROWSER_PATH`, or its Unblocker path fallback. The command
does not install a browser. It refuses a nonempty `VITNA_SEARCH_SERP_PROXIES`
pool; use the proxy route diagnostic below for that comparison, or the
retrieval checker above to validate production route selection.
This diagnostic makes no automatic retry and runs no challenge solver.
Every configuration shares persisted Google/current-connection health in
`.search-data/serp-browser-check-state.json`, protected by an exclusive
`.lock` file. `--state=PATH` can select another state file; use the same path
when comparing profiles, headed mode, or navigation. A cooling state refuses
the probe before browser launch. The first observed block starts a five-minute
cooldown; repeated blocks double it up to one hour. A successful capture
resets the block streak. Changing browser settings does not bypass this
shared cooldown, and a concurrent diagnostic is refused while its lease is held.
Healthy diagnostic starts durably reserve an interval selected uniformly
between five and fifteen seconds before capture. The state records
`nextRequestAt` across CLI
invocations, browser profiles, headed or form modes, and generated session
tokens. An early repeated probe returns `serp_routes_throttled` with zero
attempts; a cooling route returns `serp_routes_cooling`. Neither diagnostic
waits for pacing or automatically retries. State version 2 adds this pacing
record; version 1 files migrate automatically while preserving their block
history and cooldowns. The file guard coordinates these diagnostics on its
local filesystem. Ordinary production routes use separate process-local
coordination; the experimental owned adapter can share this durable gate
when its state path and declared egress identity match.
A crash or a failed state save can leave the lock in place. Repair the state
storage and confirm that the diagnostic has ended before clearing an abandoned
lock. A failed save returns `serp_probe_state_invalid` and prevents another
invocation from trusting an older cooldown record.
The command writes a JSON report to stdout with `ok`, `engine`, `profile`,
`headed`, `navigation`, `attempts`, and safe observed `stages`. A failure
includes a fixed `error` code. `latencyMs` is included after an attempted
capture; `count` is included only on success. Available state is reported as
numeric `consecutiveBlocks`, `cooldownUntil`, `nextRequestAt`, and
`retryAfterMs`. Queries,
URLs, result text, HTML, browser paths, and credentials are omitted. Exit code
0 indicates a parsed capture; exit code 1 indicates refusal or failure.
`navigationResponses` retains at most eight main-frame HTTP observations,
with only a fixed destination category, status code, and challenge flag.
It records challenge responses before an immediate abort closes the page.
When a page was opened, `network` reports request lifecycle counts and fixed
method, resource-type, and failure categories. Each count is capped at 1,024;
`truncated` marks overflow. These observations help identify failed POSTs or
redirects without retaining request URLs, bodies, cookies, or headers.
A comparison also requires the operator to make the same search manually
in an ordinary unsigned-in browser on the same connection. An automated
headed run is not that manual baseline. Automated failures alone do not
establish that the connection's IP address is blocked.
## Compare configured proxy routes
The proxy route diagnostic uses the same bounded browser probe on the
configured proxy list:
```bash
node --env-file=.env.search platform/search/serp-route-check.mjs --channel=msedge --navigation=form
```
By default it submits one fixed public Google query per configured proxy,
up to twenty routes. It performs the probes sequentially and waits for each
browser to close before the next route. It does not retry a failure or wait
through pacing or a cooldown. With no configured proxies, it returns
`serp_proxy_not_configured` before launching a browser. It does not add an
implicit current-connection probe.
An explicit selection can include that baseline:
```bash
node --env-file=.env.search platform/search/serp-route-check.mjs --routes=direct,proxy-1 --channel=msedge --navigation=form
```
`--routes` accepts up to twenty-one unique labels: `direct` and the
configured `proxy-1` through `proxy-20` routes. The whole selection is
validated before traffic; an unconfigured label is refused. `direct` here
means the isolated browser on the current connection. The standalone
`serp-browser-check.mjs` CLI remains current-connection-only.
The route checker accepts the same `--profile`, `--headed`, `--navigation`,
`--channel`, `--timeout-ms`, and `--state` browser options described above.
Its defaults are headless, direct navigation, and a 12,000 ms deadline per
probe; profile and channel inherit configured SERP or Unblocker settings,
with the native profile as the fallback. Only already installed runtimes
and operator-supplied proxy credentials are used.
Both browser diagnostics use the same state file and exclusive lock.
Persisted proxy health uses opaque identities derived from the effective
configuration. An explicit shared egress label gives aliases the same key,
including across browser profiles, navigation modes, and credential
rotation. That key shares both persisted pacing and block cooldown, so a new
session token does not bypass either guard. The current-connection
`google:direct` ledger remains shared with
the local browser diagnostic. Keep the same `--state` path for comparisons;
paced or cooling routes are refused before launch with zero attempts. Shared
aliases can therefore be throttled within one sequential checker run. The
same five-minute first block penalty and one-hour cap apply. These local
diagnostic ledgers do not coordinate ordinary production routes or distributed
replicas. The experimental owned adapter can share their durable gate on the
same filesystem and declared egress identity.
Neither identity scheme measures the provider's actual exit IP.
Stdout contains a sanitized JSON aggregate with `ok`, `samples`, `delivered`,
and `routes`. Each route entry has its safe route label and browser probe
outcome, stages, latency, result count on success, and available numeric
pacing and cooldown counters, including optional `nextRequestAt`. Queries,
URLs, credentials, credential templates,
session tokens, and persisted identity hashes are omitted. Configuration
failures return a fixed safe `error` code before traffic. Exit code 0 means
every selected route delivered a parsed capture; exit code 1 means a
refusal or failure. A successful sample does not establish Google coverage,
physical exit identity, or provider session stickiness.
## Browser container
Build the dedicated SERP image from the repository root:
```bash
docker build -f deploy/vitna-search-serp.Dockerfile -t vitna-search-serp .
```
This image includes the isolated browser runtime and selects `auto` SERP
transport with browser fallback. SERP retrieval stays off until the operator
sets `VITNA_SEARCH_SERP_ENABLED=true` and `VITNA_SERP_RESALE_APPROVED=true`,
as described in [Run without an index](#run-without-an-index).
A native index is optional. Supply those settings, the private search
client registry and any operator proxy configuration at runtime, and use a
Linux host or container runtime that permits the Chromium sandbox and user
namespaces. The image's readiness probe checks service configuration; it
does not establish successful access to a search engine. `GET /readyz`
follows the default `/v1/search` route: when a native index is the default
provider, it returns HTTP 503 while that index is empty or unavailable, even
with SERP retrieval enabled. A SERP-only instance is ready while its
retrieval adapter reports itself available. Authenticated `GET /v1/status`
still reports SERP availability if the
native index status read fails, with `index: {ready: false, unavailable: true}`.
The existing `deploy/vitna-search.Dockerfile` also supports direct SERP
retrieval without browser dependencies. Its defaults assume a native index
at `/data/index.sqlite` and `VITNA_SEARCH_PROVIDERS=vitna`. For SERP-only
operation, override the index to an empty value, replace that provider order
with `brave,exa,tavily` without supplying gateway credentials, and set
`VITNA_SEARCH_SERP_ENABLED=true`, `VITNA_SERP_RESALE_APPROVED=true` and
`VITNA_SEARCH_SERP_TRANSPORT=direct`.
Unconfigured legacy providers are skipped;
SERP retrieval remains a separate capability.
Direct egress may encounter consent, rate limits, or challenges. Browser
transport and optional operator proxies add another retrieval route but do
not guarantee success, verified geographic targeting, or broad production
coverage. This implementation does not yet provide a full substitute for a
vendor's distributed retrieval infrastructure. Local live Google browser
testing has returned `serp_blocked`; verify access on the intended operator
route before relying on it.
## Authentication, quotas, and failures
Single-page SERP requests use the same search client registry, per-client minute/day
quotas, service concurrency limit, and whole-request deadline as indexed
search, extraction, and research. An admitted SERP failure consumes one
quota slot, including route-busy, cooling, or throttled responses. Invalid requests and
the shared API's `service_busy` refusal consume no quota. Receipts record
operation, request identifier, outcome, and latency, without the query,
search URL, fetched HTML, snippets, authorization header, or proxy secrets.
Responses use `Cache-Control: no-store`. Failures use the shared
`{"error":{"code":"...","message":"..."}}` envelope. Invalid input
returns HTTP 400; authentication uses HTTP 401, and shared rate limits use
HTTP 429. Retrieval and parsing failures return a safe explicit error,
without forwarding a challenge page or an upstream error body as a result.
Every retrieval failure, a missed deadline included, returns HTTP 503 and is
told apart by its code. A proxy in front of a hosted instance can replace the
body of a 502 or 504 with its own page, which would hide the code; a 503 keeps
it. Before 2026-10-22 these failures returned 502, and `serp_timeout` 504.
A failed request's receipt records the same code as its `outcome`, so an
org's console activity and the service's journal tell failures apart too.
| Status and code | Meaning |
|---|---|
| 503 `serp_unavailable` | SERP retrieval is not enabled on this instance |
| 503 `serp_jobs_unavailable` | Multi-page jobs or their private storage are unavailable |
| 429 `serp_jobs_busy` | Job capacity is full; includes `Retry-After` |
| 503 `serp_runtime_unavailable` | A required browser runtime could not be loaded or started; refusals while its route is paused include `Retry-After` |
| 503 `serp_routes_busy` | Eligible retrieval routes are at their concurrency limit (a route that rotates its own exits first waits for a free slot while 8 s of the deadline remain); includes `Retry-After` |
| 503 `serp_routes_cooling` | Eligible retrieval routes are cooling after failures; includes `Retry-After` |
| 503 `serp_routes_throttled` | Independent retrieval starts are paced; includes `Retry-After` and no new attempts |
| 503 `serp_blocked` | Challenge, consent or sign-in page, or access block |
| 503 `serp_parse_failed` | Page not recognized by the organic-result parser |
| 503 `serp_unreachable` | Search engine could not be reached |
| 503 `serp_response_too_large` | Fetched HTML exceeds the 2 MiB limit |
| 503 `serp_invalid_response` | Upstream response is not supported nonempty HTML, or a browser challenge flag without HTTP 403 or 429 |
| 503 `serp_redirect_blocked` | Redirect outside the supported search route or past the redirect limit |
| 503 `serp_query_mismatch` | Final search URL changed the requested query or offset, or results violated an unambiguous single positive `site:` hostname constraint |
| 503 `serp_timeout` | Search engine did not finish within the request deadline |
| 499 `request_cancelled` | Caller cancelled retrieval |
The `site:` result guard permits the exact hostname and its subdomains. It
skips complex, quoted, multiple-site, or invalid-DNS syntax; it does not
interpret arbitrary search operators.
## Indexed search and cited research
`POST /v1/search` with `mode: "index"` searches Vitna's stored corpus.
`POST /v1/extract` returns stored indexed page text. `POST /v1/research`
assembles cited passages from indexed evidence. These endpoints remain
available when the operator configures their independent index and crawler.
Research source selection uses bounded candidate metadata before reading,
then rechecks the extracted page's URL and title before passage selection.
A small registry recognizes PostgreSQL, Python, and Node.js ownership and
canonical documentation version paths. When the query identifies one
requested version, explicit incompatible versions in those paths or
subject-specific titles are excluded; unknown or mixed
version evidence stays eligible. Snippet numbers, URL query parameters,
and arbitrary numeric paths do not establish a version. Cross-product or
multiple-version queries do not infer one requested version. Documentation
queries prefer recognized documentation metadata, known primary domains,
then compatible version evidence, with stable input-order ties.
`coverage.selectionFilteredSources` counts selection exclusions. This
policy does not verify content facts or establish comprehensive coverage.
Private controlled replays on 2026-10-05 checked these changes without new
Google captures. Replaying stored HTML and exact redirect mappings produced
61 result identities across seven pages from 64 original rows and 62 legacy
URL identities. It retained the first tracked YouTube listing and its
original source, excluding the later bare-link variant; distinct identities
do not establish distinct content.
For the PostgreSQL 18 selection replay, known primary-domain sources among
the top five increased from four to five, and the canonical 9.0 manual was
excluded. Three separately guarded direct publisher reads returned HTTP 200
and passed the URL/title version checks. A native research replay over those
real pages returned four exact Unicode-offset citations covering 685 quoted
characters from the 18.0 release notes. It remained `partial` and
non-exhaustive, using one source after three attempts and two skips. Saved raw Google results were
unchanged, and the replay did not mutate the indexed corpus or API results.
These are bounded regression observations, not an independent quality
benchmark or proof of broader factual accuracy or coverage.
SERP retrieval does not enqueue native discovery, add organic links to the
index, fetch result pages, or produce cited research. A SERP snippet alone
is not the indexed source text needed by the research API. See the
[search overview](overview.md) for those separate contracts and the
[migration guide](live-engine.md) for former Live/Hybrid callers.