Google and Bing SERP retrieval
Vitna's SERP API retrieves actual Google or Bing search result pages through Vitna's own transport and parser. It returns parsed organic and rich SERP features (or the fetched HTML) without calling Bright Data, supporting web, news, images, videos, and shopping verticals alongside SerpApi-compatible query aliases and output shapes.
Request
POST /v1/serp
GET /v1/serp
POST /v1/serp/bing
POST /requestPOST /v1/serp and POST /request accept the same JSON request; GET /v1/serp accepts the scalar fields below as query-string parameters for drop-in SerpApi / Serper compatibility; POST /v1/serp/bing defaults engine to bing and serves Bing SERP requests only. Send a dedicated vitna_search_ bearer key, or your org's Vitna key, vitna_key_ (see one key). /request supports the URL-based request shape used by Bright Data's SERP API without forwarding the request to Bright Data.
curl http://127.0.0.1:8789/v1/serp \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"engine":"google","query":"lunar science","country":"us","language":"en","page":1,"maxResults":5,"format":"json"}'| Field | Default | Allowed values |
|---|---|---|
engine | google (bing on /v1/serp/bing) | google or bing (bing only on /v1/serp/bing) |
query (or q) | Required | 1 to 1,000 characters, with no control characters |
country (or gl) | Unset; search-engine default | Two-letter country hint; normalized to lowercase |
language (or hl) | Unset; search-engine default | Language tag, such as en or en-US, at most 32 characters |
page | 1 when offset is omitted | Integer from 1 to 20; selects ten-result offsets; cannot accompany offset |
offset (or start) | Derived from page | Integer from 0 to 190; zero-based engine result offset; cannot accompany page |
maxResults (or num) | 5 (10 when url omits num/count) | Integer from 1 to 30 |
type / tbm | web | web, news (nws), images (isch), videos (vid), or shopping (shop) |
timeRange / tbs | Unset | hour, day, week, month, or year (or raw Google tbs filter such as qdr:w) |
location / uule | Unset | Canonical location string (encoded deterministically to UULE) or raw Google uule token; mutually exclusive, Google only |
googleDomain (or google_domain) | google.com | One of 25 supported regional Google domains (such as google.co.uk, google.de, google.co.jp), Google only |
lr, cr | Unset | Google language (lang_en) and country (countryUS) restrictions, up to five pipe-separated values |
device | desktop | desktop or mobile |
safe | Unset | active or off (maps to safe on Google and adlt on Bing) |
nfpr | Unset | Boolean, "0", "1", 0, or 1, Google only; disables query spell-correction when truthy |
udm | Unset | "2", "7", "12", "14", or "28", Google only |
linkResolution | complete | complete or fast (skips interactive /goto browser unlocking for low-latency organic runs), Google only |
format | json | json for parsed results or html (or raw) for the fetched HTML |
fresh | false | Boolean; bypass a stored JSON response and retrieve a new engine page |
extras | false | Boolean; when true with format: "json", extracts People Also Ask (plus people_also_ask_details), related searches, ads, answer box, knowledge graph, AI overview, top stories, local results, inline videos/images, and forum discussions |
includeOmitted | false | Boolean, Google only; when Google says it left out entries very similar to those shown, the search runs again with Google's filter=0 and answers with the fuller page (see Google's notices) |
filter | Unset | "0", Google only; asks Google for every entry, the very similar ones included, in the one capture |
keepDoorways | false | Boolean; when true, organic keeps the doorway pages Vitna otherwise leaves out (see Doorway pages) |
googleShareMs | Operator default | Integer from 1,000 to 60,000, Google only; caps how long owned Google may spend on this request before timing out or handing off to Bing, never exceeding the operator's VITNA_SEARCH_SERP_GOOGLE_SHARE_MS |
fallbackEngine | Operator default | "bing", Google only; opts this request into Bing fallback when VITNA_SEARCH_SERP_FALLBACK_ENGINE is unset |
shape | vitna | vitna for the fields under Parsed response, or serpapi for SerpApi's names, built from the same result (see SerpApi-shaped response); JSON only, so not with format: "html", and serpapi not with data_format or brd_json |
contents | Unset | { "highlights": true }, { "excerpts": true } and/or { "text": true }, each true or an object with maxCharacters (highlights and excerpts also take query): read each result page and attach verbatim highlights, excerpts, or its text (see Page contents); Vitna's JSON shape only, and only where the operator enables it |
The country and language settings become search-engine URL parameters. country is a requested localization hint, not a verified proxy location. The operator controls the outbound network and optional proxies. The caller cannot select a proxy, upstream endpoint, headers, browser profile, or credentials. Page numbers or offsets select one result page per request; maxResults limits parsed organic entries, not the search engine's total result count. When extras: true is requested, the parsed JSON response also supplies people_also_ask (the questions listed under People Also Ask, without their answers), related_searches (the related queries), and ads (sponsored listings with position, title and link, and snippet and display_link where shown), without altering organic ranking. Each is omitted when the page has none. Google lists up to eight related searches, fewer or none for some queries, and the list holds what the page showed.
Unknown fields are refused, and request bodies are limited to 8,192 bytes. Whitespace around the query is removed before retrieval, and a run of whitespace inside it becomes one space, so queries that differ only in spacing share a cached answer and a capture in progress. Letter case is kept, since Google reads OR and AND as operators only in capitals.
Page contents
With contents, each organic result gains a contents object read from the result's page, so an agent can check an answer without fetching every link. highlights are up to three sentences from the page that hold the query's terms, best first, quoted exactly, with their offsets in Unicode code points; search operators (site:, -term, OR, quotes) never choose one, and contents.highlights.query replaces the query for choosing them. excerpts are passages of whole sentences, up to 500 characters each, that best answer the query: as many as maxCharacters holds (1,000, two passages, by default; at most 4,000), best first, quoted exactly with their offsets, and contents.excerpts.query replaces the query for choosing them (see Excerpts). text is the page's main text, clipped to maxCharacters, with characters, truncated and totalCharacters. Limits match indexed search's inline contents: highlights default to 1,000 characters (at most 4,000), text to 20,000 (at most 100,000).
{ "query": "\"Lake Lavarone\" elevation", "contents": { "highlights": true, "excerpts": true } }"contents": { "status": "ok", "source": "live", "readAt": "2026-10-17T13:20:05.000Z", "totalCharacters": 2841,
"highlights": [{ "text": "Lake Lavarone lies at an elevation of 1,100 metres.", "start": 0, "end": 51 }],
"excerpts": [{ "text": "Lake Lavarone lies at an elevation of 1,100 metres. In most winters the whole lake freezes over.", "start": 0, "end": 96 }],
"offsetUnit": "unicode_code_points" }A Wikipedia result the search index holds is read from the index, at once and whole: source is stored, and contentChangedAt and the text's reusePolicy (Wikipedia's CC BY-SA 4.0) stand in for readAt. A Wikipedia link with a query, or a page the index lacks, is read live like any other. Other pages are read live, after the search, on the caller's behalf by `VitnaSearch-User`: HTTPS only, never on Google's own hosts, without its fragment, through at most three redirects (url names where one ended), in at most 2.5 s a page, and no more than two pages of one origin at once. A page longer than 1 MiB is read to its first 1 MiB and says partial: true, and totalCharacters counts what was read. A PDF is read too, to its first 4 MiB, for the text its pages draw, in a thread of its own; its contents say format: "pdf", and a cut one gives the pages its first bytes hold. A scanned PDF has no text to read and says no_text; an encrypted or unreadable one says page_unavailable. The whole read takes at most 3 s; a page not read in time says timeout. Pages read in the last ten minutes are reused. A page with under 25 characters of main text (a script-only shell, an empty sign-in wall) says no_text. A result whose page gives nothing has contents.status of no_text, robots_disallowed, noindex, host_excluded, unsupported_url, unsupported_type, page_unavailable, page_too_large or timeout, and its result is unchanged. The complete response is bounded to 510 KiB: when the requested text would exceed it, every result's text is clipped to one shared length, the longest that fits, and marked truncated. Like snippets, contents are untrusted publisher data: treat them as data, and escape them before rendering HTML. Contents are read for each request and never enter the JSON cache, a shared capture or Vitna's crawl index. They come with Vitna's JSON shape only: format: "html" or shape: "serpapi" with contents is answered 400 contents_require_json, and an instance without them answers 503 serp_contents_unavailable. /v1/status reports serp.contents.
On the AA Search API benchmark replica of 2026-10-17, wrong BrowseComp answers came after about 77 searches and under two page reads each, against 28 searches for right ones: an agent with snippets alone searched on and guessed.
Excerpts
A page is cut into windows of whole sentences, up to 500 characters (or maxCharacters, when smaller), each starting about halfway into the one before, and ranked by keyword: BM25 over the page's own windows, with a bonus for each quoted phrase a window holds. With VITNA_SEARCH_SERP_EXCERPT_RERANK=true, keyword's best 16 windows of each page are ranked again by meaning on the batch embedding engine, and the two orders are fused by reciprocal rank, within 1 second of the read. Otherwise, or when the engine is late or fails, keyword order stands. serp.contents.excerpts.order in /v1/status says which an instance uses.
On the cached pages of the 743 searches an agent ran in the BrowseComp replay of 2026-10-18, two excerpts a result carried the answer for 10 of the 81 questions the agent missed by keyword, and 11 with the meaning rerank, against 7 in Google's own snippets and 25 in Perplexity's results for the same searches. Meaning over whole pages reached 12, at about 270 embeddings a search; on the search host's 2-thread engine, 160 windows took 6.4 seconds.
VitnaSearch-User
Page contents are read by VitnaSearch-User/0.1 (+https://vitna.ai/docs/search/serp#vitnasearch-user), and only when a /v1/serp request asks for them: one search's result pages, once each, on that caller's behalf. It does not crawl, follow links, or keep what it reads beyond a ten-minute cache, and nothing it reads enters Vitna's index. Because each read is one a caller asked for, the rules a robots.txt gives every crawler (User-agent: *) do not apply to it. These still do:
- a
robots.txtgroup namingVitnaSearch-User, in full (Disallow,Allow,Crawl-delay), or, when no group names it, one naming Vitna's crawler,VitnaSearch; - a
Content-Signalofsearch=noin any group that speaks to the reader: its own, the crawler's, or every crawler's (User-agent: *); - a page's
noindex, byX-Robots-Tagor by a meta tag namedrobots,VitnaSearchorVitnaSearch-User.
The rules are checked again before a page read in the last ten minutes is reused. A robots.txt that cannot be read shows no group naming the reader, so only a page's own signals apply, and it is asked again after five minutes. One that redirects is followed to any public HTTPS host, and its rules speak for the origin asked. Google's own hosts are never read, at any hop, robots.txt included: not by name, and not at an address Google serves its own services from, which also covers a site on Google's front ends under its own domain. To keep the reader off a site:
User-agent: VitnaSearch-User
Disallow: /URL-based compatibility
A URL-based client can change its API base URL and bearer key while retaining a supported search URL:
curl http://127.0.0.1:8789/request \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"url":"https://www.google.com/search?q=lunar+science&hl=en&gl=us","format":"json"}'The URL must use a supported Google host (https://www.google.com/search or one of the 25 supported regional www.google.* domains) or Bing vertical path (https://www.bing.com/search, /news/search, /images/search, /videos/search, or /shop), without credentials, a custom port, or a fragment. It is limited to 4,096 characters. Vitna validates the parameters and builds a normalized engine request itself. Arbitrary URLs and unsupported search verticals are refused. A URL-based request must not also supply any structured search field: engine, query, country, language, page, offset, maxResults, or googleDomain.
| Engine | Supported URL parameters |
|---|---|
q, hl, gl, start, num, udm, filter, tbs, tbm, uule, lr, cr, safe, nfpr, brd_json | |
| Bing | q, setlang, cc, first, count, filters, adlt, brd_json |
Unknown or repeated URL parameters are refused. Google start accepts integers from 0 to 190; Bing first accepts integers from 1 to 191. Google udm accepts 2, 7, 12, 14, or 28, and filter accepts 0. Google num and Bing count must be from 1 to 30. Defaults select the first page with at most ten parsed results.
The optional legacy zone field accepts 1 to 128 ASCII letters, numbers, underscores, or hyphens and is ignored. It does not select a vendor zone or proxy. format: "raw" selects HTML unless data_format: "parsed_light" or the URL parameter brd_json=1 requests parsed JSON. parsed_light is the only supported data_format; brd_json accepts only 1. Explicit format: "html" cannot accompany either parsed-JSON selector.
For example, this supported migration request returns parsed organic JSON:
{
"zone": "former_serp_zone",
"url": "https://www.google.com/search?q=lunar+science&hl=en&gl=us",
"format": "raw",
"data_format": "parsed_light"
}This compatibility surface is a migration aid, not full compatibility with every vendor request or output field. Bright Data documents its broader /request shape and parsed response in its SERP API reference. Clients that use additional fields or depend on ads, maps, shopping, news, images, knowledge panels, or vendor-specific ranks must adapt to Vitna's organic-only contract.
Google's Custom Search JSON API is closed to new customers and is scheduled to end on January 1, 2027. Existing customers need a migration plan; it is not an available new-customer provider for this replacement. See Google's Custom Search JSON API notice.
Parsed response
An illustrative JSON response:
{
"general": {
"search_engine": "google",
"query": "lunar science",
"page_title": "lunar science - Google Search",
"timestamp": "2026-10-04T12:00:00.000Z",
"requested_country": "us",
"requested_language": "en",
"search_type": "text",
"mobile": false
},
"input": {
"original_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5",
"final_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5"
},
"organic": [
{
"rank": 1,
"title": "An example lunar science page",
"link": "https://example.com/lunar-science",
"description": "An example excerpt from a search result.",
"display_link": "example.com"
}
],
"meta": {
"provider": "vitna_serp",
"engine": "google",
"requestId": "example-request",
"latencyMs": 500,
"transport": "direct",
"coverage": "organic_only",
"parserVersion": "1",
"provenance": "search_engine",
"offset": 0,
"page": 1,
"count": 1,
"retrieval": {
"route": "direct",
"attempts": [
{ "route": "direct", "transport": "direct", "outcome": "success", "latencyMs": 500 }
],
"cache": { "hit": false, "ageMs": 0 }
}
}
}organic follows the recognized organic listing order on the fetched page. rank is the entry's position among recognized organic listings on that page. Rejected unsafe URLs and removed doorway pages can leave rank gaps, and separate listings that link to the same URL retain their observed positions. Vitna does not invent a whole-page global_rank, total result count, publication date, or verified location. description is search-engine snippet text. It is not a quote retrieved from the linked page. Missing descriptions and display links are omitted. display_link selects an observed publisher hostname or breadcrumb. Where Google's citation for a result holds only counts and an age, as on forum, video and community results ("20+ comments · 4 months ago"), it is the source line Google shows above the title instead, such as Reddit · r/buildapc or YouTube · Valve. Dates, ages and view or comment counts are never display links. A listing can include observed sitelinks; they are associated with the parent result and do not receive invented organic ranks.
timestamp is the transport's fetched-at time when available, otherwise the response assembly time. page_title comes from the fetched page or is null when unavailable. requested_country and requested_language appear only when requested. input.original_url is the normalized outbound search URL, including generated pagination and result-limit parameters. It is not a verbatim echo of a supplied URL. input.final_url is the observed URL after retrieval and permitted redirects. The observed final URL must retain the requested engine, query, and offset. A changed query or offset returns an explicit serp_query_mismatch error. meta.transport reports the transport that captured the result; meta.parserVersion identifies the parser implementation. meta.coverage: "organic_only" remains present even when the page contains other features. meta.offset records the zero-based requested offset. meta.page appears only when that offset is a multiple of ten and identifies its ten-result window. meta.count is the returned organic entry count. meta.doorwaysRemoved appears when Vitna left doorway pages out, and counts them.
When the page contains a recognized forward navigation control, pagination.next_page_link carries a normalized continuation URL derived from that control. It preserves the observed Google start or Bing first offset, validates supported search fields, and removes engine tracking parameters. If the control omits num or count, the continuation defaults to at most ten parsed results. pagination.current_page appears only when the requested offset maps to a ten-result window. Vitna does not invent a next-page URL when no control is observed. Submit that continuation as url to POST /v1/serp or POST /request. A structured client can instead use the observed Google start as offset, or Bing first minus one.
A recognized no-results page may return an empty organic list. A challenge, consent interstitial, unknown page layout, or invalid upstream response is an explicit failure. It is never represented as a successful empty search.
A page with accepted organic results, or a recognized no-results page, is a results page. Query text that the engine echoes in its title, suggestions, related searches, or no-results notice does not make it a challenge. Challenge elements, consent or sign-in forms, and consent, sign-in, or /sorry landings still fail as blocks. Google's toolbar zero count is read in any interface language, and its no-results notice is recognized in English, German, and French. Bing's no-results notice is recognized in English only.
Google descriptions support known snippet classes and the data-sncf or data-content-feature="1" markers inside recognized organic wrappers. These layout checks improve extraction after retrieval; they do not establish that Google will accept an outbound route.
For format: "html", the successful response body is the fetched HTML instead of the parsed envelope, with Content-Type: text/html and an X-Request-Id header. It uses the same page recognition and challenge checks as JSON output. Treat HTML, titles, and descriptions as untrusted external content. Escape text in a UI and keep raw HTML out of an application document's trusted execution context.
Google's notices
search_information appears when Google's page carries one of two notices about its own result list, and is left out otherwise. Neither notice changes organic.
"search_information": {
"omitted_results": { "shown": 1, "included": true, "before": 1 },
"quotes_dropped": { "no_results_for": "\"acme widget\" site:example.com", "results_for": "acme widget site:example.com" }
}omitted_results is Google's note under the results that it "omitted some entries very similar to the 1 already displayed". It is recognized by its filter=0 link, so it reads alike in any interface language. shown is the count Google states, or null when the notice holds no single number. Without includeOmitted the notice is reported and nothing more is fetched. With includeOmitted: true, Vitna fetches the page again with Google's filter=0 inside the same request's time and answers with it when it holds more results, doorway pages aside: included: true, before the first page's count, and an input.final_url that carries filter=0 while input.original_url stays the first request. Otherwise included: false and reason says why: no_more_results when the unfiltered page held no more, serp_timeout when the repeat ran out of time (it runs for at most 17 seconds, ends 2 seconds before the attempt's deadline, and is skipped when that would leave it under 3 seconds), or the repeat's own error code, such as serp_blocked. The first page is answered in each of these cases, at the latest when the repeat's time is up; only the caller's cancellation or the attempt's own deadline ends the search. The repeat is a second engine capture within one request, and its answer is cached apart from a request without includeOmitted. Google showed this notice on 3 of 30 one-result site: searches measured on 2026-10-17.
quotes_dropped is Google's note above the results that a quoted query found nothing, so it searched the same words without their quotes: no_results_for is the query as Google echoed it and results_for the query it ran. The organic results are for results_for, not for the exact phrase. It is read on English result pages only.
Doorway pages
Some sites, most of them hacked small-business sites, publish a page for each query searched, titled with the query's own words in lowercase at an address that carries none of them. Google ranks them for long, specific queries, the kind research agents write. Vitna leaves an entry out of organic when its title has four or more words, every letter a lowercase English letter, and no word of four letters or more that its link repeats. A PDF, or a page on a www. host, is always kept. Entries are removed before maxResults applies, the rest keep the engine's rank, and meta.doorwaysRemoved counts the removed entries. Multi-page jobs leave them out too. keepDoorways: true answers the page as the engine listed it, and that answer is cached apart.
On 6,575 Google results for 743 searches a research agent ran on 2026-10-18, the rule removed 1,079 (16%). One was a real page, a journal article with a lowercase title. Each of the 33 searches whose results carried the answer the agent was after still had it in a result kept.
SerpApi-shaped response
Code written against SerpApi's Google results can ask for SerpApi's field names with shape: "serpapi", on POST /v1/serp or POST /request, with structured fields or a search URL:
curl http://127.0.0.1:8789/v1/serp \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"query":"lunar science","country":"us","language":"en","maxResults":5,"extras":true,"shape":"serpapi"}'This shape is built from the same parsed result as the default one, after the cache and any shared capture, so the two shapes share cached answers and captures in progress, and authentication, quota, the deadline and fresh work the same for both. It renames and regroups fields and adds no value the parser did not observe:
{
"search_metadata": {
"id": "6b0d4f0e-7d1c-4c55-9a43-2f1b1c3f9e01",
"status": "Success",
"processed_at": "2026-10-13 09:41:07 UTC",
"total_time_taken": 3.19,
"google_url": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=0&num=5"
},
"search_parameters": {
"engine": "google",
"q": "lunar science",
"google_domain": "google.com",
"gl": "us",
"hl": "en",
"start": "0",
"num": "5",
"device": "desktop"
},
"related_questions": [{ "question": "What is lunar science?" }],
"organic_results": [
{
"position": 1,
"title": "An example lunar science page",
"link": "https://example.com/lunar-science",
"displayed_link": "https://example.com › lunar-science",
"snippet": "An example excerpt from a search result.",
"sitelinks": { "inline": [{ "title": "Missions", "link": "https://example.com/lunar-science/missions" }] }
}
],
"related_searches": [{ "query": "lunar science missions" }],
"pagination": {
"current": 1,
"next": "https://www.google.com/search?q=lunar+science&gl=us&hl=en&start=10&num=10"
},
"meta": {
"provider": "vitna_serp",
"engine": "google",
"requestId": "6b0d4f0e-7d1c-4c55-9a43-2f1b1c3f9e01",
"latencyMs": 3187,
"transport": "browser",
"coverage": "full",
"parserVersion": "1",
"provenance": "search_engine",
"offset": 0,
"page": 1,
"count": 1
}
}| SerpApi field | From | Notes |
|---|---|---|
search_metadata.id | meta.requestId | This request's identifier; a cache hit has its own |
search_metadata.status | Success, or Error for a search that failed after admission | |
search_metadata.processed_at | general.timestamp | When the page was captured, written YYYY-MM-DD HH:MM:SS UTC; a cache hit keeps its capture's time |
search_metadata.total_time_taken | meta.latencyMs | Seconds, to two places |
search_metadata.google_url | input.final_url | bing_url instead when Bing answered |
search_parameters | The request | engine, q, google_domain (always google.com), gl, hl, start, num, device (always desktop) and udm when sent, as strings; a Bing request gives cc, setlang, first and count |
organic_results[].position | rank | The position on the page; a rejected listing leaves a gap |
organic_results[].title, .link | title, link | |
organic_results[].displayed_link | display_link | The breadcrumb, or on a forum, video or community result the source line (Reddit · r/buildapc); omitted when the page shows neither |
organic_results[].snippet | description | Omitted when the page shows none |
organic_results[].sitelinks.inline | sitelinks | title and link each; the parser reads no sitelink snippets and does not tell Google's expanded block from its inline row |
related_questions[].question | people_also_ask | With extras: true; the question only |
related_searches[].query | related_searches | With extras: true |
ads[] | ads | With extras: true: position, title, link, displayed_link, and description from the ad's snippet |
pagination.current, .next | pagination.current_page, .next_page_link | current only for an offset on a ten-result page |
meta | meta | Vitna's own, unchanged |
SerpApi fields Vitna does not parse are left out rather than sent empty: an organic result's date, source, favicon, thumbnail, snippet_highlighted_words, redirect_link and rich_snippet; the answers to People Also Ask and the links of related searches; an ad's tracking_link, block_position and sitelinks; search_information, which carries result counts (Vitna's own search_information, Google's notices, is not mapped into this shape); pagination.other_pages and serpapi_pagination; and page features such as the AI Overview, perspectives, the knowledge graph, the answer box, local results, top stories and images. A recognized page with no results has an empty organic_results and status Success, where SerpApi adds an error string.
Where the operator sets VITNA_SEARCH_SERP_FALLBACK_ENGINE=bing, a Google search that Google does not answer can be answered by Bing. Both shapes say so in meta.engine, which is then bing. In this shape search_metadata gives bing_url in place of google_url, while search_parameters still records the Google request.
A search that fails after admission keeps its HTTP status, Retry-After and Vitna's error, and adds search_metadata and search_parameters:
{
"search_metadata": { "id": "c2e5b8a1-4f3d-4e6b-9a7c-1d2e3f4a5b6c", "status": "Error", "total_time_taken": 0.01 },
"search_parameters": { "engine": "google", "q": "lunar science", "google_domain": "google.com", "start": "0", "num": "10", "device": "desktop" },
"error": { "code": "serp_routes_busy", "message": "all configured routes for this search engine are in use" },
"meta": { "requestId": "c2e5b8a1-4f3d-4e6b-9a7c-1d2e3f4a5b6c", "engine": "google", "latencyMs": 8, "retrieval": { "attempts": [] } }
}error stays an object with code and message rather than SerpApi's string, because the code says whether and when to retry. A request refused before a search begins, for invalid input, authentication or quota, keeps the shared error envelope in either shape.
The Web API's MCP server offers this route to agents as the google_search tool, with SerpApi's argument names; see MCP.
Measured delivery
Three runs of live Google searches on the production search host, after its Google retrieval moved to residential exits alone on the evening of 2026-11-03 (Eastern time). Each run was a separate process on that host running the deployed SERP code as the service's own user, with the service's configuration and browser daemon: engine: "google", maxResults unset (so 5), fresh: true to skip the JSON cache, and one call a query. The first two kept three searches in flight and the third twelve. The queries were technical questions (configuration keys, flags, command lines and named techniques), and none appears in more than one run. A search counts as answered when it returned without an error. Every search, with its query, result count and time, is in serp-runs-2026-11.json.
| Finished (ET) | In flight | Searches | Answered | 95% CI | All five results | Median | p90 | p95 | Slowest |
|---|---|---|---|---|---|---|---|---|---|
| 2026-11-03 21:57 | 3 | 100 | 97 | 91.5 to 99.0% | 89 | 4.01 s | 4.43 s | 9.30 s | 20.44 s |
| 2026-11-04 00:23 | 3 | 200 | 199 | 97.2 to 99.9% | 186 | 3.89 s | 4.23 s | 4.87 s | 20.86 s |
| 2026-11-04 15:42 | 12 | 2,500 | 2,273 | 89.7 to 92.0% | 2,256 | 4.19 s | 9.66 s | 15.96 s | 22.37 s |
Times are over every search, refusals included, from the call to the parsed result on the search host. They leave out a caller's network round trip, the HTTP layer, authentication and quotas, so a request from elsewhere takes longer. The intervals are Wilson's; across the three runs 2,569 of 2,800 answered (90.7 to 92.7%), and across the two at three in flight 296 of 300 (96.6 to 99.5%). The first run followed 16 warm-up searches sent two at a time straight after the change and restart, of which 11 answered; they are not counted.
Every refusal was returned as the 503 the failure table lists rather than as an empty organic list. In the runs at three in flight each was serp_blocked (8.99 to 17.97 s); in the 200-search run it was the first search sent. In the run at twelve, 222 were serp_blocked (6.87 to 21.75 s) and 5 serp_timeout (18.78 to 22.37 s), and a refusal took a median of 14.89 s against 4.16 s for an answer. They were not spread evenly: 184 of the first 250 searches sent were answered, and 2,089 of the 2,250 after them. The run did not record why.
An answer can carry fewer than five results. In the 200-search run, 10 answers lost one or two to doorway pages (11 pages left out), and three carried 3, 1 and 0 results for a reason the run did not record. In the 100-search run, all 8 short answers were doorway removals. In the 2,500-search run, 16 answers lost one each to doorway pages and one carried 4 for a reason the run did not record. The runs recorded no attempt counts, so how many answers needed a second try is not known; 94 of 100, 193 of 200 and 2,209 of 2,500 finished under 7 s.
Five is the default because Google unlocks a page's /goto result links in turn and a page waits for its slowest link: on 2026-11-03, links 1 and 2 answered at once, links 3 to 5 within about 2.3 s of the page settling, link 6 after about 7.6 s and links 7 to 10 after about 24 s (see Run on owned infrastructure). Earlier 100-search runs that week at maxResults: 10 took 26.1 s at the median.
These are three runs on one host within a day, on one kind of query. They describe those runs, not coverage of the wider web or a guarantee of availability, and Bing was not measured. The runs called the retrieval in the host's own process, so per-key quotas did not apply: twelve in flight is what the host sustained, not what one key may send.
Retrieval routes and retries
The operator chooses auto, direct, or browser transport. The following describes ordinary routes; the experimental owned adapter below replaces Google's routes when configured. With auto and no configured proxies, Vitna prefers a JavaScript browser for Google and direct HTTPS for Bing, with bounded fallback to the other available transport. With configured proxies, auto uses only browser routes pinned to those proxies. Explicit browser uses a browser route per proxy, or a local browser route when no proxies are configured. Configured proxy routes never fall back to local egress. Explicit direct uses direct HTTPS and refuses proxy configuration. Proxy pools rotate their starting route across requests while respecting route health and capacity.
SERP browser captures allow ordinary GET images and fonts through the same public-address guard and request, byte, and deadline limits. This internal option does not allow arbitrary POSTs, media, or WebSockets. The requested Google hl or Bing setlang language is validated and forwarded to the browser locale; without a language hint the browser keeps its English default. These settings do not establish that Google will accept an egress route.
Pinning a proxy URL keeps its endpoint and effective credentials consistent throughout one browser capture. Structured proxy entries can generate a new provider session credential for each admitted attempt. For browser captures, the provider must keep the same exit for that whole context, including redirects and subresource connections. A gateway that rotates on each connection can change exits within one capture even though its configured URL stays the same. Vitna does not verify the provider's exit IP or session stickiness. Route health and pacing track declared egress groups, not measured public exit IPs.
The browser transport expects an HTTP(S) CONNECT tunnel to the checked destination address and retains target TLS certificate verification. Managed scraping proxies that intercept target TLS or return pages fetched by their own browser need a separate integration. For example, Crawlbase's Smart AI Proxy documentation states that it intercepts TLS and asks clients to disable verification. Vitna does not automatically disable verification for these endpoints.
The default request tries at most three distinct routes, bounded by the available routes and the shared whole-request deadline. Each attempt has a separate four-second default cap within the remaining deadline. Only blocks, unreachable engines, attempt timeouts, and unrecognized pages permit another route attempt. Requests keep the selected engine and query throughout. Vitna returns the first validated result page; it does not combine rankings from several routes or change Google requests to Bing requests.
Route concurrency and health are tracked in the current service process. The experimental owned Google adapter below adds a persistent admission gate. A route allows one active request per engine by default. Egress groups also allow one active request per engine by default; direct HTTPS and the local browser share that group. The first actual upstream block cools matching routes for that engine/egress group for five minutes by default. A redirect to the engine's own consent or sign-in page counts as a block. A browser challenge flag counts only with HTTP 403 or 429; otherwise the attempt fails as serp_invalid_response without a cooldown or another attempt. Repeated blocks double that cooldown, up to a one-hour default cap. A valid upstream capture on the same group resets its block streak; denied requests and cache hits neither reset nor escalate it. Other transient failures use a thirty-second default cooldown. A Google cooldown does not block Bing. If an uncached request cannot begin an attempt because all eligible routes are occupied or cooling, the service returns HTTP 503 with Retry-After. When attempts have already failed, their final retrieval failure is returned. Restarting the process resets ordinary route health. Ordinary service replicas do not share these pacing, concurrency, or cooldown limits. If replicas use the same physical exit, coordinate retrieval starts across them or run one retrieval process for that exit; adding replicas otherwise multiplies its outbound traffic.
Independent retrieval starts are paced per engine and egress group. The default interval is selected uniformly between five and fifteen seconds, averaging roughly six starts per minute for each engine/egress group. Direct HTTPS and the local browser share both pacing and the egress concurrency slot. Fresh requests follow this pacing; cache hits can return without starting a new retrieval. Internal fallback within one logical request can reuse an egress only after its previous upstream attempt and cleanup have settled, within the same request deadline.
The API does not queue requests until their next start time. If every otherwise eligible route is paced, it returns HTTP 503 serp_routes_throttled with Retry-After in seconds and meta.retrieval.attempts: []. Clients can retry after that interval. Busy routes take precedence. Otherwise, when waiting routes differ, the code and Retry-After describe the route that can start soonest, whether it is paced, cooling, or awaiting its runtime retry; equal start times report pacing. A paced route that is due sooner is never reported as another egress's longer cooldown. The shared thirty-requests-per-minute client limit caps admission; it does not promise that many upstream starts. Ordinary route pacing and egress slots are process-local; horizontally scaled instances do not coordinate them.
Browser dependencies load lazily when a browser attempt is needed. Starting the service or reading its status does not launch a browser or probe a search engine. Missing browser dependencies can therefore produce an explicit runtime-unavailable response on a browser attempt. That response pauses the affected route, for every engine it serves, for sixty seconds. Requests in that window return HTTP 503 serp_runtime_unavailable with Retry-After and no new attempts. The first request after it makes one fresh attempt, so a slow browser start does not disable the route until restart. A request never retries the same route. A missing dependency keeps failing; repair the runtime setup and restart before verifying that route again.
JSON cache and freshness
Successful parsed JSON responses can be reused by the same client for thirty seconds by default, and for up to one hour when the operator sets VITNA_SEARCH_SERP_CACHE_TTL_MS. The cache is process-local and bounded to 128 entries and 8 MiB of serialized responses. Entries are kept per API client, including pages its multi-page jobs capture, so a cache hit, its age, and its capture timestamp never reveal another client's search. HTML and failed retrievals are not cached. Responses still carry HTTP Cache-Control: no-store; the short-lived cache belongs to the service, not an intermediary or the caller's browser. When a larger maxResults capture for the same client and query parameters is already cached, smaller maxResults requests slice from that superset entry (cacheEvent: "superset"). Operators can also configure stale-while-revalidate (VITNA_SEARCH_SERP_CACHE_SWR_MS), stale-if-error (VITNA_SEARCH_SERP_CACHE_STALE_IF_ERROR_MS), and an optional WAL-mode SQLite L2 store (VITNA_SEARCH_SERP_CACHE_SQLITE_PATH) so warm JSON entries survive process restarts. Live delivery and latency gates can be evaluated against a running host with search:serp:benchmark:
npm run search:serp:benchmark -- --url https://search.vitna.ai --freshSet fresh: true to bypass a cached response. A successful new capture refreshes that cache entry. Cached results retain their original engine capture timestamp; a cache hit does not imply the search engine was contacted for that request. Cached requests still use the shared API authentication, quota, and concurrency controls. Restarting the process clears the cache. Its transient in-memory query/result data is separate from receipts, which omit queries and result content.
When the cache is full, its least recently used entry leaves first, even before its TTL ends. Captures made with fresh: true are stored too, so a load test that searches fresh can push other clients' entries out. The status cache.evictions count shows how often that happens. Operator metrics count each answered request by where its page came from (capture, cache, or shared for a joined capture, below), split by test and real traffic in vitna_api_serp_answer_sources_total. Together they show how often searches repeat within the TTL before a longer one or a durable store is worth having.
Matching in-flight JSON requests from the same client can share one capture in the current process. The canonical engine, trimmed query, effective offset, country, language, result limit, JSON format, and fresh flag must match. Fresh requests share only with matching fresh requests. Another client's identical request never joins; it needs its own route capacity like any other request. Each caller still uses its own API authentication, quota slot, request deadline, cancellation, request identifier, and latency. Cancelling one caller leaves the capture available to the others; when the last caller leaves, the producer is aborted. A shared capture runs on the route's whole timeout so later callers can join it. A route that rotates its own exits spends every second it is handed on more tries, so it gets no more than the latest waiting caller still wants. A lone Google search beside a Bing fallback hands the owned route 49 of the 55 s, so its tries come back before it leaves. Until 2026-10-12 the route got all 55, and a search that ran to its deadline lost its tries when its abandoned capture was aborted. HTML captures are separate, and failures never become cached responses.
The sharing pool is bounded to 128 active captures and 32 pending callers per shared capture, including its first caller. An abandoned producer keeps its capacity reservation until it settles. Capacity refusals use HTTP 503 serp_routes_busy with Retry-After: 1 and no new attempts.
meta.retrieval.route identifies the route that captured the returned page. meta.retrieval.attempts lists this request's attempts with route, transport, outcome: "success" or "failure", latencyMs, and an error code for failures. A cache hit has no new attempts. The sibling cache object reports hit and the cached entry's ageMs; a new capture reports hit: false, ageMs: 0. Route identifiers such as direct, browser-direct, and proxy-1 are safe labels, not proxy addresses. Joined callers receive meta.retrieval.coalesced: true with attempts: []; the first caller receives coalesced: false.
Authenticated GET /v1/status exposes serp.retrieval with configured attempt limits, cooldowns, cache counts, bytes and evictions, minIntervalMs, maxIntervalMs, maxConcurrentPerEgress, maxBlockedCooldownMs, runtimeRetryMs, and the safe engine transport preference map preferredTransports. Its coalescing object reports active, waiters, maxEntries, maxWaiters, and hits, without queries or key hashes. Its routes list includes safe route identifiers, transports, and per-engine state. Each engine state reports ready, busy, cooldown, throttled, or unavailable, active route count inFlight, shared egress count egressInFlight, attempt/success/failure counters, consecutiveBlocks, cooldownUntil, unavailableUntil, nextRequestAt, and lastSuccessAt. The latter timestamps use Unix milliseconds, with no success recorded as null. These are process-local operational observations; a ready route has capacity and is neither cooling nor paced, without an upstream availability probe.
Multi-page Google jobs
Use an asynchronous job to request up to 100 distinct organic Google links across at most ten observed result pages. The existing POST /v1/serp and POST /request remain single-page requests with maxResults at most 10. Multi-page jobs return parsed JSON and support Google only.
curl http://127.0.0.1:8789/v1/serp/jobs \
-H "Authorization: Bearer $VITNA_SEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"engine":"google","query":"lunar science","maxResults":100,"maxPages":10,"timeoutMs":300000}'| Field | Default | Allowed values |
|---|---|---|
engine | google | google only |
query | Required | Same validated query string as single-page SERP retrieval |
country, language | Unset | Same localization hints as single-page requests |
maxResults | 100 | Integer 1 to 100; target distinct result count |
maxPages | 10 | Integer 1 to 10; maximum captured result pages |
timeoutMs | 300,000 | Integer 1,000 to 300,000; whole-job deadline, including queue time |
fresh | false | Boolean; true bypasses completed-job reuse and requires a new collection unless an identical fresh job is already active |
New collection jobs begin at offset zero and make fresh, sequential, isolated captures. They follow the recognized pagination.next_page_link from each fetched page, rather than asking Google for num=100 or inventing offsets. A page with fewer than ten results can still have an observed continuation. Each attempt has a deadline of at most sixty seconds within the remaining job time and the instance's configured shared and SERP deadlines. Blocks and cooling stop the job instead of switching engine or bypassing the connection's penalty. Busy or paced routes use the exact private millisecond wait when available, subtracting cleanup time already spent; otherwise they use the public Retry-After seconds. The durable admission gate is rechecked before capture. Waits stay within the job deadline, with at most twenty tries for a logical page. The private timing hint is not an API field. A page refused admission because every service slot is in use (service_busy) or the per-minute limit is reached (rate_limited) also waits its Retry-After within the job deadline, with at most twenty admission tries. A daily limit or unavailable request accounting stops the job.
Creation returns HTTP 202 with a job envelope, a Location status path, and Retry-After: 1. Its id identifies the job; a new job begins queued, while a reused job retains its existing status. progress reports pagesCompleted, returnedResults, requestedResults, and maxPages. result appears when captured results are available, and an optional error contains a safe code. Jobs expose creation, start, finish, and expiry timestamps when applicable. An admitted job becomes running when its worker starts.
With fresh: false, an exact same-client request can reuse a successful completed job for thirty seconds after its finish time. Only jobs that reached their requested target without an error qualify; partial, failed, cancelled, and interrupted results are not reused. Engine, trimmed query, localization hints, result target, page limit, and job deadline must match. The original job ID, results, and capture timestamps remain intact.
Identical same-client queued or running requests with the same fresh flag join that one job. This is one shared cancellable job: cancelling its ID stops it for every same-client caller using it. fresh: true bypasses a completed result and joins only an identical active fresh job. Reuse does not start more page captures or consume more page quota, but each POST still uses the job-control allowance.
The creation response adds job.reuse with kind: "none", "active", or "completed", requestedFresh, ageMs, and ttlMs. Read responses omit that request-specific reuse marker. Stable job.freshness records the original job's requestedFresh policy and latest page fetchedAt, when available; reused results retain those capture times. Use the same client bearer authentication to read or cancel that job:
GET /v1/serp/jobs/{id}
DELETE /v1/serp/jobs/{id}Creating and reading jobs share a separate limit of sixty requests per minute per client. Cancellation has its own ten-per-minute allowance so exhausted polling does not prevent cancellation. These controls do not consume query quota. Reads return HTTP 200 and cancellation returns HTTP 202, both with {job}. Unknown, expired, evicted, or another client's jobs return HTTP 404 job_not_found. Each admitted logical page consumes one shared SERP quota slot, including bounded route waits, with its concurrency slot held through cleanup. Job reads and cancellation remain available while page slots are occupied. Requesting 100 links does not purchase or guarantee 100 results. Cancellation preserves captured partial results. A running job can report cancelling while its worker remains occupied until the actual browser and network cleanup settles.
The aggregate result contains general, organic, pages, and meta. Duplicate detection strips URL fragments. For exact /watch URLs on youtube.com, www.youtube.com, or m.youtube.com with one valid 11-character v value and no nondefault port, it also ignores xstg. Other query parameters, their encoding and ordering, and host aliases remain distinct. The first observed entry wins with its original link and discovery order; recovery preserves saved legacy results. Single-page SERP output preserves its observed listing rows. Each organic entry retains its page-local rank and adds position for the aggregate discovery order. Its source records the sequential capture page, actual observed offset, original pageRank, fetchedAt, and serpUrl. That aggregate position is not an invented Google global rank. pages retains each capture's URL, time, result counts, and safe metadata. Aggregate meta.timingMs separates observed admission, retrieval, pacingWait, and checkpoint time. Admission includes waits after a busy or per-minute refusal. Retrieval includes backend capture and physical cleanup; these are measured phases, not search-quality scores. Compare a fresh job separately from completed-result reuse. Google listings and snippets are a different output from native index matches or the research API's verbatim cited passages. An owned-route provider fee of zero excludes infrastructure, browser, network, storage, crawling, and operating costs.
Only reaching the requested distinct-result target produces completed, meta.complete: true, and meta.stopReason: "target_reached". Fewer results produce partial when usable links are retained, including no_next_page, page_limit, or a retrieval error; no usable results finish failed. Other stop reasons include blocked, cooling, deadline when the whole-job deadline ends collection, timeout when a page capture times out with job time remaining, both with error code serp_timeout, and cancelled. Failed, cancelled, or interrupted jobs retain usable partial results when available. The limits and observed continuation do not guarantee that Google will deliver 100 links.
One authenticated live job on 2026-10-05 at 12:37:07 UTC was accepted with HTTP 202 and requested 100 links across at most ten pages. It returned partial with 94 unique publisher URLs in 107,047 ms, ending with meta.stopReason: "page_limit" and meta.complete: false. The ten captured offsets were 0 through 90 in steps of ten. Their counts were 7, 10, 10, 10, 10, 10, 10, 9, 10, and 10: 96 raw results with two duplicate links removed. All captures used owned-google; result source references matched their captured pages, with no reported error or challenge. The local stock browser used the owned OVHcloud exit under the same durable admission gate.
This verifies one multi-page API run. The new multi-page console controls were not part of that live check; an authored real-Edge fixture separately verified creation, sequential polling, result cards, and page provenance. It does not establish sustained production reliability, geographic targeting, or guaranteed delivery of 100 links.
The manager allows one active worker, eight queued jobs, at most 64 retained records, and two nonterminal jobs per client. Records expire one hour after creation. Under load, finished results can be evicted sooner: when all 64 records are in use, a new job removes a finished one, the submitting client's oldest first, otherwise the oldest of any client. Queued and running jobs are never evicted. Service startup defaults to memory. Opt into private JSON snapshots with VITNA_SEARCH_SERP_JOBS_DIR=.search-data/serp-jobs locally, or /var/lib/vitna-search/serp-jobs on a Linux service host. These snapshots contain queries and results; protect their directory as private operator data. An exclusive writer lease permits only one service per jobs directory. Its holder refreshes the lease every thirty seconds. A restart reclaims a same-host lease whose process is gone, whose PID now names a later process on Linux, or whose PID is the restarting process itself, as PID 1 finds after a container restart. A lease left unrefreshed for two minutes is stale from any host, so a replacement container can start once its predecessor's lease is that old. A service that finds its lease replaced or removed stops admitting jobs and writing snapshots. A restart marks recovered nonterminal jobs interrupted, preserving saved partial results without automatically resuming retrieval. Recovery deletes an expired snapshot once its ID and expiry are readable, without validating the rest, and accepts timestamps that a wall-clock step or host sleep left out of order. Authenticated status exposes these availability, persistence, queue, record, and limit counts under serp.collection.
Run without an index
SERP retrieval does not require a SQLite search index or a crawler worker. Mint client credentials as described in the search overview, then enable the SERP capability in the service's private environment:
VITNA_SEARCH_CLIENTS={"pilot":"REPLACE_WITH_CLIENT_KEY_SHA256"}
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=auto
VITNA_SEARCH_SERP_TIMEOUT_MS=10000Enabling retrieval does not authorize redistribution. VITNA_SERP_RESALE_APPROVED=true records the operator's decision that collecting Google and Bing results and redistributing them through Vitna's API, including fetched HTML, is permitted. Keep both settings unset until that decision is made. Startup and the live checker refuse VITNA_SEARCH_SERP_ENABLED=true without this confirmation, rather than running with SERP retrieval silently off.
Start the service:
node --env-file=.env.search platform/search/start.mjsThe direct transport uses the built-in Node runtime and makes bounded HTTPS requests to supported search engines. It requires no Bright Data API key, zone, search adapter, or provider account. Starting the process makes no search-engine probe.
| Environment variable | Default | Meaning |
|---|---|---|
VITNA_SEARCH_SERP_ENABLED | false | Enable independent Google/Bing SERP retrieval; also requires VITNA_SERP_RESALE_APPROVED=true |
VITNA_SERP_RESALE_APPROVED | false | Operator confirmation that collecting Google/Bing results and redistributing them through the API is permitted; true or false |
VITNA_SEARCH_SERP_TRANSPORT | auto | Bounded direct/browser route selection; explicit direct or browser also supported |
VITNA_SEARCH_SERP_TIMEOUT_MS | 10,000 | Integer 100 to 60,000; retrieval deadline also bounded by the shared request deadline |
VITNA_SEARCH_SERP_PROXIES | Unset | Operator JSON array of at most 20 HTTP(S) proxy URL strings or structured route entries; requires auto or browser transport |
VITNA_SEARCH_SERP_ATTEMPTS | 3 | At most 1 to 5 distinct attempts per request, within one deadline |
VITNA_SEARCH_SERP_ATTEMPT_TIMEOUT_MS | 4,000 | Integer 100 to 60,000; per-attempt cap within the remaining deadline |
VITNA_SEARCH_SERP_ROUTE_CONCURRENCY | 1 | Integer 1 to 4; concurrent requests per route/engine pair |
VITNA_SEARCH_SERP_EGRESS_CONCURRENCY | 1 | Integer 1 to 4; concurrent requests per engine/egress group; shared by direct HTTPS and local browser |
VITNA_SEARCH_SERP_MIN_INTERVAL_MS | 5,000 | Integer 0 to 60,000; minimum jittered interval between independent starts |
VITNA_SEARCH_SERP_MAX_INTERVAL_MS | 15,000 | Integer 0 to 60,000; maximum interval; must be at least the minimum |
VITNA_SEARCH_SERP_COOLDOWN_MS | 30,000 | Integer 1,000 to 300,000; transient-failure route cooldown |
VITNA_SEARCH_SERP_BLOCKED_COOLDOWN_MS | 300,000 | Integer 1,000 to 3,600,000; engine/egress cooldown after a block |
VITNA_SEARCH_SERP_MAX_BLOCKED_COOLDOWN_MS | 3,600,000 | Integer 1,000 to 3,600,000; repeated-block cooldown cap; must be at least the base blocked cooldown |
VITNA_SEARCH_SERP_CACHE_TTL_MS | 30,000 | Integer 0 to 3,600,000; successful JSON reuse window, at most one hour; 0 disables reuse |
VITNA_SEARCH_SERP_CACHE_ENTRIES | 128 | Integer 0 to 1,024; maximum cached responses, also bounded by 8 MiB; 0 disables caching |
VITNA_SEARCH_SERP_JOBS_DIR | Unset; in-memory jobs | Optional private JSON snapshot directory, such as .search-data/serp-jobs locally or /var/lib/vitna-search/serp-jobs on Linux; one service writer per directory |
VITNA_SEARCH_SERP_BROWSER_PROFILE | Inherited Unblocker profile, otherwise native | native or cloakbrowser; the latter requires a browser executable path |
VITNA_SEARCH_SERP_BROWSER_CHANNEL | Inherited Unblocker channel, otherwise bundled Chromium | Optional installed browser channel, such as msedge |
VITNA_SEARCH_SERP_BROWSER_PATH | Inherited Unblocker path, otherwise Playwright default | Optional operator-selected browser executable path |
VITNA_SEARCH_SERP_OWNED_SSH_HOST | Unset | Experimental Google route: operator-configured SSH host alias, such as owned-host; requires auto or browser |
VITNA_SEARCH_SERP_OWNED_EGRESS_ID | Unset | Required declared physical exit label, such as owned-exit; follows the proxy egress-label rules |
VITNA_SEARCH_SERP_OWNED_BROWSER | stock | Experimental owned browser: stock uses the local installed browser; camoufox uses a separately provisioned remote Linux worker |
VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT | Unset | Required for camoufox only: private remote installation root /tmp/vitna-firefox. followed by 8–32 letters, digits, underscores or hyphens |
VITNA_SEARCH_SERP_OWNED_STATE_PATH | .search-data/serp-browser-check-state.json | Private persistent gate shared with matching diagnostics; relative paths resolve from the service working directory |
VITNA_SEARCH_SERP_OWNED_PROFILE_ROOT | .search-data/browser-controls | Private root for fresh local stock-browser profiles; relative paths resolve from the service working directory; unused by camoufox |
VITNA_SEARCH_SERP_OWNED_MIN_INTERVAL_MS | 5,000 | Integer 1,000 to 60,000; minimum interval for the owned route's persistent admission gate |
VITNA_SEARCH_SERP_OWNED_MAX_INTERVAL_MS | 15,000 | Integer 1,000 to 60,000; maximum interval; must be at least the owned minimum |
VITNA_SEARCH_SERP_OWNED_ROTATING_EXITS | false | camoufox only: the remote daemon rotates, spaces and cools its own residential exits, so the route neither paces nor rests (its interval settings are unused) and a failed search records a strike only |
VITNA_SEARCH_SERP_OWNED_CONCURRENCY | 1 | Integer 1 to 32; captures the route runs at once; above 1 requires rotating exits. Slot N holds its lease in <state path>.slot-N |
VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON | false | camoufox only: send each search to the daemon's own socket (nc -U over the SSH host alias, or a local Unix socket when VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET is also set), instead of starting the private worker, which saves about 150 ms a search. Without VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET, needs a daemon running in the root and OpenBSD nc on the remote host. Route N sets its own _N_ value |
VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET | Unset | camoufox with VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON=true only: safe absolute .sock path (at most 100 characters, such as /run/vitna-camoufox/daemon.sock) when Search and the Camoufox daemon run on the same host, connecting over a local Unix domain socket with no ssh or nc process. When set, VITNA_SEARCH_SERP_OWNED_SSH_HOST and VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT are optional. Route N sets its own _N_ value |
VITNA_SEARCH_SERP_FALLBACK_ENGINE | Unset | bing only: a Google search the owned route does not answer, for any reason but a cancelled request (blocked, cooling, throttled, busy, unreachable, an invalid answer, or out of its time), goes to Bing with the rest of the deadline. Until 2026-10-12 only the first four and running out of time did, and over 4,500 load-test searches that night 43 failed outright without trying Bing: 25 on busy routes, 15 unreachable, 3 invalid answers. Owned Google then stops 6 s before the deadline (at most a quarter of it) so Bing can still answer, and a Bing route still in its pacing interval is waited for whenever its next turn leaves 1.5 s to answer, however many turns other searches take first; until 2026-10-18 a search that lost one turn to another failed, about 5% of the AA benchmark's searches at a 30 s Google share. A Bing route busy with another search's hand-off is polled every 100 ms until it frees, while 1.5 s would still be left; until 2026-10-13 the second of two searches handed over together failed at once as serp_routes_busy. On 2026-10-10/11, 41 of 500 long-tail searches spent the whole 55 s on Google and failed without trying Bing, while 10 Google deliveries came in the last 6 s; a free Bing route answered in about 0.4 s |
VITNA_SEARCH_SERP_CONTENTS | false | true lets a request ask for page contents, read live from each result page; false answers such a request 503 serp_contents_unavailable |
VITNA_SEARCH_SERP_EXCERPT_RERANK | false | true ranks excerpts by meaning as well as keyword. It needs the embedding settings and VITNA_SEARCH_EMBED_BATCH_URL, since the engine answers one request at a time and a search's query embedding would otherwise wait behind excerpt windows |
VITNA_SEARCH_SERP_GOOGLE_SHARE_MS | Unset | 1000 to 60000: the most of a search's time owned Google gets before timing out or handing off to Bing, never more than the deadline less any Bing reserve. Unset, Google gets all of that. The answered p90 is the searches Google answers late or never. By day on 2026-10-12, 102 of 457 Google answers came after 10 s, and the 8.6% Bing answered came at 37 to 49 s. Replayed with a 20 s share, the p90 fell from 32 to 21 s and Google's share from 91 to 84%; with 15 s, to 16 s and 81%. Overnight, when far more searches miss Google, a 20 s share cut Google from 66 to 49%. Bing's route takes a start every 5 to 15 s, so check it keeps up with what the share sends it |
VITNA_SEARCH_SERP_GOOGLE_SHORT_SHARE_MS | Unset | 1000 to 60000: overrides VITNA_SEARCH_SERP_GOOGLE_SHARE_MS when maxResults is 5 or fewer, never more than the deadline less any Bing reserve. Under --goto-respect-num 1 --goto-pace-ms 1000, maxResults <= 5 stays inside Google's 5-link zero-hold /goto regime (~3.8 to 4.1 s delivered p50) while maxResults = 6 hits a ~7.6 s 6th-link hold (~9.6 s delivered p50) and maxResults 7 to 10 waits out the 22 to 24 s /goto hold ceiling (~26 to 27 s delivered p50), so operators can set a ~20 s share for short requests and a ~42 s share for 6-to-10-result requests |
The SERP-specific browser profile, channel, and path override the corresponding VITNA_UNBLOCKER_BROWSER_PROFILE, VITNA_UNBLOCKER_BROWSER_CHANNEL, and VITNA_UNBLOCKER_BROWSER_PATH settings. Omitted SERP-specific settings inherit those operator values. The stock owned Google route instead requires an explicit absolute VITNA_SEARCH_SERP_BROWSER_PATH; its stock browser does not inherit a channel or Unblocker executable path. Camoufox requires its remote installation root and does not require a local browser executable. An owned route requires the SSH host, egress label, and the selected browser's configuration. OWNED_REMOTE_ROOT is refused with OWNED_BROWSER=stock.
The opt-in browser integration test uses authored local response fixtures and the installed runtime; it does not query search engines:
VITNA_SEARCH_SERP_TEST_BROWSER=1 node --test tests/search-serp-browser.test.mjsSet the Unblocker browser path, channel, and profile when using an installed browser instead of bundled Chromium.
The ordinary local browser transport reuses Vitna Unblocker's browser runtime and requires its Playwright dependency and Chromium installation. Install those in platform/unblocker as documented in the Unblocker overview. Proxy URLs contain no path beyond /, query, or fragment. Effective routes must be unique, as described below. Proxy configuration belongs in the private operator environment; URLs, credentials, and templates are not exposed through responses, authenticated status, receipts, or diagnostic reports. Browser retrieval does not run a CAPTCHA solver or wait for a challenge to be solved.
Browser requests keep the installed runtime's coherent browser headers and user agent. The service does not rotate random user-agent strings to disguise one browser as another. Direct retrieval sends a bounded Accept-Language matching the initial engine URL's hl or setlang, with en-US,en;q=0.9 when no hint is supplied. It requests Accept-Encoding: identity so its HTML size bound applies to the received body. Each browser attempt uses an isolated context and its configured pinned proxy, when present.
On Linux, the host or container runtime must support Chromium's sandbox and user namespaces. Run the browser service as an unprivileged user with the sandbox enabled. A sandbox launch failure requires fixing that runtime's sandbox support.
Run on owned infrastructure
Vitna can run the API, browser captures, organic parser, route scheduler, and CONNECT gateways on operator-controlled machines. This path requires no managed scraping service or proxy provider API. Run the owned egress gateway on each chosen machine, then register its authenticated endpoint in the search service's private environment:
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=browser
VITNA_SEARCH_SERP_PROXIES=[{"url":"https://serp:[email protected]:8792","egressId":"owned-cloud"}]The endpoint is a placeholder for an operator-owned listener with TLS. Use a separate gateway port when Search already occupies that machine's port 8791. Keep the gateway's credential hash on the gateway and its plaintext connection credential only in the private browser configuration.
The search release includes the gateway modules and the three deploy/vitna-egress templates. Adapt the service template's working directory, Node executable, and environment path to the host installation. Packaging these files does not install a service or publish a listener.
Declare the same egressId for listeners or machines sharing a physical exit. Adding containers or gateways on that exit adds capacity without establishing a new network route. Owning the stack does not establish that Google accepts its requests; validate each available route with the diagnostics below and retain its block cooldowns.
Experimental stock Edge over owned SSH
An opt-in Google adapter runs the installed desktop Edge browser through an operator-owned SSH exit with public IPv6 connectivity. Configure a noninteractive SSH alias privately, with host verification and TCP forwarding available. Browser-to-Google TLS retains target certificate verification. This SSH transport selects checked public IPv6 targets, with no IPv4 fallback. The desktop must remain online. A cloud API process cannot use this Windows browser without a connected worker on that desktop.
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=auto
VITNA_SEARCH_TIMEOUT_MS=30000
VITNA_SEARCH_SERP_TIMEOUT_MS=30000
VITNA_SEARCH_SERP_OWNED_BROWSER=stock
VITNA_SEARCH_SERP_BROWSER_PATH=C:/Program Files (x86)/Microsoft/Edge/Application/msedge.exe
VITNA_SEARCH_SERP_OWNED_SSH_HOST=owned-host
VITNA_SEARCH_SERP_OWNED_EGRESS_ID=owned-exit
VITNA_SEARCH_SERP_OWNED_STATE_PATH=.search-data/serp-browser-check-state.json
VITNA_SEARCH_SERP_OWNED_PROFILE_ROOT=.search-data/browser-controlsThese settings select owned-google exclusively for Google, with one attempt and no fallback to ordinary routes. Bing keeps its configured transport and proxies. Allow twenty to thirty seconds for both the shared request deadline and SERP deadline; the smaller deadline still bounds the capture. Existing route defaults remain unchanged until explicitly configured. Each capture uses a fresh profile and headed stock browser, without headless or browser automation flags. It submits the actual Google GET search form. Before accepting a capture, it checks that result rows and the observed next-page URL agree across two observations 150 milliseconds apart. Owned-route starts retain a random five-to-fifteen-second interval by default. Operators can set VITNA_SEARCH_SERP_OWNED_MIN_INTERVAL_MS and VITNA_SEARCH_SERP_OWNED_MAX_INTERVAL_MS within the bounds above. These settings change the same durable admission gate; they neither reset a cooldown nor change the declared egress identity.
On Windows, the stock launcher assigns the suspended browser to its owned Job Object before resuming it. Cleanup terminates that job and requires confirmation that its assigned process tree is empty before releasing the lease or deleting the profile. This containment applies to the Windows helper. Transient profile locks permit bounded removal retries, with the allocated directory, root identity, and real paths rechecked before every retry. A replaced directory is preserved; missing drain confirmation keeps the lease held and prevents runtime reuse. The renderer accepts delayed drain confirmation only within the original request deadline. Exact cleanup proof releases a temporary launcher fence; unverified physical cleanup keeps the fence in place.
The renderer resolves recognized opaque Google /goto links with bounded native-browser GETs through fresh subpages in the same context, using maxRedirects: 0. Native interception reads status and Location headers, then aborts every response, including HTTP 200, before following a redirect, downloading its body, or executing scripts. Validated publisher destinations are recorded for the parser without fetching them or rewriting the captured HTML. An unrecognized wrapper response is an explicit failure. At most two link-resolution workers share this capture's context, request budget, and deadline. Mappings retain the observed listing order, and failure or cancellation retains both workers' cleanup before releasing the lease. SERP pages themselves remain sequential. Non-navigation POSTs are restricted to the main frame at exactly /gen_204, /log, /httpservice/retry/enablejs, and /httpservice/retry/jserror on https://www.google.com: at most 32 per capture, with at most 64 KiB of reported body data per request. A response-stage gate aborts POST redirects before another hop. Public-address, request, byte, and deadline guards remain in force.
Native proxy authentication supplies the capture's private credential only for a Basic challenge from its exact loopback gateway with the Vitna realm. Server authentication and mismatched or repeated challenges are refused. Credentials stay out of general HTTP authentication and target headers. A gateway 407 is a runtime/setup failure, without a Google block penalty.
The dedicated Google capture admits at most 128 browser requests, while SSH remains capped at 80 tunnels. HTTP request interception does not prevent every speculative TLS preconnection. Those connections still pass the gateway's DNS, public-address, transfer-byte, and tunnel limits.
The shared state file reserves starts five to fifteen seconds apart and persists block cooldowns from five minutes up to one hour. Matching declared egress identities and state paths share this gate across API and diagnostic invocations; the exclusive lease remains held through actual cleanup, even after caller timeout. Keep the physical exit label stable. It is an operator declaration, not verification of the exit's public IP. Ordinary SERP pacing and cooldown environment settings do not reconfigure this durable gate. The owned route's process-local retrieval still follows the operator's cooldown and cache settings, as Bing's routes do: VITNA_SEARCH_SERP_COOLDOWN_MS, VITNA_SEARCH_SERP_BLOCKED_COOLDOWN_MS, VITNA_SEARCH_SERP_MAX_BLOCKED_COOLDOWN_MS, VITNA_SEARCH_SERP_CACHE_TTL_MS, and VITNA_SEARCH_SERP_CACHE_ENTRIES. A capture still needs the gate's admission.
Authenticated status joins the gate deadlines this process last observed to owned-google's process-local state. A reserved start reads throttled until its nextRequestAt, and a durable block cooldown reads cooldown until its cooldownUntil. Each capture opens the gate afresh, so a start reserved by another process sharing the state file appears only after this process observes it, usually through a refusal.
The implementation and live retrieval evidence remain distinct:
| Check | Verified evidence and limit |
|---|---|
| Windows process lifecycle | Native Edge authored capture and early-abort fixtures passed 2/2, confirming process drain and profile cleanup |
| Native response interception | Authored POST redirect and /goto HTTP 307, 308, and 200 fixtures passed; this verifies guards, not live Google access |
| Gateway authentication | Authored proxy-auth fixture suite passed 5/5, including native Edge CONNECT and server-challenge checks |
| Earlier Google prototype | One capture produced nine listings, eight with opaque /goto links; one separate GET returned a publisher destination with HTTP 302 |
| Full authenticated owned API | Three distinct fresh Google requests returned HTTP 200 through owned-google/browser on 2026-10-05: “example domain” returned nine publisher URLs in 5,477 ms, “NASA solar system” five in 4,989 ms, and “Python documentation” seven in 4,588 ms; all 21 URLs resolved without a challenge |
| Multi-page owned API | One authenticated job captured ten observed Google pages on 2026-10-05, returning 94 unique publisher URLs from 96 listings in 107,047 ms; source references matched, with no error or challenge; the 100-link target ended partial at the page limit |
| Local console browser flow | An isolated browser connected with the existing local key, selected Google SERP, submitted “example domain,” and rendered five publisher links after HTTP 200 in 3,959 ms |
These samples used the online PC's stock Edge browser and an owned cloud exit. The Python sample also verified full service startup at the local Search console. Health, readiness, and authenticated status returned HTTP 200, with the existing 65-page pilot index loaded. Browser and SSH cleanup completed. This small sample does not establish production reliability, geographic targeting, broad Google coverage, or a full replacement for Bright Data's retrieval infrastructure. The route remains experimental and can still face challenges.
Experimental Camoufox on an owned Linux host
Set VITNA_SEARCH_SERP_OWNED_BROWSER=camoufox to use an operator-provisioned remote Firefox worker through the same private SSH alias, declared exit, state file, and owned pacing. This mode needs no local stock browser or desktop worker. Provision platform/search/serp-camoufox-worker.py in the private remote installation alongside Camoufox 0.5.7, Playwright 1.62.0, the pinned Firefox 156 beta 34 build, private setup metadata, and permissions for its authenticated cgroup v2 parent. The repository has no supported public provisioner. The worker validates private paths and declared pin metadata; operators must verify installed binaries and packages during provisioning.
VITNA_SEARCH_SERP_ENABLED=true
VITNA_SERP_RESALE_APPROVED=true
VITNA_SEARCH_SERP_TRANSPORT=auto
VITNA_SEARCH_TIMEOUT_MS=60000
VITNA_SEARCH_SERP_TIMEOUT_MS=60000
VITNA_SEARCH_SERP_OWNED_BROWSER=camoufox
VITNA_SEARCH_SERP_OWNED_REMOTE_ROOT=/tmp/vitna-firefox.PRIVATE_INSTALLATION
VITNA_SEARCH_SERP_OWNED_SSH_HOST=owned-host
VITNA_SEARCH_SERP_OWNED_EGRESS_ID=owned-exit
VITNA_SEARCH_SERP_OWNED_STATE_PATH=.search-data/serp-browser-check-state.jsonReplace the installation placeholder privately. Stock remains the default; the remote option changes no ordinary routes or deadlines unless configured.
Each search on such a route runs the private worker over SSH, and the worker starts a second Python process for the capture. On the box on 2026-10-17 each start cost about 85 ms (17 ms for the interpreter, 19 ms to compile the worker, 39 ms for its imports), so about 170 ms of every search went to starting processes before the daemon was asked. With VITNA_SEARCH_SERP_OWNED_DIRECT_DAEMON=true the service instead writes the search to the daemon's socket, <root>/daemon.sock, with nc -U over the same SSH host alias, and reads the daemon's one-line answer: about 20 ms outside the daemon, measured the same night. When Search and the daemon run on the same Linux host, --socket-path and --socket-group on serp-camoufox-daemon.py expose an additional Unix socket in a non-world-writable runtime directory (such as /run/vitna-camoufox/daemon.sock, mode 0660 when --socket-group is set and 0600 otherwise), and VITNA_SEARCH_SERP_OWNED_DAEMON_SOCKET connects to it directly from Node with no ssh or nc process at all (under 1 ms per capture vs ~150 to 300 ms to spawn ssh ... nc -U). The answer is checked exactly as the worker's is (the page's identity, its redirect mappings and a parse that is neither a block nor a failure), and the daemon's budget keeps the worker's margin of 2 s or a tenth of the request. The daemon owns its browsers and their cleanup, so no receipt is awaited: a cancelled search closes its SSH client, and the daemon finishes or abandons that search on its own, as it does when a worker is stopped. Keep the SSH alias's ControlMaster connection open (ControlPersist): a search that has to open a new one pays its handshake, about 240 ms on the box, on either path.
A daemon with a residential pool tries up to 20 exits per search and spaces and cools each exit itself. The route's own rests were built for one IP and work against that pool: a search the daemon could not land locked the whole route for minutes, and one search at a time turned a second caller away. Set VITNA_SEARCH_SERP_OWNED_ROTATING_EXITS=true on such a route and it neither paces nor rests: failures count strikes only, and its interval settings are unused. VITNA_SEARCH_SERP_OWNED_CONCURRENCY (up to 32) lets that many searches run at once, each holding its own durable lease slot. When every slot is taken, a new search waits for one to free instead of being refused, as long as its deadline would still leave 8 s to capture (for a shared capture, the deadline of the latest caller still waiting on it). A search with less time left, or one whose try already failed, is still answered serp_routes_busy at once. A waiting search holds one of the instance's VITNA_SEARCH_CONCURRENCY request slots, so set that well above the route's concurrency: at the default of 4, a route serving 3 leaves room for one waiter and the next request is refused as 429 service_busy instead. Give the daemon more warm browsers than that (--max-slots, one upstream session line each): a search fails over browser by browser and a blocked one relaunches, so three searches on three browsers kept waiting on each other. Such a route also keeps all but 15 s of the request deadline per fallback route, rather than an even share. A route searching from a server's own address must keep both unset. Numbered routes never inherit them from route 1. The worker creates a fresh headless context and checks stable result and continuation evidence while preserving the canonical search identity. Its authenticated CONNECT guard validates every DNS answer before pinning a public target address. Browser traffic is GET-only, bounded to 128 routed requests and 128 CONNECTs, 20 MiB of aggregate bidirectional proxy TCP payload, and 2 MiB of captured HTML. Unknown background destinations remain denied without invalidating an otherwise valid capture; forbidden main navigation still fails. A reCAPTCHA frame or widget marks a challenge on any page. Challenge wording counts only on a page without recognized organic results, and never in the title, search form, result cards or scripts, which echo the query.
Up to ten observed opaque /goto links use Python TLS header-only GETs through that same guard, with at most two workers. This is a hybrid browser/Python retrieval path, not browser-native redirect mapping. It does not share Firefox cookies, follow publisher destinations, or rewrite captured HTML. Each attempt has isolated private request files. Stdin EOF, exact-attempt cancellation, and deadlines stop its owned cgroup; a matching durable receipt must prove child exit and an empty group before the API releases its gate, within the bound below. Losing SSH does not itself prove browser cleanup. A capture whose SSH run ends without that receipt fails as serp_unreachable, never as the caller's cancellation, unless a receipt recovered afterwards proves its page or a different failure. Recovery resends exact-attempt cancellation until the worker answers it, then polls for the receipt, backing off from 5 to 60 seconds. A cancellation that finds no directory for its attempt, because SSH failed before the run created it or the host was unreachable, records a terminal request_cancelled receipt in that directory's place, so recovery ends with proof. A run that arrives later for the same attempt then refuses before starting a browser. If the installation itself is gone, for example after a reboot cleared /tmp, nothing can answer and the bound applies: recovery holds the gate for at most 180 seconds, longer than any worker supervisor for the attempt can run, then releases it without proof. The capture's private drain then reports serp_camoufox_cleanup_unconfirmed, and service shutdown, which waits no longer than that bound, logs that a resource could not finish closing. Neither names the host or the query.
Once its receipt proves cleanup, an attempt keeps only that receipt: the worker deletes its request, progress, capture and DOM diagnostic files and its private home and temporary directories, so captured pages are not kept for later inspection. Each new run first removes at most eight attempt directories: any whose cleanup receipt is more than 15 minutes old, and any other attempt more than two hours old for which no cgroup exists. Removal happens as runs start, so an idle host keeps its latest receipts until its next run.
On 2026-10-05, this optional route was verified through authenticated local API requests with fresh: true and the actual Search console, using the owned OVH Camoufox worker without a paid retrieval service, proxy provider, or solver:
| Integrated live check | Observed result |
|---|---|
| Fresh API, “Python asyncio TaskGroup documentation,” page 1 | HTTP 200; nine verified organic results in 36,202 ms |
| Fresh API, “What are the latest US AI chip export restrictions?” | HTTP 200; nine organic results in 23,252 ms, including Congress and Reuters listings |
| Fresh API, the same Python query at page 2 / offset 10 | HTTP 200; ten organic results in 7,843 ms; the two Python pages contained 19 distinct publisher URLs |
| Local console, “site:bis.gov H200 January 2026 license review policy” | HTTP 200; four BIS links rendered in 6,875 ms |
Further controlled fresh samples returned nine Rust results in 13,935 ms, nine headphone results in 11,136 ms, nine for an accented French query in 9,506 ms, and three results on matching hosts for a quoted MDN query with maxResults: 3 in 9,697 ms. Python page 3 at offset 20 returned ten results in 9,398 ms, bringing the three pages to 29 distinct publisher URLs. A fresh Rust collection targeting 18 links across two pages finished completed with 18 distinct links: 26,069 ms wall time and 23,162 ms reported job time. The fresh PostgreSQL 18 MERGE baseline returned nine results with nine snippets in 8,463 ms, including 8,354 ms of owned capture, without cache reuse or coalescing. These additional checks included six successful single-page captures and two Rust collection pages before the empty-page change. After that change, a verified empty response brought the accepted checks to nine captures: seven standalone pages, including the empty result, and two Rust collection pages. No CAPTCHA, HTTP 403, or HTTP 429 was recorded in this additional sample.
Limits appeared alongside those successes. A NASA site: query retained its address-bar identity but yielded mostly unrelated or off-site listings; the scope guard refused it with serp_query_mismatch. The first French call returned HTTP 503 with Retry-After: 2 before capture. An improbable quoted query intended to test empty results twice reached HTTP 504 near sixty seconds; the stored DOM later showed the genuine Google title, query and canonical URL, an explicit “About 0 results” count, an AI-only overview, and no organic listings. The initial shell completed with three main-document HTTP 200 responses. Narrow parser and worker support passed an offline regression with that stored AI-only DOM. A later fresh retry of the original quoted query returned HTTP 200 with nine organic listings in 8,122 ms, without cache reuse or coalescing. Google had changed its response; this failed the expected empty-result check and is excluded from the accepted captures.
A separate nonexistent scoped query returned HTTP 200 with organic: [] in 6,884 ms through one fresh owned attempt, followed by clean idle state. This verifies genuine empty API output, but does not establish that the original AI-only layout was reproduced live. Forced-cancellation receipts can retain stale initial counters, so zero counters alone do not prove zero traffic.
A separate no-Google cgroup self-test confirmed child and grandchild cleanup. The earlier four-result diagnostic below retains its separate browser and header-mapping timings; those timings are not API latency. The integrated route remains experimental. This small controlled sample does not establish sustained reliability, soak or load performance, unchallenged future access, or public production deployment.
Warm standby browser daemon and latency optimizations
On 2026-10-07, the Camoufox worker was augmented with an optional standby daemon (serp-camoufox-daemon.py) to reduce single-query latency from the 3.6s to 4.7s cold baseline down to sub-second and low-second times.
Cold browser process startup accounts for approximately 1.1s to 1.3s of each independent search attempt. When active, vitna-camoufox-daemon.service maintains a pre-warmed headless Camoufox browser instance listening on a private Unix domain socket at /tmp/vitna-firefox.PRIVATE_ROOT/daemon.sock (permissions 0600, owned by the unprivileged worker user).
Key architectural properties of the standby daemon include:
- Isolated ephemeral contexts: Each incoming search query creates a fresh
BrowserContextviabrowser.new_context(). The context navigates the query, settles the DOM, unwraps redirects, and is closed immediately upon completion. No cookies, cache, storage, or history leak across queries. - Parallel redirect unwrapping: Google
/gotoopaque redirect links are unwrapped concurrently through the local guarded CONNECT proxy withGOTO_WORKERS = 4, reducing redirect resolution overhead from ~800 ms to ~250 ms. - Resource filtering: Image requests (
resource_type == "image") are aborted at the route level, eliminating unnecessary asset downloads (~170 KB payload reduction per query) without affecting search result layout or text. - Gecko subsystem tuning: Firefox preferences disable hardware video decoding, media autoplay, background animations, and session history to minimize memory consumption and thread churn.
- Resource bounds and idle auto-shutdown: The daemon automatically closes the browser process after 10 minutes of inactivity (
IDLE_TIMEOUT = 600), guaranteeing zero idle browser RAM footprint. It recycles the browser process after every 50 requests (MAX_REQUESTS = 50) to prevent memory accumulation. The production unit's cgroup allowedMemoryMax=6GandTasksMax=2048on 2026-10-13, when its ten browsers used about 2.6 GB and 982 tasks. - Profiles a stop leaves: Playwright removes each browser's temporary profile when the browser closes, but stopping the service kills its driver first, so every restart left each browser's profile in the unit's
TMPDIR(ROOT/work): 1,185 of about 29 MB, 34 GB, by 2026-10-17. At start, before it launches a browser, the daemon lists the Playwright and stock Chrome directories there (playwright_firefoxdev_profile-*,playwright-artifacts-*,vitna_chrome_slot-*), and once it serves it removes them, sparing any a running process names on its command line. It does this only whenTMPDIRisROOT/work, never in a shared one. - Seamless cold fallback: The supervisor worker checks for the daemon socket first. If the daemon is stopped, restarting, or unresponsive, the worker falls back immediately to cold in-process browser execution.
Live benchmarks on the primary vitna-api host on 2026-10-07 confirmed:
| Benchmark Query | Cold Execution Latency | Warm Daemon Server Latency | Total Round-Trip (Remote Client) |
|---|---|---|---|
| Single-result query (“apple investor relations”) | 3,658 ms | 745 ms | 1.57 s |
| 9-result query with 9 unwrapped redirects (“nvidia quarterly earnings report”) | 3,820 ms | 1,512 ms | 2.17 s |
All supervisor receipt checks, attempt isolation, and cgroup cleanup guarantees remain fully enforced.
--fast-relaunch 1 (off by default) keeps each browser slot's Playwright driver running across relaunches, replacing it only after a launch on it fails, and lets an old browser close while its replacement starts. A relaunch after a refusal took about 2.5 s on the API box on 2026-10-13 (medians of 5): closing the old browser 306 ms, the exit check 721 ms, starting the new browser 828 ms and its first context and page 619 ms. A slot launched, relaunched five times and closed there with the real browser (no exit check, no search traffic) took 1,751 ms a relaunch without the setting and 1,145 ms with it: the close it waited for went from 336 ms to none, and the browser start from 835 to 567 ms. A slot's driver stops when the slot closes, after the old browsers on it. The first try on each browser records browserStartMs and browserCloseMs, the close its relaunch waited for (null when no relaunch came before it).
Failover across verified exits
With upstream residential proxies, each search fails over across up to 20 distinct exits (DEFAULT_MAX_ATTEMPTS) within the request's SERP budget. Every attempt has 8 s and runs on one browser slot whose exit was checked as a US address that is not cooling; a challenged exit cools for ten minutes and its slot rotates to a new session.
Failed tries got faster (1.7 to 1.9 s each by 2026-10-12), so the cap, not the deadline, became what ends Google: 384 of 427 searches Google missed stopped at exactly 20 tries, with 38.9 s used at the median of a budget near 48 s, while tries after the 13th still delivered 6.4%. --max-attempts (1 to 40, default 20 in legacy profile, 2 in --profile standard) sets the total try cap, and --max-block-attempts (0 to 40, default 0 off in legacy profile, 2 in --profile standard) stops a search immediately once Google returns that many serp_blocked responses so a 120-second query hold never cascades across additional warm residential exits; a 40-try history is about 100 KB. The worker reads the history back through a progress file capped at 1 MiB, which holds the longest one a search can make: 42 tries (the cap plus a race's overrun, below) with every field at its bound, about 750 KB, built by the daemon tests. Every answer says what ended it in endedBy: delivered, attempts (the cap), exits (no browser could give it an exit, below), deadline, or stopped (a race another chain won).
A search the deadline ends has to report before the launcher stops waiting, and the launcher counts its budget from before the worker started. So the worker hands the daemon its own remaining time less 2 s, or a tenth of it on short budgets, as room to write its files and exit. Until 2026-10-12 it handed over all of it, and such searches were cut off with their answer unread: 48 of the 49 that Google missed at --max-attempts 32, and 3 of 3 at 40, left receipts without the daemon's tries.
A search with no exit to be had ends at once as serp_unreachable, which Search hands to Bing, instead of waiting out its deadline. That is so in two cases. Every browser has already failed to find this search an exit, or the provider has stopped giving exits: 20 exit probes in a row have failed across the pool, the last within 30 s, and no browser is still live. On 2026-10-12, from 16:00:38 to about 16:13, every DataImpulse line answered 407 TRAFFIC_EXHAUSTED, no browser could launch, and each search waited its whole 49 s before Bing answered it. A probe that answers ends the run of failures, and after 30 s without a failed probe the next search probes again, so the pool resumes by itself once the provider is back.
An exit that just delivered (proven) delivered 79.8% of the first tries it served and 25.9% of the later ones, against 29.5% and 7.3% for other exits (22,708 tries, 2026-10-10 to 12), yet 72% of its tries went to searches that had already missed. --proven-first-wait-ms (0 to 5000, unset by default) keeps proven exits for first tries: a later try takes one only when no other warm browser is free, and a first try waits up to that long for one that is busy or coming off its spacing. Each try records firstTry, provenExit and, on a first try under the setting, provenWaitMs.
A 100-search run on 2026-10-08 delivered 56. In its last 43 searches, 246 exit attempts went as follows: 177 met Google's reCAPTCHA, 31 stalled, 21 delivered. A stall is a results page whose header Google flushes (title and search box, about 84 KB of DOM) and whose body never arrives; each was waited out to the attempt deadline. Two changes spend the budget on more exits instead:
- Stall cut-off: an unrecognized page that stops changing for 2.5 s (
STALLED_PAGE_GRACE) ends the attempt asserp_timeoutwithunrecognizedReason: "google_page_stalled". Healthy captures settle within about a second; no stall recovered in production. - Exit freshness: an exit proved by a launch probe or a post-capture check within 15 s (
EGRESS_FRESH_SECONDS) skips the before probe, about a second per attempt. A cooling exit is still refused, the post-capture check still runs on every delivered page, and the attempt history marks the skip withegressCheckSkippedandegressVerifiedAgeMs. - A page that changes while its links resolve: Google can keep rendering after a results page has held still. When the final check finds the page changed but still a results page for the same search, the daemon resolves only the links that are new and checks once more, instead of discarding a delivered page (about 280 KB through the exit) and starting a fresh try. It fails as before if the page changes again. The attempt history marks this with
pageChangedDuringRedirects. About 2% of searches hit it on 2026-10-09.
--fingerprint-os (linux, windows or macos, default linux) sets the operating system the browser fingerprint presents.
Each line of the upstream proxy file (--upstream-proxies-file) is one warm browser's session: host:port:user:pass, user:pass@host:port (DataImpulse's dashboard export) or a full http://user:pass@host:port URL. After a block the browser rotates to a new exit before relaunching, in the provider's own way. Decodo keeps the session in the username (...-session-<id>-...), so the daemon swaps the id. DataImpulse keeps it in the port: each port from 10000 to 20000 holds one exit for the dashboard's rotation interval (measured 2026-10-10), so the daemon moves the browser to a random new port on the same login. Set DataImpulse to Sticky; with Rotating, the check page, the results and the exit checks would each leave from a different IP. Either way the exit is verified through ip.decodo.com/json, which answers through both.
--direct-exits (4, 6 or 4,6, unset by default) adds one browser per family that leaves from the host's own IPv4 or IPv6 address, through the same guard with no upstream line. Direct browsers come on top of --max-slots and first: a direct address is tried whenever its spacing allows, and the residential browsers take the rest. --direct-interval-ms (4000 to 60000, default 6000) spaces searches on each direct address, proven or not. On the API box on 2026-10-13, one browser a family at 10 searches a minute delivered 593 of 601 first tries over IPv4 and 297 of 301 over IPv6 (98.7% each, median 0.84 and 0.80 s), against 61 to 75% through the residential lines beside them. An address's standing moves with use, though: by 2026-10-14 the IPv6 delivered 78%, steadily, with udm=14 or the full page alike, and the IPv4 was refused 4 times in 5. Google refused single E2B addresses within 8 to 10 minutes at 27 or more searches a minute, and within about 40 at 15.
Nothing can rotate a direct address, so a refusal relaunches its browser on it, and the address cools only when 7 of Google's last 10 answers to it were refusals, for 10 minutes; one more refusal after that cools it again until deliveries push the refusals out. With --keep-proven-exits 1, a refusal after a delivery keeps the browser instead, unless it is the one that cools the address. Simulated at 10 searches a minute, a steady 22% refusal rate keeps an address in use 91% of the time and 80% keeps it in use 1%. A residential exit's cooldown after every refusal idled both addresses for most of a 30-minute run beside production, and since a first version counted direct browsers within --max-slots, their cooling left that arm two browsers. A browser closed while its address cools is relaunched in the background once it has cooled, at most every 30 s, so a family whose route is down costs one probe a half-minute. A direct browser stays warm whatever --idle-timeout says, is launched without geoip as it was measured, and has its address probed at launch only: nothing can move it under a page, and probing each search would load the echo from one address. Its probes leave the provider-outage watch alone, and its record stays out of the network judgments, which its rate would skew. Each try records directFamily (4, 6 or null).
--direct-sources (public addresses of this host, comma-separated, unset by default) adds one direct browser per address, bound to it: its connections and its launch probe leave from that address, and the address echo must report that same address back or the browser is not launched. It is for addresses added to the host for searching, such as OVH Additional IPs on ens3, each spaced --direct-interval-ms apart (60000 for one search a minute) and cooled by the same rule. By 2026-10-14 the box's own IPv6, after a day of tests at ten a minute, was refused on all 30 searches at one a minute; whether a fresh address used gently stays healthy is what a day at one a minute on new addresses is to show.
--static-proxies-file (a file, unset by default) adds one browser per line whose exit never changes, such as a dedicated ISP proxy's own port, written as the upstream lines are; blank lines and # comments are skipped. Static browsers come after any direct ones and ahead of the residential lines, on top of --max-slots, and are kept as a direct address is: never rotated, spaced --direct-interval-ms apart, cooled only when 7 of Google's last 10 answers to the address were refusals, never closed for idleness, and relaunched in the background once cooled. Unlike a direct browser, a static one launches with geoip from its exit, as a residential one does, and its pages are checked after they deliver, since its provider could move it. Its launch probe, through its own line, proves the exit until such a check fails, so its tries skip the probe before Google. Its probes leave the provider-outage watch, which is for the rotating lines, alone. With --keep-proven-exits 1, a refusal after a delivery keeps its browser, but never one that cools the address. A fixed exit, static or direct, takes a search's first try only: a later try runs while Google holds the search, when any exit is refused, and each refusal costs an address that cannot be swapped (after one, a proven exit's next first try failed 24.6% of 658 times against 15.3% of 2,850 after a delivery, 2026-10-14 and 15), so later tries go to the residential browsers, and a daemon with none ends a refused search at once. --fixed-retry-exits 1 (off by default) relaxes that exclusion for at most two failed static proxy tries per search (static_failures < 2) while still keeping direct host addresses (--direct-exits, --direct-sources) first-try-only and preferring free proven exits ahead of unproven static lines in retry windows, so a first try that lands on a flagged static ISP port can fail over onto one more healthy static ISP port before falling through to residential exits. Once a warm fixed exit has delivered at least once, or during its first two cold tries (consecutiveBlocks < 2), it takes every first try it can, whether or not its last try delivered: in the ISP trial of 2026-10-15 a line whose last try was a refusal it had not kept delivered 32 of its next 36 first tries, against 247 of 327 for the residential exits that took first tries while it waited; a never-proven fixed exit (successes == 0) with 2 or more consecutive refusals yields priority to warm rotating and proven exits so externally hot static lines on a fresh health ledger cannot starve rotating exits across 70 cold refusals before the 7-of-10 window cools them. A fixed exit also rests 3 hours when its first tries over the last hour delivered under --fixed-decay-floor-pct (0 to 99, default 85, 0 never), once it has 20 and at least 8 of them were refused: the box's own IPv4 delivered 98.7% of first tries on 2026-10-13 and was refused 4 times in 5 a day later, a slide the 7-of-10 window never sees. The 8 refusals keep a short streak from resting a good line, since 20 first tries is two minutes at 10 searches a minute: on 2026-10-16 a line rested for 4 refusals in its first 20 and delivered 168 of its next 172. Replayed over every ISP line's first tries from 2026-10-15 to 17, the 8 turn 10 rests into 3, and a line refused 1 time in 5 still rests, by about its 40th first try. A line with peers, the other fixed exits with 20 first tries in the last hour, rests only when it delivers 8 points or more under their median, or under 70% whatever they do. When every line dips at once, as all five did from 05:30 to 07:40 UTC on 2026-10-16 (84% to 91%), a rested line's first tries go to the residential exits, which delivered 73% of theirs that night, while the two left running delivered 92% and 94% of their first tries over the next two hours. Over the same replay this keeps the one rest that was a line's own (84% beside peers at 98%) and drops the two in that dip (84% and 85% beside 91% and 88%). A fixed exit refused on 8 first tries in a row, with no delivery between them, rests 1 hour, however long they took and whatever its peers do, and back from that rest 2 more refusals in a row rest it again: the 7-of-10 window cools a refusing line 10 minutes at a time, one first try a cooldown, so a line Google has flagged never reaches 20 first tries in an hour. On 2026-10-17 four ISP lines refused every first try (0 of 9 each) for an hour after the AA benchmark's agent queries and were never rested, each try holding its search until Google's window. A line delivering 85% refuses any 8 first tries in a row about once in 4 million, and 2 in a row 2.25% of the time. The rest is an hour, not three, because a burst of re-sent queries, which Google refuses from any exit, gave every working line its 8 within minutes the same night, and 3-hour rests left no line at all while plain queries delivered 20 of 20. A flagged line costs 2 refused first tries an hour.
--fixed-long-words (3 to 32, default 0, off) makes a search of that many words or more a long one, as an agent's are, and a fixed exit takes a long search's first try at most once every --fixed-long-interval-ms (1000 to 3600000, default 60000). A long first try that finds every fixed exit paced goes to the residential browsers, and never waits for a proven exit it may not take; short searches keep the lines' own spacing. Ten ISP lines took short queries at 8 to 10 a minute each for 6 to 10 hours at 97% delivered (2026-10-16), while the AA benchmark's agent queries, long and often quoted, at about 3 to 5 a minute an address flagged five of them within minutes (2026-10-17). Each try records longQuery. Each try records staticLine, the line's number in the file from 1. The daemon refuses a file with a line it cannot read, a repeated line, or a line on a port that rotates its exit: a DataImpulse port, which is a session, or port 8000 of Oxylabs's self-service dedicated ISP gateway, disp.oxylabs.io, which gives a random address of the plan on each request where each port from 8001 holds one (Oxylabs's documentation, retrieved 2026-10-14), or the same port of isp.oxylabs.io, the gateway an account's dashboard gives for the plan. Its errors name the line's number, never its text, which carries a password.
--proven-interval-ms (1000 to 10000, default 5000) spaces a proven exit, one whose last outcome was a delivery within the last five minutes, this far apart; every other exit keeps 10 s. At 10 s for all, a browser on a good exit sat idle while searches queued for one. An A/B on 2026-10-10 (four alternating runs of 100 searches): proven exits reused under 7 s apart delivered 86% of tries (84% at 10 s), the median wait for a browser before a try fell from 513 ms to 0 (p90 3.0 s to 1.1 s) and the median search from 6.3 s to 5.4 s, with blocked tries and bandwidth unchanged. Each attempt records the spacing its admission set as egressIntervalMs.
--proven-seconds (60 to 3600, unset by default, which keeps 300) sets how long an exit stays proven after it delivers, for the slot preference, the spacing above and each try's provenExit alike. Replayed over the load tests' 31,636 tries to 2026-10-12, a first try delivered 76.6% of the time when its exit had delivered under a minute before, 74.2% at 5 to 15 minutes (31 tries), 43.8% at 15 to 60 (16) and 33.3% past an hour (33), against 29.4% for an exit never seen. Under load no first try met a gap of 1 to 5 minutes, so five minutes ends the preference just when sparse traffic could use it. The samples past five minutes are small: the setting is for a trickle test of 900 to 1800 s, not yet a default.
--race-width (1 to 3, default 1) runs up to that many chains of tries for one search at once. A search fails only when every try that fits before its deadline is refused: a failed try costs about 2.3 s, so about 22 fit, and with 7.6% of long-tail tries delivering that predicted 17.6% failures against 16% measured (2026-10-11); across eight runs of the same 100 queries, misses per query followed chance, not the query. The extra chains take only a warm browser while no other search waits for one, so a busy daemon serves exactly as before. They share the exits already tried, the first delivery answers, and the others finish the try they are in and stop. They share the cap too: no chain starts a try once the chains have made --max-attempts between them, so a raced search ends at most two tries over it, the ones still running. Capped chain by chain, three chains at 40 could make 120 tries, more than the progress limit holds, and a history past that limit fails the search with its delivered page. Every chain's tries are in the answer's attemptHistory, tagged raceChain, with raceWidth and raceChainsTried beside them.
--retry-window-ms (1000 to 30000, unset by default) changes what a search does once Google has refused it. Google lets a refused search through again about every 10 s: over 5,710 tries at production's setting on 2026-10-11 and 12, a later try admitted 10 to 11.5 s, 20 to 21.5 s or 30 to 31.5 s after the search's first delivered 21.9% of the time, and one admitted between those windows 4.6%. The gaps held 68% of tries and 64% of bytes but 28% of Google's answers. Live receipts the same night agreed, 9.2% against 2.8%, on browsers of the same age inside and outside the windows. With the flag, a refused search's later tries wait, holding no browser, for windows at its first admission plus the period, twice the period and so on, each 1.5 s long, and are admitted on the mark. --retry-window-tries (1 or 2, default 2) lets a window take a second try, no earlier than 0.8 s past the mark, when the first ends inside it; a window's tries never run together, which is how racing lost. --retry-window-after-refusals (1 or 2, default 1) lets a refusal on the first try or an unproven exit try one more exit immediately before windows begin, while a later refusal on a proven exit still starts windows at once. --retry-window-from-refusal (0 or 1, default 0) anchors each retry window a full --retry-window-ms period after the latest Google refusal finishes rather than from the first try's pre-navigation admission, so probeBefore, proxy TLS and immediate strikes do not shorten the 10 s quiet gap before Window 1 or fire a second turn 0.8 s after a refusal inside a window; window turns are counted only after probeBefore succeeds, so an idle sticky session whose exit rotated (ip_changed) before Google is contacted yields immediately to the next warm slot without consuming Window 1. A failure that is not a refusal does not push the search to the next window, and a window that cannot fit before the deadline ends the search, leaving Bing the time. Each try records sinceFirstAdmissionMs and, when it aimed at a window, windowTargetMs. The flag needs --race-width 1. What an A/B of the two has to show is whether the answers found between windows today come back in the next window.
--window-proven-exits 1 (off by default; needs --proven-first-wait-ms) lets a later try that starts in one of those windows, every 10 s after the search's first admission and 1.5 s long, take a proven exit that is free, as a first try would, without waiting for one. In the windows a proven exit doubles a later try's chance: over the production-setting runs of 2026-10-12/13, later tries admitted in a window delivered 36.5% on a proven exit (52 tries) against 17.0% on another (802). Only 6% of window tries had one, since the proven first-try wait keeps proven exits for first tries, where they are worth more still (75.0% against 40.7%), so a window try never waits for one. The window is judged when the pool hands the try its browser, not when the try starts waiting for one, so a try that waits into a window takes a proven exit there and one whose wait outlasts a window does not. A try aimed at a window by --retry-window-ms takes its browser 0.5 s before the mark, before Google's window opens, and counts as that window's try; without that, the two flags together never gave a window try a proven exit. Each try records windowProven.
--keep-proven-exits 1 (off by default) keeps an exit that has delivered through its first failure in a row. After a refusal the exit and its browser stay, and the refusal's cooldown is cleared; after a timeout or an unreachable check the exit stays and the browser is relaunched on it. A second failure in a row moves the browser to a new exit, as without the flag. Without it, a refusal sends an exit that has been delivering to the back of the queue for at least 10 minutes and its browser hunts through fresh exits, each one a browser relaunch. In A/Bs beside production on 2026-10-13 (one browser per arm, 10 searches a minute, one try a search), a proven exit kept through one refusal delivered the next search 16 times in 25, where a fresh exit delivered 8 to 13%, and keeping proven exits took first tries from 60.7% to 82.6% (about 448 searches an arm). Each try records keptExit: browser, exit, or null when the slot moved on.
--spare-gap-refusals 1 (off by default) treats a later try's refusal between Google's windows (above), on an exit that has delivered, as the query's and not the exit's: it neither cools the exit nor counts toward --keep-proven-exits' second failure, the browser stays, and the exit's record changes only in when it was seen. Google holds a refused query whatever the exit: with both levers on beside production on 2026-10-14 (1,200 searches), later tries between windows delivered 7.6% on exits that had just delivered and 8.4% on the rest, and an exit kept through such a refusal delivered the next first try on it 31 times in 42. Yet two of them in a row were what swapped out 133 of the 179 exits that run dropped after delivering. An exit that never delivered is swapped and cooled as before, which keeps fresh exits coming for refused searches, and a search may take an exit it spared once more, in one of Google's windows, where later tries on proven exits delivered 17 of 26 in the same run. A first version spared every exit, and beside production its refused searches ran out of exits: 18 of 211 ended at their deadline after 5 to 7 tries, against 4 of 214 for the control. Nor does a refusal between windows give any exit its first verdict, which --network-yield-floor-pct judges networks by: in the same run 915 of the 1,145 fresh exits Google first answered met it between windows, where 79 delivered (8.6%), against 34 of 76 on a first try, so the floor was judging how often a network's exits were handed held queries. A refusal on a first try or in a window, and any failure that is not a refusal, counts as before. Such a try records keptExit as spared, and a try on an exit the search spared records sparedRetry.
--after-check (wait or background, default wait) decides whether a delivered page waits for its post-capture exit check, the fresh probe through the same line that starts when the page settles. On 2026-10-11 that wait was about 0.5 s at the median, and the check passed 132 times out of 132. With background the validated page goes back at once (probeAfterWaitMs: 0, exitCheckAfterReply: true) and the check finishes on its own, keeping its browser until it settles, so the next try there still starts from a proven exit. A check that fails withdraws the exit's proof, so the next try probes it before Google, and the page already returned stands: what background gives up is the guarantee that that page came from its stated exit. Each try reports the daemon's running afterReplyChecksPassed and afterReplyChecksFailed. Measured the same night over 400 long-tail searches in alternating arms, background made the delivering try 0.50 s shorter at the median and 0.61 s at p90. The wait for a browser, tries, blocks and bytes did not change, and 1 of 162 background checks failed. Production runs background from 2026-10-11 23:46 UTC.
Every try that gives up as google_page_stalled records mainBodyFinished: false when Google sent a page's header and never finished its document, true when a complete page simply showed no results. On long-tail queries on 2026-10-10, a stalled try held a page of 74 to 75 KB every time (121 tries) and cost 3.6 s before the 2.5 s grace ended it. --stall-samples (0 to 32, default 0) keeps the last N such pages for an hour in stall-observations/ under the daemon root, beside a JSON report of the status, page size, hold time and mainBodyFinished, so the page can be recognized at once. The directory is private (mode 0700, files 0600, generated names) and is never mixed with challenge-observations/. A page can carry the query, so it stays off unless an operator is diagnosing stalls; each try that kept one names it as stallSampleId.
Every try also records requestKinds: each request the page made and what the guard did with it (allowed, cached, skipped, or refused: for its host, method or type), as method, request type, a Google host name (any other host reads other) and at most two path segments, never a query, twelve kinds at most. failedRequestKinds counts allowed requests the browser then failed, by browser error code, and a stalled try records pendingRequestKinds, the allowed requests still unanswered when the grace ran out. Serving a stalled check page the one Google file it asked for left it stalled (2026-10-11), so these name what else it waits on.
They showed what most stalls were: Google's quiet refusal of the results request. The check page passes, and the try's second document is a complete page titled for the query, with the search form and its empty containers but none of the results column (#search, #rso, #rcnt, #center_col, #botstuff); it asks only for a logging beacon and a font, as every delivered page does, and never moves on. Once that second document has finished arriving and the latest response is a success, the daemon ends the try after 1 s unchanged as serp_blocked, with unrecognizedReason google_page_held and heldMs, instead of waiting out the 2.5 s stall as a timeout. The exit then cools down like any other refusal (a timeout records nothing against it), and the operator monitor counts the try as a challenge. A first document, one still arriving, a page moving on, a page with a results column, or one titled only "Google Search" still stalls as before.
--held-grace-ms (100 to 5000, default 1000) sets how long that page must stay unchanged. About one long-tail try in four ends this way, so the grace is paid often. A page that looked held and then turned into results is what a shorter grace would lose, so a try that recovers records heldRecoveredMs, how long its page looked held, and every try reports the daemon's running heldPagesEnded and heldPagesRecovered.
Before a browser launches on a new exit, the daemon may pass over the exit's network and probe another (at most twice a pick, about 0.8 KB a probe). By default a network is passed over when, over 30 or more of its tries, it delivered under half as often as the pool, except for one exit in five. --network-yield-floor-pct (0 to 50, default 0, off) judges a network by Google's first verdict on each of its fresh exits instead, a delivery or a block, so one proven exit delivering again and again cannot lift it: a network whose fresh exits delivered under the floor, over 5 or more of the exits the daemon remembers, is passed over except for one exit in ten. Replayed over 17,773 fresh long-tail exits from 2026-10-10 to 12, a 5% floor would have passed over 24% of them, which delivered 4.6% of the time against 12.0% for the rest.
A try Google redirects away from its own search ends as serp_redirect_blocked (7.6% of long-tail tries in one run on 2026-10-11). It records redirectDrift, which says what moved and never a value: its source (the navigation request, its response, the page URL, or the proxy's CONNECT guard), the target's host class and first path segment, whether a fragment appeared, how many q parameters there are and whether the query matches (exactly, ignoring case, or containing ours, with the change in word count), whether start matches, and the names of query parameters added or removed against ours.
When the redirect leaves www.google.com for another host, the slot moves to a new exit at once and the exit cools for an hour, as after a block, though it counts as no challenge. Every redirect read on 2026-10-11 (103 tries in two runs) came from one exit that Google sent to www.google.com.hk/url, its country-domain redirect, which follows the exit rather than the query; the slot that kept that exit failed every search that reached it. A redirect that stays on www.google.com keeps its exit, since another query or offset may be the query's own.
--goto-resolver (http2 or browser, default browser) chooses who asks Google where each result link points. http2 is the Python HTTP/2 client described above, from the same exit, and sends no cookies or Referer. browser uses the page's own fetch with a manual redirect, credentials: "same-origin" and no Referer, so the lookups carry Firefox's TLS and reuse the non-anonymous HTTP/2 connection that loaded /search (rather than partitioning into LOAD_ANONYMOUS and opening extra upstream proxy CONNECT tunnels: 11 vs 13 connects per capture and 326 ms vs 586 ms 2-link redirect median in a 16-query Latin-square on one warm residential exit on 2026-11-03, 16/16 delivered with 0 holds); the daemon reads Location from the response the browser reports, and the publisher is never requested. Each attempt records gotoResolver. An A/B on 2026-10-09 found no difference in how often Google blocked an exit's next search after either, so TLS on the lookups is not a block signal; browser is the default because it skips the lookup client's own tunnel and handshake, about 10 KB of residential traffic per delivered page (296 KB against 306 KB for a delivering attempt). When --goto-respect-num 1 targets 1 or 2 initial links under --goto-refresh-max-ms or --goto-pace-ms, render_search also starts that zero-pace burst once during DOM stability settling (never cancelling or restarting in-flight tokens, cutting 2-link redirects phase median to 251 ms).
--goto-refresh-max-ms (0 to 60000, default 0, off) lets the browser and http2 resolvers wait out Google's hold on a result link. On 2026-11-02 the first two or three links on a page answered a 302 and each further one a 200 "Verifying your request" page whose meta refresh names a new /goto link and a wait that grew link by link: 3, 4, 5, 7, 12, 22 and 24 s. Once its wait was up, a plain fetch of the new link answered the 302, with no cookies and no script (34 of 34 across six pages). Off, such a link fails the search as serp_invalid_response, which made every served page fail. On, navigation and page settling keep their 8 s attempt ceiling, link lookups wait until the page has settled so streaming cards do not cancel in-flight holds, and a settled page extends its redirect budget by --goto-refresh-max-ms within the search's deadline. Each lookup reads the 200's body, waits the named seconds plus half a second, and fetches the link it names, up to two holds per link; a hold whose waits add up past the cap, or that cannot end before the search's deadline, is refused at once and fails the search as before. A page's links wait side by side, so its slowest hold sets the cost: 22 to 24 s on a page of ten links, longer than the 18 s VITNA_SEARCH_SERP_GOOGLE_SHARE_MS production ran on 2026-11-02, so turning this on means setting at least 30000 and raising that share and the caller's budget past 35 s. Each try records gotoHolds, gotoHoldsRefused and gotoHoldWaitMs, the longest wait one link was given.
--goto-max-links (1 to 10, default 10), --goto-respect-num (0 or 1, default 0) and --goto-pace-ms (0 to 2000, default 0) shape how many discovered /goto links a capture unwraps and how 3-to-5-link lookups are spaced. On 2026-11-03 a 49-page, 7-arm Latin-square sweep (n = 7 per arm across 4 warm residential Camoufox slots, 294 /goto lookups, 49/49 delivered) mapped Google's per-impression /goto unlock curve (shared across exits, browser contexts and document navigations for a single SERP impression; 94 to 120-byte AEAD protobuf tokens, zero publisher URLs in HTML, <script> blocks or XHRs): links 1 and 2 answer instant 302s (~0.64 to 0.80 s total SERP latency on warm slots), links 3, 4 and 5 unlock non-uniformly at ~0.50 s, ~1.05 s and ~2.30 s after page settle (14/14 zero-hold 302s on links 3 and 4 at 550 ms step, 12/14 zero-hold on link 5 at 750 to 850 ms step where the 14% held tail pushed 5-link p90 to 6.25 to 6.54 s, and 7/7 zero-hold on link 5 past 2.65 s at 3.15 s unwrap p50 / 3.23 s p90), link 6 hits a ~7.6 s server-side hold (49/49 = 100.0% held across every arm from 0 to 1000 ms, 8.03 to 8.79 s unwrap / ~9.63 s capture p50), and links 7 through 10 hit a ~24 s wall-clock unlock ceiling regardless of spacing. --goto-max-links caps unwrapped /goto links per capture; --goto-respect-num 1 bounds that cap by the search URL's num parameter (unwrapping only num links upfront when num < 5 and pulling spare links up to the 5-link ceiling on demand only when an earlier result resolves to a doorway page, while num=5 caps strictly at 5 links rather than spilling into link 6's 7.6 s hold, clamped to --goto-max-links); and --goto-pace-ms scales the non-uniform per-index unlock schedule (0, 0, 550, 1100, 2650) ms by pace_ms / 1000 when 3 to 5 links are unwrapped, while leaving 1-to-2-link and 6-to-10-link captures unpaced so neither pays extra delay. Each try records gotoDiscovered, gotoSelected, gotoSpareUsed and gotoPaced. When --goto-refresh-max-ms or --goto-pace-ms is active, non-first-try attempts and --retry-window-ms window sleeps reserve the expected post-settle /goto unwrap time (minimum_unwrap_seconds: 24.75 s for 7-to-10-link pages under a 25 s+ hold cap, 8.75 s for 6-link pages, 2.90 s for 5-link pages, 1.35 s for 4-link pages, and 0.80 s for 3-link pages at --goto-pace-ms 1000, 5.75 s for unpaced 5-link pages, and 0 s for 1-to-2-link pages) on top of the 2.5 s try minimum (27.25 s total for num=10, 5.40 s for paced num=5), breaking immediately instead of sleeping 10 s into a retry window that cannot finish unwrapping /goto links before the deadline. That reserve lets a 20 s maxResults=5 budget and a 42 s maxResults=10 budget each fit one post-hold 10 s retry window and end failed searches at ~12.3 to 12.8 s rather than burning 32 to 57 s across doomed late windows.
--ab-arm-b and --ab-salt run an A/B search by search rather than in blocks of runs. On 2026-10-12, runs of 100 searches on one unchanged setting swung from 68 to 93 Google answers, as much as any setting tested, so an A/B that alternates runs measures the hour as much as the setting. With both flags, each search falls in arm a, the daemon's own settings, or arm b, those given to --ab-arm-b on top, by a hash of the salt and its query with each run of whitespace as one space. Both arms then share every minute, the pool and its exits; a report can tell any search's arm from its query alone; and a new salt each run moves about half the queries to the other arm, so over several runs each query is seen in both. Arm b takes only settings a search reads for itself: --max-attempts, --max-block-attempts, --proven-first-wait-ms, --held-grace-ms, --retry-window-ms, --retry-window-tries, --retry-window-after-refusals, --retry-window-from-refusal, --window-proven-exits, --keep-botguard-cookies, --warm-google-session, --goto-refresh-max-ms, --goto-max-links, --goto-respect-num, --goto-pace-ms and --fixed-retry-exits, with the same bounds as the daemon's own; windows in arm b need --race-width 1. The arms still share one pool, so the proven first-try wait, which changes which browser a search takes, and retry windows, which free a browser between windows, are measured beside the other arm's searches rather than in a pool of their own. Each answer records its arm as abArm.
--browser-engine (camoufox or chrome, default camoufox) and optional --chrome-path let the daemon run stock Chrome or Chromium instead of Camoufox behind the same socket protocol, guarded local proxy, exit-health record and supervisor checks. When DISPLAY is unset on Linux, the pool starts one private Xvfb display (1366x768x24, -nolisten tcp, discovered through -displayfd) shared across its slots and restarts it if it dies. Each slot spawns Chrome in its own process group with a private vitna_chrome_slot-* profile and HOME directory under TMPDIR, --remote-debugging-port=0 on 127.0.0.1, --disable-blink-features=AutomationControlled, QUIC and non-proxied WebRTC UDP disabled, and --host-resolver-rules mapping every non-loopback hostname off-host so DNS and TCP stay inside the slot's StandingGuardedProxy (127.0.0.1:0 with require_auth=False, since Chrome does not take proxy credentials in --proxy-server). Playwright attaches over loopback CDP (connect_over_cdp) and reuses the slot's driver across relaunches when --fast-relaunch 1 is set. On 2026-11-02 Google's BotGuard refused CDP browser.new_context() (incognito context with 1280x720 device-metrics overrides) and context.route_web_socket()'s injected WebSocket wrapper script, so ChromePageContext opens a fresh tab on contexts[0] with page-level request routing and clears cookies and Google origin storage (Storage.clearDataForOrigin) when the tab closes, preserving Google's challenge-pass cookies (__Secure-STRP, AEC, NID, SEARCH_SAMESITE and legacy SG_SS; on 2026-11-03 Google retired SG_SS in favor of __Secure-STRP + NID + SEARCH_SAMESITE, and retaining the full set per verified exit IP collapsed cold 2-hop BotGuard redirects from 1.03 s to a single 200 /search hop in 0.64 to 0.72 s across 49/49 warm Camoufox captures) when --keep-botguard-cookies 1 is enabled. With --warm-google-session 1 (off by default), each stock Chrome slot primes https://www.google.com/?hl=en through its guarded exit on launch and retains Google cookies (NID, AEC, __Secure-STRP, DV, SEARCH_SAMESITE) and origin storage across delivered captures on that same exit IP, wiping cookies and Storage.clearDataForOrigin immediately on any block, capture failure, cold A/B arm or session rotation; each try records warmSessionApplied, warmSessionRetained, googleSessionWarmup and googleSessionWarmupMs. Because Chromium over CDP suppresses Playwright page.on("response") events on manual opaque redirects, BrowserRedirectClient reads 3xx Location headers from a page CDP session (Network.responseReceivedExtraInfo and Network.requestWillBeSent redirectResponse) while waiting out 200 refresh holds under --goto-refresh-max-ms. On the OVH host on 2026-11-02, a 26-slot warm Chrome pool (10 static ISP lines plus 16 sticky residential lines) plus Xvfb warmed with --warm-google-session 1 in 13.0 to 15.6 s using 7.98 to 8.05 GB of physical memory (Pss, about 296 to 298 MB per slot), --fast-relaunch 1 started replacement Chrome browsers in 274 to 519 ms, and each try records browserEngine (camoufox or chrome). In sequential 50-query maxResults=10 runs at concurrency 3 (--goto-refresh-max-ms 30000, timeoutMs=58000), cold Chrome delivered 30/50 (60.0%, 13/50 first-try, 17/37 retry recovery, 290 holds waited and 12 refused; delivered-only latency 36.4 s p50 and 56.6 s p90, 25.9 s first-try p50), whereas two runs adding --warm-google-session 1 and --fixed-retry-exits 1 delivered 33/50 (66.0%, 28/50 first-try) and 42/50 (84.0%, 35/50 first-try, 7/15 retry recovery, 345 holds waited and 2 refused; delivered-only latency 25.9 s p50 and 35.4 s p90, 25.6 s first-try p50), or 75/100 (75.0%, 63/100 first-try) pooled across two 50-query runs of the same query list. In standalone 100-distinct-query runs on separate fresh 26-slot pools at concurrency 3 (--warm-google-session 1 --fixed-retry-exits 1 --goto-respect-num 1 --goto-pace-ms 1000 --goto-refresh-max-ms 30000), maxResults=10 (timeoutMs=58000) delivered 79/100 (79.0%, 95% Wilson CI 70.0 to 85.8%; 66/100 first-try, 13/34 retry recovery; quartiles Q1 21/25, Q2 20/25, Q3 15/25, Q4 23/25; unconditional all-100-query latency 26.1 s p50, 47.7 s p75, 52.9 s p90; delivered-only latency 25.7 s p50, 37.3 s p90; 651 holds waited and 6 refused), and maxResults=5 (timeoutMs=35000, started immediately after on an empty health ledger while the 10 static lines were hot: 0/9 static ISP vs 72/91 = 79.1% residential first-try) delivered 76/100 (76.0%, 95% CI 66.8 to 83.3%; 72/100 first-try; quartiles Q1 6/25 while cooling the 10 hot static lines, then Q2 23/25, Q3 23/25, Q4 24/25 on warm residential slots; 0 /goto holds waited or refused across all 76 delivered pages; unconditional all-100-query latency 5.74 s p50, 28.2 s p75, 32.3 s p90; delivered-only latency 4.06 s p50, 6.18 s p90, 4.03 s first-try p50). Failed queries run to deadline, so unconditional all-query p90 remains deadline-bound whenever 10% or more of queries fail. To remove hour-to-hour and pool-draw confounding, a simultaneous two-period AB/BA crossover across two 13-slot daemons (40 paired maxResults=10 queries dispatched in lockstep and swapped across both pools) delivered 17/40 (42.5%, 10/40 first-try, 25.8 s delivered p50) with --warm-google-session 1 --fixed-retry-exits 1 against 4/40 (10.0%, 0/40 first-try, 37.4 s delivered p50) on cold Chrome (13 vs 0 discordant delivered pairs, two-sided McNemar p = 0.00024; 10 vs 0 discordant first-try pairs, p = 0.0020). On a fresh 26-slot pool running 60 maxResults=5 queries (timeoutMs=45000) split query-by-query via --ab-arm-b="--goto-pace-ms 1000", overall delivery was 50/60 (83.3%, 36/60 first-try): most of the latency drop relative to maxResults=10 comes from Google rendering 5 /goto links instead of 10 (avoiding the ~24 s hold ceiling on links 7 through 10), while --goto-pace-ms 1000 cut first-try p50 from 6.80 s (29/32 delivered, 88 holds waited, median 3 holds per page) to 4.09 s (21/28 delivered, p = 0.17 vs unpaced; 0 holds waited across all 21 delivered pages; 4.30 s vs 7.12 s delivered p50, and 5.02 s vs 7.14 s unconditional all-query p50). Follow-on batches on an already-pruned pool delivered 19/20 (maxResults=5, 4.39 s delivered p50) and 14/15 (maxResults=2, 2.76 s delivered p50, 0.88 s min, where Google already renders only 2 /goto links so pacing and num capping are no-ops). Follow-up 100-query fresh-pool runs with minimum_unwrap_seconds and _fixed_ready measured tighter deadlines across all intermediate trials (each wiping egress-health.json while the 10 static lines were already cooling): cutting to zero retry windows (12 s maxResults=5 and 35 s maxResults=10 on re-sent query sets) ended failures in 2.8 to 3.0 s p50 (5.2 s and 26.9 s unconditional p90) but dropped delivery to 12/100 and 28/100 (rising to 17/20 in the final 20 maxResults=10 queries once proven residential exits stabilized), while retrying between windows (15 s / 35 s) recovered only 5/86 (5.8%) and 1/62 (1.6%) inside Google's 10 s query hold (19/100 and 39/100 overall, Q4 25/25 on maxResults=10; reverted), and an asymmetric 20 s (1-window) / 35 s (0-window) re-run on reused sticky ports delivered 35/100 (maxResults=5, 12.1 s unconditional p50, 13.8 s p90, 4.39 s delivered p50, 12.8 s failed p50) and 3/100 (maxResults=10 at 44.3 queries/min closed-loop retry flood). Sizing both budgets for one post-hold 10 s retry window plus unwrap reserve (20 s maxResults=5 via VITNA_SEARCH_SERP_GOOGLE_SHORT_SHARE_MS and 42 s maxResults=10 via VITNA_SEARCH_SERP_GOOGLE_SHARE_MS, disjoint sticky ports) delivered 69/100 on maxResults=10 (69.0%, 95% CI 59.4 to 77.2%; 67/100 first-try, 0/10 hot static ISP cleared in 10 tries, 67/90 = 74.4% residential first-try; Q1 1/25, Q2 18/25, Q3 25/25, Q4 25/25, or 60/60 first-try across deciles D5 to D10; 0 holds refused; unconditional all-100 latency 26.1 s p50, 27.6 s p75 vs 47.7 s at 58 s, 27.8 s p90 vs 52.9 s at 58 s; delivered-only 27.4 s p50, 28.0 s p90; failed-only 12.8 s p50, 14.4 s p90 vs ~53 s at 58 s) while the repeatedly re-sent maxResults=5 set delivered 7/100 (2.5 s proven spacing) and 2/100 (10 s proven spacing; failed-only 12.3 to 12.5 s p50). On previously unseen 100-query sets (bench_query(1000..1099) and bench_query(1100..1199)), maxResults=5 at 20 s delivered 48/100 overall (48.0%, 95% CI 38.5 to 57.7%; Q1 0/25 and Q2 5/25 during cold residential warmup, then Q3 21/25 = 84% and Q4 22/25 = 88%, or 43/50 = 86.0% first-try across the warmed second half; 0 /goto holds waited or refused; unconditional all-100 latency 11.7 s p50, 12.4 s p75 vs 28.2 s at 35 s, 13.6 s p90 vs 32.3 s at 35 s; delivered-only 4.49 s p50, 5.11 s p90; failed-only 12.3 s p50, 14.9 s p90) and maxResults=10 at 42 s (started 30 s later on a fresh cold pool) delivered 6/100 (6.0%, 95% CI 2.8 to 12.5%; Q1 0/25, Q2 0/25, Q3 1/25, Q4 5/25; 25.9 s delivered p50; 12.5 s failed p50, 15.5 s unconditional p90). Two follow-up unseen fresh-pool pairs with --retry-window-from-refusal 1 and --retry-window-after-refusals 1 first exposed probeBefore (ip_changed on idle sticky sessions) consuming --retry-window-tries 1 window turns before Google was contacted (bench_query(1200..1299) at 22 s: 28/100, Q1 0/25, Q2 0/25, Q3 12/25, Q4 16/25, 4.01 s delivered p50, 13.3 s failed p50, 22 serp_unreachable; bench_query(1300..1399) at 44 s: 0/100, 13.1 s failed p50, 42 serp_unreachable), and after moving window-turn accounting past probeBefore (bench_query(1400..1499) at 25 s and bench_query(1500..1599) at 46 s) reduced serp_unreachable to 0/100 in both runs while delivering 2/100 (D10 2/10, 4.34 s first-try, 12.9 s failed p50, 16.0 s unconditional p90) and 0/100 (13.6 s failed p50, 16.3 s unconditional p90), confirming that one-window budgets bound failed-query tail latency to ~12.3 to 13.6 s p50 (~13.6 to 16.3 s p90) with zero /goto holds refused, while cold-pool runs with egress-health.json wiped and all 10 static ISP lines externally hot (~1 to 2% fresh residential discovery rate per cold IP vs 64 to 100% once proven) bifurcate sharply by how early 2 to 3 clean residential exits are discovered.
A simultaneous paired head-to-head benchmark across two symmetric 13-slot cold pools (5 randomly split static ISP lines plus 8 disjoint sticky residential ports per daemon, egress-health.json empty on both, concurrency 3 per daemon, 200 previously unseen distinct queries bench_query(1600..1799) paired by topic and coin-flipped per pair so zero query strings overlapped across engines; even pairs maxResults=5 at 25 s, odd pairs maxResults=10 at 46 s; shared --goto-refresh-max-ms 30000 --goto-respect-num 1 --goto-pace-ms 1000 --fixed-retry-exits 1 --retry-window-ms 10000 --retry-window-from-refusal 1 --retry-window-after-refusals 2 --retry-window-tries 2 --window-proven-exits 1 --keep-proven-exits 1 --spare-gap-refusals 1 --proven-first-wait-ms 1000 --proven-interval-ms 2500 --fast-relaunch 1, plus --warm-google-session 1 on stock Chrome) isolated cold-pool exit discovery from hour-to-hour variance: Camoufox delivered 68/100 (68.0%, 95% Wilson CI 58.3 to 76.3%; 54/100 first-try, 4/15 static ISP and 50/85 = 58.8% residential first-try; 14/46 = 30.4% retry recovery across 190 tries and 40 window sleeps; quartiles Q1 14/25, Q2 14/25, Q3 21/25, Q4 19/25, or 40/50 = 80.0% with 95% CI 67.0 to 88.8% and 37/50 = 74.0% residential first-try across the warmed second half; 29 serp_blocked and 3 serp_timeout; unconditional all-100 latency 16.6 s p50, 26.1 s p75, 27.5 s p90; delivered-only 25.7 s p50, 28.0 s p90; failed-only 15.4 s p50, 17.9 s p90) against 0/100 (0.0%, 95% CI 0.0 to 3.7%; 0/6 static ISP and 0/94 residential first-try across 305 tries and 90 window sleeps; 99 serp_blocked and 1 serp_timeout; failed-only 14.3 s p50, 17.2 s p90) on stock Chrome (68 vs 0 discordant topic pairs, two-sided exact McNemar p = 6.8e-21). Within Camoufox's 100 queries, maxResults=5 (n = 50, 25 s) delivered 33/50 (66.0%, 95% CI 52.2 to 77.6%; 25/50 first-try at 4.85 s p50 and 5.60 s p90, 8/25 = 32.0% retry recovery; 0 /goto holds waited and 0 refused; unconditional latency 6.63 s p50, 16.5 s p75, 18.7 s p90; delivered-only 5.17 s p50, 18.6 s p90; failed-only 15.6 s p50, 19.7 s p90) and maxResults=10 (n = 50, 46 s) delivered 35/50 (70.0%, 95% CI 56.2 to 80.9%; 29/50 first-try at 26.3 s p50 and 27.8 s p90, 6/21 = 28.6% retry recovery; 280 /goto holds waited and 0 refused; unconditional latency 26.1 s p50, 26.8 s p75, 39.2 s p90; delivered-only 26.4 s p50, 39.8 s p90; failed-only 14.2 s p50, 16.7 s p90). Both engines exited with zero surviving processes or leftover directories (Camoufox 4.65 GB PSS across 13 slots, 625 ms fast-relaunch p50; stock Chrome 4.24 GB peak PSS across 13 slots, 306 ms fast-relaunch p50); camoufox remains the default engine because its cold residential discovery rate held up where stock Chrome's collapsed, while --browser-engine chrome stays opt-in.
Proxy traffic per capture
Residential exits are billed per GB. On 2026-10-08 a delivering attempt moved about 1.1 MB through its exit, 952 KB of it Google's script bundles (www.google.com/xjs/), OneGoogle bar (www.gstatic.com/og/) and fonts, all cacheable for a year and downloaded again by every fresh context. Organic results are in the HTML, so by default the daemon does not load those assets; BotGuard (/js/bg/) and everything else still load. A delivering attempt then moves 165 to 187 KB, and 23 such captures each delivered ten organic results. --load-static-assets restores them. Whatever static assets still load are kept in a cache shared by the slots (--static-cache-mb, default 96, 0 to disable): public, long-lived, cookie-free responses only, keyed by exact URL. A challenged attempt moves about 62 KB.
Alternative retrieval experiments
On 2026-10-05, an isolated Python environment on the owned OVH host ran Google-only SearXNG at revision `d48c4b555421e824342c51d68482dd0898e54d0f`, bound to loopback. Its /wml/search request used curl_cffi with a Nokia user agent. All three measured paths received Google HTTP 403; SearXNG returned API HTTP 200 with zero results and an access-denied report:
| Tested outbound path | Observed latency |
|---|---|
| Host default connection | About 421 ms |
| Fixed owned IPv6 | About 371 ms |
| Private SSH CONNECT relay through the PC's own network | About 645 ms; one relay connection transferred 8,641 bytes |
The host worked, but these samples did not establish unchallenged Google retrieval. They used no third-party search service, Custom Search fallback, or production provider integration.
4get's pinned Google implementation and configuration guide at revision 03ba5d7b5ed3dc91b3e7d8b6278d8f52818a99ae require a separate 4play Firefox service for Google; the 4get Docker container alone is insufficient. This alternative was reviewed, not installed or live-tested.
Whoogle's official end-of-life announcement dated July 24, 2026 says it no longer returns results: Google blocked its last working non-JavaScript user agent, and the Custom Search fallback was unworkable. The repository was archived on GitHub on August 14, 2026. Whoogle was reviewed only, not installed or live-tested.
The reviewed GitHub topic entries largely required paid-provider accounts. An independent candidate, omkarcloud/google-scraper at revision `11a5562251a8c330475a1fd312e5f572db85d7e8`, uses a Camoufox Firefox session to obtain cookie identity, then replays requests with curl_cffi. Its optional paid solver and proxy services were excluded from this investigation.
An earlier separate private experiment installed Camoufox 0.5.7, Playwright 1.62, and browser build 156 beta 34 on the owned OVH host, with verified cgroup process cleanup. A fresh direct headless capture for site:bis.gov H200 January 2026 license review policy returned HTTP 200 without a detected challenge in 36.237 seconds, including browser startup. Four BIS organic results were verified with the existing parseSerpHtml parser, canonical search identity, and single-site guard.
After an admission gap, a separate direct Python SSL diagnostic made header-only GETs to the four observed opaque Google /goto links. All returned HTTP 302 destinations on www.bis.gov or media.bis.gov, with 423 ms of actual network time. It followed no publisher destination and closed every connection before recording proof. This step used no Firefox cookies, user-agent spoofing, or third-party provider.
These were separate admitted diagnostics, not an end-to-end API latency measurement or production integration, and do not establish sustained reliability. The browser diagnostic checks identity only at the final search URL and accounts for bytes after response bodies complete; these limitations apply only to the private experiment.
Configure proxy routes
Existing URL strings retain their fixed credentials. A structured entry accepts url, optional egressId, and optional session with username and/or password credential templates. For example, this placeholder configuration requires the operator's provider to support the illustrated username syntax:
[
{
"url": "http://proxy.operator.example:8000",
"egressId": "shared-exit",
"session": {
"username": "ACCOUNT-session-{session}",
"password": "OPERATOR_PASSWORD"
}
}
]Store the array in private VITNA_SEARCH_SERP_PROXIES. Credential templates are literal, unencoded strings; Vitna encodes the resulting URL credentials. At least one supplied template must contain {session}; other placeholders are refused. Each admitted attempt generates one cryptographically random 32-character lowercase hexadecimal token and substitutes that same token everywhere {session} occurs. That attempt uses one generated proxy URL throughout its isolated browser context. A later attempt generates a new token. A supplied template replaces its entire credential field; fields without a template keep the URL's static credential. Template syntax and the provider's session stickiness remain operator responsibilities.
egressId is an operator declaration that routes share a physical exit. It accepts 1 to 32 lowercase letters, digits, and hyphens, starting with a letter. direct, browser-direct, and proxy-N names are reserved. Different proxy URLs with the same label share per-engine pacing, concurrency, and block cooldown in the service process. Declare the shared label when different endpoints use the same exit; separate route labels do not establish separate physical exits. Replicas still have no shared coordinator.
Routes receive safe proxy-1 through proxy-20 identifiers in array order. Duplicate detection compares the endpoint and effective fixed credentials or templates. Changing a credential that its template overrides, or assigning another egress label, does not make the same effective route distinct. Every attempt uses an isolated context without ordinary browser state, and a configured proxy pool has no implicit local-egress fallback.
Validate an operator route
Run the explicit live checker after enabling SERP and configuring the intended runtime and egress routes:
node --env-file=.env.search platform/search/serp-check.mjs --engines google,bing --rounds 1 --timeout-ms 10000This command performs actual engine HTTPS/browser requests. It is separate from startup, readiness, and the local fixture integration test. By default it checks both engines once, using three fixed public queries per engine: six fresh JSON retrieval requests. Each request still has its configured route attempt budget. --engines can select google, bing, or both; --rounds accepts 1 to 3, bounding the run to at most eighteen samples. --timeout-ms accepts 100 to 60,000 and is clamped to the configured SERP timeout. Omitting it uses that configured timeout.
Before another fresh sample, the checker respects the observed nextRequestAt and can wait up to sixty seconds for pacing. A sample refused with serp_routes_busy or serp_routes_throttled started no capture; the checker waits that refusal's interval and retries the same sample, within sixty seconds of such waits and sixty retries per sample. A refusal needing a longer wait remains a failure. These client-side waits are separate from the sample's latency measurement, and each retry has its own per-probe deadline. The checker does not sleep through cooldowns or retry failed samples; those outcomes remain failures in its report.
The command writes a JSON report to stdout with ok, samples, delivered, successRate, latency percentiles p50Ms and p95Ms, actual route-attempt count attempts, safe per-route counts in routeAttempts, the configured transport, and numeric route count configuredRoutes. Its engines entries report those sample/delivery/latency counts and failure-code totals per engine. successRate is a fraction from 0 to 1. samples counts logical retrieval requests; attempts includes the actual backend attempts and retries. Queries, search URLs, result content, HTML, credentials, and proxy addresses are omitted. Save stdout to .search-data/serp-check.json if an operational record is useful. Configuration or invocation failures return a safe ok: false, error report. Exit code 0 means every requested response was delivered; exit code 1 means a request or the setup failed. A successful bounded run establishes outcomes for its sampled requests and configured routes, without establishing whole-engine coverage or future availability.
Diagnose a browser route
The optional standalone browser diagnostic submits one fixed public Google query through the current connection, using an isolated browser context. Choose one configuration for each invocation:
node platform/search/serp-browser-check.mjs --channel=msedge
node platform/search/serp-browser-check.mjs --channel=msedge --headed --navigation=formThe default is the native profile, headless mode, direct search navigation, and a 12,000 ms timeout. --channel=msedge or --channel=chrome selects an installed browser; otherwise the configured SERP or Unblocker channel is used, then bundled Chromium. --headed opens an automated browser window. --navigation=direct opens the search URL; --navigation=form opens the Google homepage and submits its visible search form. Both use one deadline. --timeout-ms accepts 1,000 to 20,000, using the equals form, such as --timeout-ms=20000.
--profile=cloakbrowser uses an already installed runtime selected by VITNA_SEARCH_SERP_BROWSER_PATH, or its Unblocker path fallback. The command does not install a browser. It refuses a nonempty VITNA_SEARCH_SERP_PROXIES pool; use the proxy route diagnostic below for that comparison, or the retrieval checker above to validate production route selection. This diagnostic makes no automatic retry and runs no challenge solver.
Every configuration shares persisted Google/current-connection health in .search-data/serp-browser-check-state.json, protected by an exclusive .lock file. --state=PATH can select another state file; use the same path when comparing profiles, headed mode, or navigation. A cooling state refuses the probe before browser launch. The first observed block starts a five-minute cooldown; repeated blocks double it up to one hour. A successful capture resets the block streak. Changing browser settings does not bypass this shared cooldown, and a concurrent diagnostic is refused while its lease is held.
Healthy diagnostic starts durably reserve an interval selected uniformly between five and fifteen seconds before capture. The state records nextRequestAt across CLI invocations, browser profiles, headed or form modes, and generated session tokens. An early repeated probe returns serp_routes_throttled with zero attempts; a cooling route returns serp_routes_cooling. Neither diagnostic waits for pacing or automatically retries. State version 2 adds this pacing record; version 1 files migrate automatically while preserving their block history and cooldowns. The file guard coordinates these diagnostics on its local filesystem. Ordinary production routes use separate process-local coordination; the experimental owned adapter can share this durable gate when its state path and declared egress identity match.
A crash or a failed state save can leave the lock in place. Repair the state storage and confirm that the diagnostic has ended before clearing an abandoned lock. A failed save returns serp_probe_state_invalid and prevents another invocation from trusting an older cooldown record.
The command writes a JSON report to stdout with ok, engine, profile, headed, navigation, attempts, and safe observed stages. A failure includes a fixed error code. latencyMs is included after an attempted capture; count is included only on success. Available state is reported as numeric consecutiveBlocks, cooldownUntil, nextRequestAt, and retryAfterMs. Queries, URLs, result text, HTML, browser paths, and credentials are omitted. Exit code 0 indicates a parsed capture; exit code 1 indicates refusal or failure.
navigationResponses retains at most eight main-frame HTTP observations, with only a fixed destination category, status code, and challenge flag. It records challenge responses before an immediate abort closes the page. When a page was opened, network reports request lifecycle counts and fixed method, resource-type, and failure categories. Each count is capped at 1,024; truncated marks overflow. These observations help identify failed POSTs or redirects without retaining request URLs, bodies, cookies, or headers.
A comparison also requires the operator to make the same search manually in an ordinary unsigned-in browser on the same connection. An automated headed run is not that manual baseline. Automated failures alone do not establish that the connection's IP address is blocked.
Compare configured proxy routes
The proxy route diagnostic uses the same bounded browser probe on the configured proxy list:
node --env-file=.env.search platform/search/serp-route-check.mjs --channel=msedge --navigation=formBy default it submits one fixed public Google query per configured proxy, up to twenty routes. It performs the probes sequentially and waits for each browser to close before the next route. It does not retry a failure or wait through pacing or a cooldown. With no configured proxies, it returns serp_proxy_not_configured before launching a browser. It does not add an implicit current-connection probe.
An explicit selection can include that baseline:
node --env-file=.env.search platform/search/serp-route-check.mjs --routes=direct,proxy-1 --channel=msedge --navigation=form--routes accepts up to twenty-one unique labels: direct and the configured proxy-1 through proxy-20 routes. The whole selection is validated before traffic; an unconfigured label is refused. direct here means the isolated browser on the current connection. The standalone serp-browser-check.mjs CLI remains current-connection-only.
The route checker accepts the same --profile, --headed, --navigation, --channel, --timeout-ms, and --state browser options described above. Its defaults are headless, direct navigation, and a 12,000 ms deadline per probe; profile and channel inherit configured SERP or Unblocker settings, with the native profile as the fallback. Only already installed runtimes and operator-supplied proxy credentials are used.
Both browser diagnostics use the same state file and exclusive lock. Persisted proxy health uses opaque identities derived from the effective configuration. An explicit shared egress label gives aliases the same key, including across browser profiles, navigation modes, and credential rotation. That key shares both persisted pacing and block cooldown, so a new session token does not bypass either guard. The current-connection google:direct ledger remains shared with the local browser diagnostic. Keep the same --state path for comparisons; paced or cooling routes are refused before launch with zero attempts. Shared aliases can therefore be throttled within one sequential checker run. The same five-minute first block penalty and one-hour cap apply. These local diagnostic ledgers do not coordinate ordinary production routes or distributed replicas. The experimental owned adapter can share their durable gate on the same filesystem and declared egress identity. Neither identity scheme measures the provider's actual exit IP.
Stdout contains a sanitized JSON aggregate with ok, samples, delivered, and routes. Each route entry has its safe route label and browser probe outcome, stages, latency, result count on success, and available numeric pacing and cooldown counters, including optional nextRequestAt. Queries, URLs, credentials, credential templates, session tokens, and persisted identity hashes are omitted. Configuration failures return a fixed safe error code before traffic. Exit code 0 means every selected route delivered a parsed capture; exit code 1 means a refusal or failure. A successful sample does not establish Google coverage, physical exit identity, or provider session stickiness.
Browser container
Build the dedicated SERP image from the repository root:
docker build -f deploy/vitna-search-serp.Dockerfile -t vitna-search-serp .This image includes the isolated browser runtime and selects auto SERP transport with browser fallback. SERP retrieval stays off until the operator sets VITNA_SEARCH_SERP_ENABLED=true and VITNA_SERP_RESALE_APPROVED=true, as described in Run without an index. A native index is optional. Supply those settings, the private search client registry and any operator proxy configuration at runtime, and use a Linux host or container runtime that permits the Chromium sandbox and user namespaces. The image's readiness probe checks service configuration; it does not establish successful access to a search engine. GET /readyz follows the default /v1/search route: when a native index is the default provider, it returns HTTP 503 while that index is empty or unavailable, even with SERP retrieval enabled. A SERP-only instance is ready while its retrieval adapter reports itself available. Authenticated GET /v1/status still reports SERP availability if the native index status read fails, with index: {ready: false, unavailable: true}.
The existing deploy/vitna-search.Dockerfile also supports direct SERP retrieval without browser dependencies. Its defaults assume a native index at /data/index.sqlite and VITNA_SEARCH_PROVIDERS=vitna. For SERP-only operation, override the index to an empty value, replace that provider order with brave,exa,tavily without supplying gateway credentials, and set VITNA_SEARCH_SERP_ENABLED=true, VITNA_SERP_RESALE_APPROVED=true and VITNA_SEARCH_SERP_TRANSPORT=direct. Unconfigured legacy providers are skipped; SERP retrieval remains a separate capability.
Direct egress may encounter consent, rate limits, or challenges. Browser transport and optional operator proxies add another retrieval route but do not guarantee success, verified geographic targeting, or broad production coverage. This implementation does not yet provide a full substitute for a vendor's distributed retrieval infrastructure. Local live Google browser testing has returned serp_blocked; verify access on the intended operator route before relying on it.
Authentication, quotas, and failures
Single-page SERP requests use the same search client registry, per-client minute/day quotas, service concurrency limit, and whole-request deadline as indexed search, extraction, and research. An admitted SERP failure consumes one quota slot, including route-busy, cooling, or throttled responses. Invalid requests and the shared API's service_busy refusal consume no quota. Receipts record operation, request identifier, outcome, and latency, without the query, search URL, fetched HTML, snippets, authorization header, or proxy secrets.
Responses use Cache-Control: no-store. Failures use the shared {"error":{"code":"...","message":"..."}} envelope. Invalid input returns HTTP 400; authentication uses HTTP 401, and shared rate limits use HTTP 429. Retrieval and parsing failures return a safe explicit error, without forwarding a challenge page or an upstream error body as a result.
Every retrieval failure, a missed deadline included, returns HTTP 503 and is told apart by its code. A proxy in front of a hosted instance can replace the body of a 502 or 504 with its own page, which would hide the code; a 503 keeps it. Before 2026-10-22 these failures returned 502, and serp_timeout 504. A failed request's receipt records the same code as its outcome, so an org's console activity and the service's journal tell failures apart too.
| Status and code | Meaning |
|---|---|
503 serp_unavailable | SERP retrieval is not enabled on this instance |
503 serp_jobs_unavailable | Multi-page jobs or their private storage are unavailable |
429 serp_jobs_busy | Job capacity is full; includes Retry-After |
503 serp_runtime_unavailable | A required browser runtime could not be loaded or started; refusals while its route is paused include Retry-After |
503 serp_routes_busy | Eligible retrieval routes are at their concurrency limit (a route that rotates its own exits first waits for a free slot while 8 s of the deadline remain); includes Retry-After |
503 serp_routes_cooling | Eligible retrieval routes are cooling after failures; includes Retry-After |
503 serp_routes_throttled | Independent retrieval starts are paced; includes Retry-After and no new attempts |
503 serp_blocked | Challenge, consent or sign-in page, or access block |
503 serp_parse_failed | Page not recognized by the organic-result parser |
503 serp_unreachable | Search engine could not be reached |
503 serp_response_too_large | Fetched HTML exceeds the 2 MiB limit |
503 serp_invalid_response | Upstream response is not supported nonempty HTML, or a browser challenge flag without HTTP 403 or 429 |
503 serp_redirect_blocked | Redirect outside the supported search route or past the redirect limit |
503 serp_query_mismatch | Final search URL changed the requested query or offset, or results violated an unambiguous single positive site: hostname constraint |
503 serp_timeout | Search engine did not finish within the request deadline |
499 request_cancelled | Caller cancelled retrieval |
The site: result guard permits the exact hostname and its subdomains. It skips complex, quoted, multiple-site, or invalid-DNS syntax; it does not interpret arbitrary search operators.
Indexed search and cited research
POST /v1/search with mode: "index" searches Vitna's stored corpus. POST /v1/extract returns stored indexed page text. POST /v1/research assembles cited passages from indexed evidence. These endpoints remain available when the operator configures their independent index and crawler.
Research source selection uses bounded candidate metadata before reading, then rechecks the extracted page's URL and title before passage selection. A small registry recognizes PostgreSQL, Python, and Node.js ownership and canonical documentation version paths. When the query identifies one requested version, explicit incompatible versions in those paths or subject-specific titles are excluded; unknown or mixed version evidence stays eligible. Snippet numbers, URL query parameters, and arbitrary numeric paths do not establish a version. Cross-product or multiple-version queries do not infer one requested version. Documentation queries prefer recognized documentation metadata, known primary domains, then compatible version evidence, with stable input-order ties. coverage.selectionFilteredSources counts selection exclusions. This policy does not verify content facts or establish comprehensive coverage.
Private controlled replays on 2026-10-05 checked these changes without new Google captures. Replaying stored HTML and exact redirect mappings produced 61 result identities across seven pages from 64 original rows and 62 legacy URL identities. It retained the first tracked YouTube listing and its original source, excluding the later bare-link variant; distinct identities do not establish distinct content.
For the PostgreSQL 18 selection replay, known primary-domain sources among the top five increased from four to five, and the canonical 9.0 manual was excluded. Three separately guarded direct publisher reads returned HTTP 200 and passed the URL/title version checks. A native research replay over those real pages returned four exact Unicode-offset citations covering 685 quoted characters from the 18.0 release notes. It remained partial and non-exhaustive, using one source after three attempts and two skips. Saved raw Google results were unchanged, and the replay did not mutate the indexed corpus or API results. These are bounded regression observations, not an independent quality benchmark or proof of broader factual accuracy or coverage.
SERP retrieval does not enqueue native discovery, add organic links to the index, fetch result pages, or produce cited research. A SERP snippet alone is not the indexed source text needed by the research API. See the search overview for those separate contracts and the migration guide for former Live/Hybrid callers.
