Find out what is blocking your requests - the proxy, the browser, the host, or the target itself.
Built and maintained by NodeMaven, who sell
proxies. Every number here is generated from the run files in data/runs/,
which are committed, so a reader can recompute any of them rather than take
them.
A failed request tells you that something failed. It does not tell you what.
The target may be refusing the exit address. It may be reading the browser. The handshake may be the discriminator. The machine you are running on may be the variable. Or the proxy may be failing before the request reaches the target at all.
proxy-benchmark separates those and measures them in one time window. It runs
the same targets across controlled combinations of:
- engines - plain HTTP clients, stock Chromium, patched browsers, CDP drivers and anti-detect frameworks;
- proxy paths - gateways, providers, countries, sticky sessions, and a direct arm with no proxy at all;
- browser configuration - resource blocking, headful, timezone and locale alignment, humanized input where the engine has it;
- hosts - when the machine itself turns out to be part of the experiment, which here it did.
Every attempt is one JSONL row under data/runs/ carrying the full parameter
set, and the tables below are generated from those rows rather than typed in.
Bring your own proxy. No NodeMaven account is needed. A gateway is a .toml
file and an engine is a module plus one registry line, so neither ever edits the
runner - point it at a proxy you already own.
The usual debugging loop is: request failed, so the proxy must be bad, so change the proxy, so it failed again, so the browser must be detected. None of those steps is justified by a single request.
This asks the question with a control instead:
The output that matters is not "this request passed". It is: this variable changed, these did not, and the outcome moved with it.
Three of the answers that came back, each with its denominator and its run file:
- The unmodified browser finished in the leading group on Amazon. Stock Chromium 96% (419/436) against 63% (288/457) for the lowest anti-detect engine, over 3530 judged attempts. Six engines sit within four points at the top, no test separates them, and the control is one of the six. Full table
- The same code, gateway and target scored 39% on one machine and 0% on another. 24/61 from a Windows workstation against 0/84 from a Linux VPS in overlapping hours, Fisher p = 3.7e-11. The client machine is a variable a proxy comparison usually holds fixed without saying so. The split
- Chrome spends 43 MB per fresh profile talking to Google before you ask it
for anything. 43.2 MB of a 43.4 MB idle window, on a browser parked on
about:blank. On a metered residential exit that is billed traffic for a file no target ever sees. How it was counted
The rest of them, including the five that replaced an earlier conclusion of ours.
All 22 sections are here rather than the popular ones, because the sections a
reader most needs are usually the ones a hand-picked list leaves out. Every
anchor above is checked by test_every_link_inside_the_repository_resolves,
which exists because an earlier index in this file spent a day pointing at a
heading that had been deleted.
A real measurement, no proxy account, no browser download. Three packages, and 31 s to run on a fresh clone measured 2026-08-27.
git clone https://github.com/nodemaven/proxy-benchmark && cd proxy-benchmark
python -m venv .venv
.venv\Scripts\Activate.ps1 # macOS, Linux: . .venv/bin/activate
pip install -r requirements-core.txt
python scripts/benchmark.py --engines http --targets ddg_serp \
--queries 5 --direct --preset none
It prints the plan and what it will cost before it sends anything, then one line per attempt and a summary:
engine target exit n pass verdicts
http-direct ddg_serp direct 5 100% {'ok': 5}
The first run is deliberately the bare HTTP client with no proxy. There is no browser to download, no proxy account to configure, and nothing to spend, and it establishes that the harness works before a second variable is introduced.
Every attempt is also a JSONL row under data/runs/, which is the only thing
this repository treats as evidence.
From here: the same run through a proxy, which needs no account with anybody; real browsers instead of a bare client; or check a number here rather than take it.
This file is the map. The depth is deliberately split by purpose, so nothing has to be read before the first run:
Want the numbers - RESULTS.md. Want to know how they were arrived at, including
the ones we got wrong first - NOTEBOOK.md. Never used a terminal -
docs/quickstart.md assumes nothing; this file assumes you know what a
ClientHello is.
Three layers can reject a request, and debugging fails when you inspect the one above whichever is actually refusing you. That is the whole reason the axes are separate.
A different handshake does not by itself explain a different pass rate. Measure the two together - here they did not agree.
The handshake row has three entries because there are three routes to a JA4 and they answer different questions. Pick by what you are asking:
Only the first is per-attempt; the other two are per-engine and are the ones to
re-run after a browser upgrade. tls-echo is the only route to obscura, which
refuses to navigate to localhost at all, and the only route to the HTTP/2
SETTINGS hash and header order, which a listener that answers nothing cannot
provoke. Cross-checked 2026-09-02 on this host: the eight engines both probes
reach agree character for character.
A run is a matrix, and every axis is independent: target, engine, provider, country, gateway parameters, resource blocking, headful, geo alignment, entry shape, warm-up rung. They do not nest inside one another, and that is the point - any one can be varied while the rest are held, and all of them are on every row.
Two properties do most of the work:
- Cells interleave at batch granularity, never run in sequence. Finishing one engine before starting the next would measure the afternoon as well. The hour is the largest confound this repository has found: the same gateway, country and browser moved 69 points to 52 between two windows of one afternoon.
- A matrix that could not be controlled is refused, not quietly adjusted. If
an option applies to some engines and not others, the run would compare a
humanized Camoufox against an unhumanized everything else and read it as an
engine difference. Every engine declares what it supports and --dry-runrefuses the matrix before it starts.
11 frameworks, one registry line each. Anything missing from the machine reports itself unavailable and names the install command, and the rest of the matrix still runs - --dry-run prints that list.
Any of them takes a :direct suffix, which runs that engine around the gateway inside the same matrix, so the proxy and the no-proxy arm are measured in one window rather than an hour apart.
Six targets, chosen because they fail differently rather than because they are
popular: google_serp, bing_serp, ddg_serp, amazon_search,
walmart_search, and ipinfo - which is not a target but an echo service, used
to prove the path works before anything is concluded from a refusal.
Every attempt writes one JSONL row: engine and engine version, target, provider
and gateway parameters, country, preset, headful, geo, entry shape, warm-up rung,
session and query, the machine it ran on, the verdict and the marker counts
behind it, the failure reason, timing, and bytes. ROW_FIELDS in
nmbench/engines/base.py is the schema. Those files are the source of truth, and
the tables in this README and in RESULTS.md are generated from them - a number
nobody has to remember to update is a number that cannot drift.
Verdicts come from page content, not HTTP status. The same Google reCAPTCHA
page arrived once as 429 and once as 200, so a run judged by status
scores the second as a success. There is no boolean success column:
The last two are the ones that decide whether a benchmark measures anything:
- emptyis not- block. Google hands a scriptless client a 92 KB "enable JavaScript" scaffold that stays on- /searchand rejects nothing, so scoring it as a block credits Google with a refusal it never made. 14 of 14 such rows carried- enablejsand none carried- recaptcha.
- erroris ours. A timed-out selector, a browser that would not launch and a query that never reached the box produced no evidence, so they produce no verdict. A harness that counts its own crashes as target refusals can manufacture a very convincing result while measuring almost nothing.
Seven more columns are easy to misread:
-
hostwas added on 2026-09-02, so an absent one means "nobody wrote it down", not "the machine was unknown". Every row before that date is attributed to a machine by its timestamp, which works only because the two machines here happened to run at different times - host and date are one variable under two names in every table built from those rows. This matters more than a provenance column usually would: the largest unexplained result in this repository is a difference between two computers, 39% (24/61) against 0% (0/84) at p = 3.7e-11 on the same target, engine, entry shape and gateway parameters, and nothing on disk could name which computer.host_osandhost_cpussit beside it because a label groups rows and does not explain them. SetNMBENCH_HOSTto the machine's name in the notes; unset, the column holds a hash of the hostname, because these files are public and a hostname names somebody's infrastructure.
-
tls_ja4is empty on most rows for a structural reason, not a missing one. It is the engine's JA4, read off the ClientHello as it passes through the local CONNECT relay, so only the engines that need that relay have one -zendriver,seleniumbaseandbotasaurus. The Playwright-driven engines take proxy credentials directly and never send a handshake through this process, so their rows carrynulland therelayedcolumn beside it says why. Measured 2026-09-02 overdata/runs/, that is 3815 of 16579 attempt rows. The fingerprint is computed innmbench/tlsfp.pyrather than asked of an echo service, and it was checked against one: on the same client,tls.peet.wsand this repository agree character for character.Every engine can still be fingerprinted, off the run, with python scripts/probes/tls_clienthello.py. It points each engine in turn at a listener on this machine that answers nothing, so it needs no live host and spends no traffic, and it reaches the Playwright-driven engines the relay route cannot.
-
A JA4 is a property of the browser build at least as much as of the engine. Measured 2026-09-02 with the chromiumengine and nothing varied but the binary: Playwright's bundled Chromium 151.0.7922.34 givest13d1516h2_8daaf6152771_806a8c22fdea, and the installed Chrome 149.0.7827.201, reached with--channel chrome, givest13d1516h2_8daaf6152771_d8a2da3f94cd. The extension lists are byte for byte the same and so is the cipher hash; the whole difference is three signature algorithms,0904,0905,0906, that the newer build offers and the older one does not. That is why the same probe putschromiumandpatchrightin one group andzendriver,seleniumbase,botasaurus,cloakandcurl_cffiin another - it is the Chrome version each happened to launch, not anything the libraries do differently. Readengine_versionbeside this column before treating a split as a property of the engine.
-
A refused address diverts the request, and the status does not say so. Google answers a refusal by sending the request to /sorry/, with a 200 about a quarter of the time.report.was_served- a 200 whose final URL keeps the host and path asked for - is the test, and it is a property of the exchange, so nothing has to know a target's name.
-
A batch is one session, and a session is the unit. Ten queries through one browser is one identity doing ten searches; ten browsers doing one query each is a different experiment. The claim has been false once - until 2026-08-11 Camoufox opened a fresh context per query and discarded its cookie jar while every other engine carried one - and session-continuityis the offline probe that caught it.
-
bytesis two measurements andrelayedsays which. Playwright engines count throughpage.routeand see page resources; the relay counts sockets and sees request headers and TLS overhead as well. Never pool them. The relay figure is what a provider bills, and it adds a loopback hop, soelapsed_msis not comparable acrossrelayed.
-
The matrix carries an unmodified control. chromiumis Playwright's Chromium with no arguments, no user agent override and no patches;navigator.webdriveristrueand stays that way, because without it a pass rate cannot be told apart from the target letting everything through.tests/test_engines.pyreads the source ofChromiumEngine.openand fails ifargs=oruser_agentappear.
Response bodies are kept, gzipped. A verdict is one word about 92 KB of
markup, and the question that decides a report is usually asked after the run.
Non-ok bodies plus a sample of the passes, controlled by --no-bodies and
--sample-ok. The archive is gitignored, unlike data/runs/, because exit
addresses appear in embedded links. Re-reading 250 stored Amazon bodies offline
found an Akamai interstitial and an AWS WAF challenge filed as refusals, and moved
21 historical rows at no traffic cost.
What the current evidence supports, engine by engine and target by target, from the 10432 attempt rows in data/runs/benchmark_*.jsonl. pass is ok over judged attempts - harness and path failures are counted separately and excluded from the denominator, because an engine that crashes is not an engine the target refused.
On the one target with enough evidence to rank engines, the top of the table is a tie and not a podium: chromium, rebrowser, botasaurus, camoufox, zendriver, seleniumbase sit within 4 points of each other and a two-sided Fisher exact, corrected for the 7 comparisons made, separates none of them. The first of them is chromium, which is the unmodified control.
Amazon and the two smaller search engines are a win. The Google row is not an engine comparison and must not be quoted as one. Every cell of it was taken on one Linux VPS. Run again with the same engine through the same gateway, a Windows workstation was served 39% (24/61) against 0% (0/84) from the VPS, two-sided Fisher p = 3.7e-11; cut to the one window where both machines were running at once it is 36% (8/22) against 0% (0/10), p = 0.035. The floor is real, it belongs to that client, and it is not a property of the proxies.
Full tables -> RESULTS.md - the 130-hour run (benchmark_20260819T055927Z, 2026-08-19 06:00 to 2026-08-24 16:12 UTC) engine by engine, Google day by day, and everything measured before it, split by host and by path.
Three row counts appear on this page and they count different things. The
badge counts every line in the committed data/runs/*.jsonl - matrix runs,
probe-and-hold windows, fingerprints, gateway checks. The table above counts only
matrix attempts that carried a query, which is benchmark_*.jsonl minus the
plan and summary lines. The rows column is per target within that. Each is
generated from the files and a test fails when the badge and the files disagree.
Everything below was measured with this harness between 10 August and 1 September 2026, and none of it is a standing fact about the internet: a target's defences move, so a rate measured in that window is evidence about that window. The dates sit here once rather than on each line, because a reader deciding whether to trust one of these needs the run id and the denominator, and those live in the section each line links to along with the date it stopped being true.
- Chrome spends 43 MB per fresh profile talking to Google before you ask it
for anything. 43.2 MB of a 43.4 MB idle window on
optimizationguide-pa.googleapis.com, on a browser parked onabout:blank. At one profile per attempt that is about 43 GB per thousand attempts, billed as residential traffic, for a file no target ever sees. How it was counted
- On Amazon the unmodified browser finished in the leading group. Stock Chromium 96% (419/436) against 63% (288/457) for the lowest anti-detect engine, over 3530 judged Amazon attempts in one 7630-row run. Six engines sit within four points at the top and no test separates them, so the top of that table is a tie rather than a ranking - and the control is inside it. Full table
- The same code, gateway and target scored 39% on one machine and 0% on another. 24/61 from a Windows workstation against 0/84 from a Linux VPS in overlapping hours, Fisher p = 3.7e-11. The client machine is a variable a proxy comparison usually holds fixed without saying so. The split
- On DuckDuckGo one substring in the User-Agent accounted for the whole
split. 95 of 95 pass for engines whose UA omits HeadlessChrome, 0 of 50 for the two that carry it, across three browser families and two drivers. Nothing else varied moved it, so a headless Chromium measured there is reporting its own UA rather than its exit. The split
- On Google the exit address dominated everything else we varied. Given a served page, pass was 83 of 83 and did not vary by country, while the chance of being served ran from 13% to 62% depending on the exit. The browser did not separate the cells, and no absolute rate here should be read as current. The decomposition
- The TLS handshake explains neither Google nor Amazon. Chromium, Patchright and Obscura emit a byte-identical ClientHello and their pass rates differ by 44 points. If you do compare, compare JA4 - Chrome shuffles extension order per connection, so a JA3 difference between two Chromium engines is noise. What was read
- None of the Chromium-driving engines changes its TLS fingerprint, and the
fingerprint tracks the Chrome version instead. Measured 2026-09-02 over nine
engines on one host: rebrowseron Chrome 136,cloakon 146 andseleniumbaseon 149 emit one identical JA4,chromiumandpatchrighton 151 emit a different one, and the split follows the browser version with nothing left over for the library. Their HTTP/2 fingerprint is identical too. So a JA4 being compared between these tools is a stock Chrome's, because underneath it there is one - and if a target is refusing you, the handshake is not what told it.curl_cffiis the exception that shows the rule: it is the only engine here that picks a fingerprint deliberately, and it picked Chrome's. The groups
- Timezone and locale alignment did not pay off in either arm we ran. Flat on Patchright (34% against 35%), and zendriver lost six sevenths of its yield, 57% down to 9%, p = 0.0008. Two engines is a thin basis for a rule, but nothing measured here argues for switching it on. Both arms
- Our own harness was getting the pool banned. One unauthenticated CONNECT per session, sent by the browser before anything else, was tripping an IP ban that looked like a gateway floor for days. How it was found
- One page of warm-up moved nothing in our window. 32% against 30%, intervals almost coincident. This refutes nobody: the protocol came to us from an operator, and the 75% that travels with it was mentioned in conversation as a figure once reached - not as a before-and-after pair, and with no denominator behind it. What our arm rules out is one page, which is what the ladder now goes past. The ladder
- Six pages of warm-up moved a great deal, and it held on three separate
days. Four rungs interleaved inside one run, because the hour is the largest
confound here: 11%, 24%, 33% and 82% at warm depths 0, 2, 4 and 7, over
35, 34, 33 and 33 judged attempts, cold against deepest z = 5.82. Two later
runs carried the cold rung and the deepest rung alone and read 24% against 86%
over 88 and 86 attempts, and 19% against 84% over 42 and 61. Pooled over the
three days, 20.0% (33/165) cold against 84.4% (152/180) at depth 7,
z = 12.0. Every attempt is Chrome 151.0.7922.34, headful, through Patchright,
on one host, and every figure is the probe phase judged as served against
challenged. What the depth is doing is not in these rows: four of the six
pages are Google's own, so "Google's infrastructure was told about this exit"
and "the browser lived through six navigations" both fit every row. The rung
that separates them holds the depth at six and swaps the four Google surfaces
for third-party pages carrying the same tags; it is declared in the target and
has no rows on disk, so what is published here is an effect without a
mechanism.
data/runs/probehold_20260831T222129Z.jsonl,data/runs/probehold_20260901T210934Z.jsonl,data/runs/probehold_20260904T000605Z.jsonl
Five of these eleven replaced an earlier claim of ours, and both versions are still in the notebook - Amazon, the warm-up, the Google levels, the idle traffic and the ban. The Amazon one reversed outright: on a workstation in early August, Camoufox was served 90% while every Chromium engine met the throttle, which read as a Firefox-against-Chromium result. On the server the unmodified control came out on top and the Firefox reading was gone. A number here is a reading of the hours it was taken in, and the ones that changed are labelled rather than quietly edited.
Python 3.11 or newer, run from a checkout. There is no [project] section to
install, because the committed query lists and data/ are part of the instrument.
python -m venv .venv
.venv\Scripts\Activate.ps1 # macOS, Linux: . .venv/bin/activate
pip install -r requirements-dev.txt
python -m playwright install chromium
python -m patchright install chromium
python -m rebrowser_playwright install chromium
python -c "import cloakbrowser; cloakbrowser.ensure_binary()"
camoufox fetch
copy .env.example .env # macOS, Linux: cp .env.example .env
Playwright, Patchright, rebrowser and cloakbrowser pin four different Chromium
builds and no two share a download, so a fresh machine fetches four browsers -
the build is what several findings here are about. zendriver, seleniumbase and
botasaurus download nothing and drive the host's installed Chrome, so a machine
without Chrome loses three engines and the rest carry a build nobody pins.
None of it is mandatory. An engine whose dependency is missing reports itself
unavailable and names the install command; the rest of the matrix runs.
--dry-run prints that list, so run it first on a new machine.
Headful on a headless host needs xvfb-run -a. --headful is the difference
between Chrome/... and HeadlessChrome/... on the wire, which is the whole of
the DuckDuckGo finding. A virtual display does not restore the GPU, so WebGL falls
back to software and a headful server run is not a headful workstation run.
Obscura is not on PyPI. Download the -stealth archive, unpack it, put the
directory on PATH. The plain archive is a different build and the stealth patches
are the thing being measured.
.env holds NODEMAVEN_LOGIN, NODEMAVEN_PASSWORD, NODEMAVEN_HOST and
NODEMAVEN_PORT. The prefix is the provider id, so oxylabs.toml reads
OXYLABS_LOGIN and two accounts sit in one .env - which is what a matrix
interleaving two providers needs. .env is gitignored, config.py is the only
reader, and it resolves on first use so everything else imports and tests on a
machine with no account.
Before spending anything:
make check # ruff plus the offline suite
python scripts/benchmark.py --dry-run
The suite is offline. A green suite is the precondition for spending traffic.
python -m nmbench lists every command, what it answers, and which ones spend
traffic. It is a dispatcher: the remaining flags go to the script untouched, and
every script still runs directly by path.
python -m nmbench # what exists, and what it costs
python -m nmbench benchmark --dry-run
python -m nmbench engine-fingerprint # offline, sends nothing
A first matrix, one engine against the unmodified control:
python scripts/benchmark.py --engines patchright,chromium \
--targets google_serp --queries 40 --batch 10 --headful
The :direct suffix puts the same browser on both sides of the gateway inside one
window; two runs an hour apart would measure the hour as well. A global --direct
forces every cell direct and cannot be partly undone by a spec that omitted the
suffix.
python scripts/benchmark.py --engines chromium,chromium:direct,camoufox \
--targets amazon_search --queries 100 --batch 10
Resume skips attempts already judged, so an interrupted run does not re-ask the targets:
python scripts/benchmark.py --resume data/runs/benchmark_<stamp>.jsonl
Then read what it said, and what it really cost:
python scripts/analysis/report.py
python scripts/analysis/calibrate.py
Arriving at /search?q= is one request carrying a query string, with no keystroke
behind it, no referrer and no form submission - a shape no person produces. It is
now an axis: entry is on every row, url for that shape and home for landing
on the front page and typing into the box.
python scripts/probes/probe_and_hold.py --engines patchright,zendriver \
--identities 20 --series 3
The protocol is an operator's: one sticky exit per session, type on the front
page, drop the address if the probe is refused, hold it for a series if the probe
is served. Read the result with scripts/analysis/held.py.
The operator's protocol above includes opening a page or two on the target before asking it anything, and a figure of 75% travels with it. Measured here, one page moved 32% to 30%.
Be careful what that is being compared against, because this document was not for two days. The 75% reached us in conversation, as a number an operator had once seen on their own pool, country mix and hour. It was never stated as a before-and-after pair, so there is no 20%-to-75% effect to fail to replicate and no claim of anyone's to refute. What this arm has is its own denominator, and that is all it has.
That result has two readings and one arm cannot tell them apart: either warming
does nothing, or one page is not warming. --warm is a ladder rather than a
switch so the second reading gets a denominator.
N1 and N3 are controls on composition, not steps in depth. Each matches
the delivered page count of the chain rung it answers and shares none of that
rung's target-owned pages, which is why they are excluded from the cumulativeness
rule below - being a superset of the rung they control is the one thing they must
not be. Neither is a Google-free arm and the output should not be read as one:
entry=home navigates to the front page before it can type, so every arm
contacts the target immediately before the probe.
Three things make the gaps readable rather than decorative:
-
The rungs are cumulative and each ends on the same page. L3is a strict superset ofL2, which is a strict superset ofL1, and all three finish on the pageL1visits before the front page. So whateverL1buys is held while the rungs above it vary,warm_depthis an ordering, and a difference between two rungs is a difference in what was added rather than in two unrelated sequences. A test enforces this rather than a comment asking for it.
-
All rungs interleave in one process. The hour is the largest confound this repository has: the same gateway, country and browser moved 69 points to 52 between two windows of one afternoon, and the same target went 39% on one host and 0% on another in overlapping hours. Rungs run one after another would price the hour and call it depth.
-
The pages belong to the target, not to the probe. A probe that knew a domain would be a probe that could warm one target better than another. A rung a target has not declared is refused rather than answered with a shorter one, because a row labelled L3whose warm-up wasL1's is a wrong result and not an error - it looks exactly like the deeper warm-up not helping.amazon_searchdeclaresL1only: the rungs above it were designed against Google's refusal and nothing here says they transfer.python scripts/probes/probe_and_hold.py --targets google_serp
--warm off,L1,L2,L3 --identities 24 --series 5 --dwell 20,45
For a run nobody is going to watch, scripts/run_ladder.py wraps that one
command. It does not change the shape of the experiment - the rungs still
interleave inside a single process, because a supervisor that ran them in turn
would reintroduce the confound the interleaving exists to remove. What it adds
is a preflight that refuses a bad plan or a dead pool in seconds rather than at
hour three, a log per attempt under data/logs/, and a restart rule that is
deliberately narrow: an attempt is retried only if it died within ten minutes,
because a run that fell over on startup has lost nothing while one that fell
over at hour two is worth more than a second attempt at a different hour. Two
attempts are two run files, and the summary says not to pool them.
It also defaults to --engines patchright rather than to the registry default,
for a reason that is a measurement: through the pool at
google_serp, patchright answered 96 ok of 223 while botasaurus managed 1 of
87, seleniumbase 0 of 86 and camoufox 0 of 33. A ladder on an engine that cannot
reach the target compares four zeroes.
python scripts/run_ladder.py --identities 12
What this run cannot do, stated before it is run. At 24 identities per rung, a move from 39% to 60% is Fisher p ~ 0.25 - not a result. The ladder is a sieve on direction: it says which rung is worth 90 identities, and the confirming run is a separate one. Quoting a rung ordering off 24 apiece would be the same error as the four discordant pairs at p = 0.125 elsewhere in this repository.
Cost is dwell, and it is most of the run: at --dwell 20,45 the three warm rungs
average 65, 130 and 195 seconds per identity, so 24 identities is about 2.6 hours
of dwell before a single probe, hold or gap is counted.
What the ladder does not reach. Every rung is a sequence of navigations -
visit() is one goto, which is the one method every engine's page object has,
which is why warming needs no engine support. Clicking a link, clicking a result
and refining a query are a different shape of session and none of them is here.
If the ladder comes back flat, that is the next thing to build rather than a
conclusion that history does not matter.
Any proxy works - bought from anyone, or running on a box you own - and no account
with anybody is needed. Four values in .env:
CUSTOM_HOST=1.2.3.4
CUSTOM_PORT=8000
CUSTOM_LOGIN=your_login
CUSTOM_PASSWORD=your_password
Check it before spending anything on it. Ten CONNECTs, a few hundred bytes, nothing sent to any target. It is the only check that separates a wrong password from an unreachable host, because the gateway answers both with a status that names neither:
python -m nmbench gateway-health --provider custom
Then run whatever you like through it:
python scripts/benchmark.py --providers custom \
--engines http --targets bing_serp --queries 20 --preset none
data/providers/custom.toml is already written for the shape most proxies have:
one endpoint, a login, a password, no settings encoded in the username. Nothing to
transcribe, no code.
A gateway with no session parameter cannot be asked for a different exit, so every attempt leaves from one address. The runner prints this on the plan line:
If your provider does sell countries or sticky sessions inside the username, copy
_template.toml and write the dialect down there instead.
A provider is a username format: gateways take country, sticky session and quality
filter inside the proxy username, and every vendor picks its own separators and
names. So it is a file rather than a module - data cannot branch, and
tests/test_repository.py reads the runner's source and fails if it ever compares
against a provider name.
cp data/providers/_template.toml data/providers/oxylabs.toml
# fill in the dialect, then set OXYLABS_LOGIN and OXYLABS_PASSWORD in .env
python scripts/benchmark.py --providers nodemaven,oxylabs \
--engines patchright --targets google_serp --queries 40 --batch 1
--providers is an axis like every other one: cells interleave at batch
granularity, because provider A at 10:00 against provider B at 14:00 measures the
afternoon. The cell key names the provider only when the axis is varied, so runs
recorded before the axis existed still match --resume.
Every definition declares its provenance, and it is the first field to read.
status = "measured" means rows in data/runs/ came through that gateway;
status = "documented" means the dialect was transcribed from the vendor's
documentation and never sent a byte. --dry-run prints it.
That is load-bearing because a wrong username is invisible. The gateway
measured here answers an unrecognised parameter name with HTTP 200 and the setting
silently dropped, so the run completes and every row claims a setting that was
never applied. A name outside known_params is refused before a request exists,
and --param is validated against every provider in the matrix before the first
cell opens.
Only nodemaven.toml ships, and it is the only gateway any number here was
measured through.
An option only some engines implement is the failure this harness is built against: the run would compare a humanized Camoufox against an unhumanized everything else, and that reads as an engine difference. Every engine declares what it supports and the runner refuses a mixed matrix outright.
The table is a summary and the code is the authority: --dry-run refuses a matrix
before it starts, rather than leaving a reader to check a list that has rotted.
--humanize has three values and two of them are different clients, not two
settings of one. engine is the browser synthesising its own input - Camoufox
and cloak do this inside the binary and nothing outside can see how. trueman is
a pointer model in nmbench/pointer.py driven from outside through page.mouse,
so it works on any Playwright-driven engine including the unmodified control,
which is the point: a cursor axis measurable only on the two anti-detect engines
would confound the pointer with everything else those binaries change. They are
alternatives and never stacked - running both would compose two hands into one
path and produce a movement neither model describes - and camoufox and cloak
launch with their own humanization off under trueman.
trueman runs only in scripts/probes/probe_and_hold.py --entry home, and
both runners refuse it elsewhere rather than accepting it. A pointer exists only
where something is clicked: benchmark.py navigates to a search URL and clicks
nothing, and so does --entry url. Accepting the flag there would write
humanize_mode=trueman on rows whose cursor never moved, which is the failure
this whole section is built against.
Pass both arms at once - --humanize off,trueman - rather than running two
commands. It takes a comma list there the way --warm, --geo and --entry
do, and the arms interleave at identity granularity inside one window. It was a
single value until 2026-09-03, which meant the only way to get a control was to
run it again afterwards, and on this target the hour between two runs moves the
yield further than any flag in this table has: 69% to 52% between two windows of
one afternoon. A sequential pair would have measured that and called it the
cursor. The mode joins the cell key as /hand-off or /hand-trueman only when
more than one is asked for, so a run with a single mode still matches --resume
against every file taken before the axis existed.
Rows carry five columns for it. humanize_mode is the string, beside the older
boolean humanize which stays for the runs already on disk. pointer_ms is how
much of elapsed_ms was spent walking - a deliberate walk to a search box is on
the order of a second - so elapsed_ms - pointer_ms is the number comparable
against an unhumanized arm. pointer_device is which of the two fitted device
profiles the session drew, without which two rows of one arm are not comparable
on any timing metric. pointer_overruns against pointer_points says whether
the intervals the page saw were the model's or this host's driver's, which is
the headless question below in a column.
All four are null, not zero, when no pointer was driven. Zero is a different
statement and a reachable one: a walk of zero length emits no paced points, so
pointer_points = 0 means the cursor was already on the target.
Pair it with --headful, or half the model does not reach the page.
Measured 2026-09-03 on a local Chromium and a data: URL, four arms of 18 paced
points: headful the delivered interval median is 7.00-7.15 ms against a model
asking 7.11-7.22, with 1 overrun of 18; headless it is 16.65 ms with 14-16
overruns, because page.mouse.move() awaits a CDP reply that is frame-bound at
one 60 Hz frame with no window. The positions are the model's either way. The
intervals are the driver's when headless, and two of the detector's 19 metrics
are about intervals - so a headless trueman arm is a geometry experiment and
has to be reported as one. On a server --headful means xvfb-run -a, and
whether a virtual display gives the real frame clock is not yet measured.
Nothing here shows any target reads any of it. The model was fitted against
one person's captured traces and scored by a 19-metric detector in
lab/probes/trace_compare.py; that says it is hard to tell apart from that one
person, not that it changes a verdict. The axis exists to find out.
A mixed matrix needs --preset none. The default is light, and blocking for
some columns and not others measured 4 KB against 9.9 MB on the same Google
refusal page - a 2000x engine difference produced entirely by the
flag. It moves verdicts too: a page that never loads its script is judged on
markup that was never finished.
--countries needs no engine feature, because the host country is the
alignment - the browser reports this machine's timezone and language list whatever
address it leaves from:
python scripts/benchmark.py --engines camoufox,chromium,chromium:direct \
--countries ru,us --targets bing_serp --queries 20 --preset none
A direct cell has no country, so the axis collapses for it and it is built once.
--geo align hands the browser the exit's own timezone through the browser's
emulation rather than by patching a JavaScript property, which reads back
unpatched from an iframe and from a Web Worker. The unmodified control stays at
False: the axis is read within one engine, aligned against unaligned, in one
window. Whichever was used is on every row.
--chrome-binary holds the browser fixed across the six engines that can be
pointed at one, so the engine is the variable rather than the build it happens to
bundle. It is off by default and that is deliberate: every row already on disk was
measured with each engine on its own browser, and a silent default would make new
rows incomparable with the old ones without any column saying so. A pinned run is
labelled -pinned and engine_version carries the build that actually launched,
so the intent and the outcome are separate columns and can be checked against each
other.
The size of what it controls, measured 2026-09-02 by probes/tls_clienthello.py:
unpinned, the engines run Chrome majors 136 to 151 and their TLS fingerprints
split by major and not by library. Pinned to one Chrome, all six land on one
value, and three of them changed build to get there. See "The TLS handshake,
engine by engine" in RESULTS.md.
Each target draws from its own committed query list. A shop and a search
engine have to run in one window and cannot take the same strings: asked
"photosynthesis exam questions", Amazon answers with an empty shelf, which is
indistinguishable from a soft refusal once it is a verdict. --query-list forces
one list on everything when that is the question.
Everything above on one page, for reading rather than for learning from. The code
is the authority: python -m nmbench lists every command and marks the ones that
send nothing, and -h prints the flags with the reasoning attached.
Start here, by what you are trying to do.
Every flag of the matrix runner. Defaults are what you get for saying nothing, and two of them are worth knowing before a first run.
--countries defaults to us, and us is the worst setting measured - 13%
of US exits served against 58% on any. A first run on the defaults looks worse
than this pool actually is. The default is not a recommendation: country is part
of the cell key, so changing it would stop all 44 committed benchmark files from
matching --resume. Pass --countries any, or both if the country is the
question - us and any in one window is what turns the gap into a finding
rather than into a flattering number.
--batch is the other one. --batch 1 opens a fresh browser per query, so
every attempt is a new identity on a new exit; --batch 10 is one identity doing
ten searches. Different experiments, different questions, both recorded on every
row. At --batch 1 a Chrome-driving engine also pays a fresh profile's 43 MB
vendor fetch on every attempt.
This harness sends real traffic to real targets through a shared production pool. More requests do not make a better experiment, and past a point they stop making an experiment at all - once a target is reacting to the harness, what is being measured is the harness.
The circuit breaker is not an error handler. N consecutive failures stop a cell and it stays stopped: every retry after a refusal confirms automation to the target and degrades the exit ranges for every other customer on the account. There is no "error, new sid, retry" path here.
N is measured. Over 129 cells and 1464 attempts, the chance an attempt succeeds
given the failures before it in its own cell is 75% at zero, 5.8% at five and 1.6%
from the sixth onward. Stopping at 5 records a partial refusal as a total one;
running past 10 spends about 98 retries per delivered page. --breaker defaults
to 10, and CircuitBreaker stays at 5 because every google_429 run on disk was
measured there.
Pause between requests. 3-5 seconds minimum.
Never print or paste Proxy-Authorization. It is base64, not encryption.
data/runs/ is committed, and masked. Exit addresses are reduced to their /24
and the proxy username to <login> - those are real people's home connections, and
the username identifies the account. Masking happens at the one choke point every
row passes through, and tests/test_runs_are_publishable.py fails if a full
address ever reaches disk.
It is committed because every claim above names the run it came from, and several
of those claims are corrections that were only possible because the original rows
were still there. They are not a baseline for your own numbers: a rate here is
a reading of the hours it was taken in. data/runs/README.md says what each
filename prefix holds and what the masking guard has already missed twice.
Estimate anything above ~100 requests. --dry-run prices traffic from
per-target constants calibrated by scripts/analysis/calibrate.py, each carrying
the run it was read from, and names which targets were measured. Read the hours as
an order of magnitude.
Every attempt this repository has made is committed under data/runs/ as one
JSONL row, and every claim above is a count over those rows. scripts/analysis/
reads that directory and nothing else - no network, no credentials, no browser -
and imports only the standard library. A clone and three commands, with nothing
installed and nothing spent:
python scripts/analysis/report.py --all # the matrix runs
python scripts/analysis/held.py # the probe-and-hold runs
python scripts/analysis/playbook.py # what to try first, ranked by lower bound
Where each headline lands:
Four caveats the tools print and a table cannot:
- Pooled is the weaker reading. report.py --allpools runs from different weeks into a number belonging to neither. It says so at the top and marks every incomplete cell; one file is stronger:report.py data/runs/<file>.jsonl.
- unmeasuredis not zero. A cell stopped by ten burned exits in a row was never served a body, so it carries no observation of the engine at all. Printing 0% there would hand the pool's condition to the framework, which is why the DuckDuckGo losers are quoted out of the served block and not the pass-rate table.
- The CONNECT floor was the harness's own traffic, and every row above it
predates the fix. 23% against 0% put the failures on the proxied path rather than on a
flaky local link, and a second machine on another line in another datacentre
read 25%, which looked like proof that the path was the provider's problem. It
was not that simple. HTTP proxy authentication is challenge-response, so a
browser handed credentials opens the first CONNECT of each session without
one, takes the 407 and retries; the gateway counts unauthenticated requests
per address and bans on a threshold. At one session per attempt the harness
generated one such CONNECT per attempt and banned itself, on both machines
equally - which is why a second network could not see it. Measured inside one
uninterrupted run either side of the gateway-side fix: ERR_EMPTY_RESPONSE207 of 1131 attempts before, 1 of 1004 after.probes/proxy_auth_shape.pyreproduces the client half against a proxy on loopback and spends nothing. Full account inNOTEBOOK.md.
- held.pywith no argument pools every probe-and-hold window, where- NOTEBOOK.mdquotes the three that varied geo. The aligned arm is the same 45 probes either way; the unaligned arm picks up rows from windows where geo was not the axis and reads 42% rather than 46%. Direction survives, magnitude moves.
The aggregator is optional. The format is one JSON object per line and the columns
are defined by ROW_FIELDS in nmbench/engines/base.py, so counting something is
five lines. The DuckDuckGo split, from scratch:
import collections, glob, json
tally = collections.Counter()
for path in glob.glob("data/runs/benchmark_*.jsonl"):
for line in open(path, encoding="utf-8"):
row = json.loads(line)
if row.get("target") == "ddg_serp" and not row.get("direct"):
tally[row.get("engine"), row.get("headless"), row.get("verdict")] += 1
print(sorted(tally.items(), key=str))That prints camoufox 44 ok, obscura 44 ok and patchright headful 7 ok with
no refusal between them, against 27 and 23 captcha for the two headless Chromium
cells with no pass between them. 95 and 50, straight out of the rows.
peek.py <file> prints one line per attempt for reading a single run by hand.
Two things are deliberately not checkable. The bodies are not published -
gzipped into gitignored data/artifacts/, because exit addresses turn up in
embedded links. And exit addresses are recorded as their /24, because a
residential pool is other people's home connections. No analysis reads a full
address back out, so nothing above depends on the masked half.
If a number cannot be traced back to a run, it does not belong here.
nmbench/ the reusable package - this is what gets published
├── config.py credentials from .env, per provider, on first use
├── providers.py loads data/providers/*.toml: one gateway's dialect each
├── proxy.py username DSL builder + client-side validation
├── gateway.py CONNECT probe, exit address lookup, /24 masking
├── relay.py local authenticating CONNECT forwarder + byte counter
├── breaker.py circuit breaker, one per matrix cell
├── console.py keeps progress output from killing the run
├── matrix.py cells, round-robin scheduling, resume, cost estimate
├── queries.py loading the committed query lists
├── blocking.py resource blocking presets, byte counters
├── targets.py url building + content-based verdicts
├── stats.py the Wilson interval every rate here is quoted with
├── sink.py JSONL output, one file per run
├── artifacts.py gzipped response bodies, so a verdict can be re-read
├── __main__.py python -m nmbench <command>, one entry point
└── engines/ one module per framework, one shared contract
├── base.py the row schema and the contract every engine implements
├── http.py plain requests client, the no-browser control
├── chromium.py unmodified Chromium (the control) and Patchright
├── camoufox.py patched Firefox over Playwright
├── cloak.py patched Chromium handing back a Playwright browser
├── rebrowser.py a Playwright fork patching the Runtime.enable leak
├── obscura.py Rust browser with its own renderer, over CDP
├── seleniumbase.py Chrome over ChromeDriver, the WebDriver family
├── zendriver.py Chrome over raw CDP, no WebDriver and no Playwright
├── botasaurus.py Chrome over raw CDP, a second one, for the 2x2
└── curlcffi.py scriptless client wearing Chrome's ClientHello
scripts/ README: which of these can spend money
├── benchmark.py the matrix runner: engines x targets, one time window
├── probes/ one file per question, each cheap and single-purpose
├── analysis/ aggregation over data/runs/, sends nothing
└── tools/ generators for committed inputs
data/
├── providers/ README: one .toml per gateway, and why it is not code
├── queries/ README: committed inputs, one seed, two lists
└── runs/ README: masking, filename prefixes, how to read a row
docs/ README: quickstart and the two findings write-ups
tests/ offline suite: verdicts, scheduler, DSL, hygiene
Every folder a reader lands in from the file list has its own README, because a directory listing on GitHub is where navigation actually starts.
The split between probes/ and analysis/ tells a reader at a glance which files
can spend money: anything under analysis/ only reads data/runs/. A probe that
happens to send nothing says so, and python -m nmbench marks it [offline].
CONTRIBUTING.md has the rules that are not obvious from the code, most of them
there because the instrument has already been broken that exact way by a commit
that passed every test at the time. The two that catch people first: do not harden
the unmodified control, and nothing branches on an engine, provider or target name.
When adding an engine, a target or an experiment: keep the variable under test explicit, record enough to reproduce the result, do not compare runs from unrelated time windows, keep target refusals separate from harness and transport failures, add offline tests for classification and scheduling, and regenerate the derived blocks rather than editing a generated number by hand.
The most useful issue you can open is that a number here is wrong. Bring a denominator.
pip install -r requirements-ci.txt
make check # ruff plus the suite: offline, no credentials, no browser