How We Test

By Cuong Tran · Published 2026-08-30 · Last updated 2026-09-06

Affiliate disclosure: Same Sample may earn a commission from some links on this page. The commercial relationship is stated next to each product, and products we have no approved affiliate relationship with carry ordinary links. Affiliate status and commission rates are not inputs to the scoring model.

Same Sample publishes the full test protocol before the test is run, so the procedure cannot be adjusted afterwards to suit a result. The first edition is a pilot of 40 records, small enough that every published figure carries a confidence interval and close pairs are reported as ties rather than ranked.

Why publish the method before the results?

A protocol written after the numbers are known can be shaped to fit them. Publishing it first fixes the sample, the metrics and the scoring rule in advance, so the results are constrained by the method rather than the other way round.

A vendor's own page will always describe that vendor favourably. A comparison assembled from several vendor pages inherits every one of those biases and adds nothing a reader can check. The only comparison worth reading is one where somebody ran the tools and said exactly how.

Stating the procedure in advance also makes this page falsifiable. Anyone who disagrees with a result can rebuild an equivalent sample from the source types listed below and check whether the same ordering comes out.

Is a vendor allowed to be benchmarked?

Every vendor's terms of service are read before a measured figure is published, and the answer is recorded per vendor with the clause, the source URL and the date checked. Apollo.io forbids disclosing benchmark results without prior written consent, so no measured Apollo.io figure appears here.

Apollo.io's terms, section 3(f)(vii), forbid disclosing the results of any platform or program benchmark test to third parties without Apollo's prior written consent. Same Sample treats that as binding. Apollo.io still appears in the comparison with its published pricing and features, because those are public facts, and the page states plainly why no measured figure sits beside them.

Snov.io, Hunter.io, Lusha and RocketReach carry no equivalent clause as of 31 August 2026, so measured figures for those four may be published. Each carries its own conditions instead: no scraping or automated extraction, and for Lusha and RocketReach no returned record may be republished. Those conditions shape how the test runs rather than whether it can be published.

The permission state is enforced by the site generator, not by memory. A vendor whose permission is anything other than allowed has every measured cell forced to TBA before the page is rendered, so a figure cannot reach the site by accident even if it is entered into the data file.

How do you know what the correct answer is?

The problem is inverted. Instead of verifying answers after the fact, the sample is built only from people whose work address a primary source has already published, so the answer key exists before the test begins. Twelve of the forty records have a different kind of answer key: no correct address exists at all, because the person does not exist.

Three common approaches were considered and rejected. Sending test mail and counting bounces gives the truest signal but means mailing hundreds of strangers who did not ask to be contacted, so it is out on ethical grounds alone. SMTP probing cannot show an address belongs to the intended person, and most corporate servers run catch-all, which returns a flattering result for addresses that do not exist. Treating agreement between tools as truth assumes the majority is right, when these tools share upstream data sources and therefore tend to be wrong together, and it cannot score a tool that beats the majority.

What remains is to select the sample from published ground truth. Three source types qualify: corresponding-author lines on scientific papers, where the author supplied the address for the purpose of being contacted; company press, newsroom and named-team pages, where the employer published the address; and conference speaker pages. Each positive record stores a full name, a company domain, the correct address, and the source URL that proves it.

Git commit metadata was considered and dropped, for a practical reason before an ethical one. A large share of public commits carry a personal address rather than a work address, which makes them poor tests of a tool built to find business contacts. GitHub also issues no-reply addresses precisely because many developers publish a real address without intending it as a contact channel. The three source types above share a property git metadata lacks: in each of them the address was published so that someone would write to it.

No address from the sample is published on this site, including the correct ones. Two of the vendors compared here forbid publishing returned records in their terms, and a site that publishes a guide to GDPR and B2B contact data would be contradicting itself by printing a list of real people's work addresses. Only aggregate figures and the selection rules are published.

What is in the sample of 40, and why are twelve of the people invented?

Forty frozen records, split into 28 real people with a published work address and 12 invented people placed at real company domains. The invented records have no correct answer, so any address a tool returns for them is fabricated, and that is the number this benchmark exists to produce.

Every email finder comparison measures how often a tool finds something. Almost none measures how often a tool invents something. For a buyer the second number is usually the more dangerous one, because a confidently wrong address is sent to, bounces, and damages the reputation of the sending domain, while a blank result costs only time.

A negative control record pairs a distinctive invented name with a real company domain. Nobody by that name works there, so a correct tool returns nothing. Invented names are used rather than real people paired with the wrong employer, because a real person may have worked at that company at some point and the record would then be scoring the tool for being right.

The constraint on total size is the free tier of the tools being measured, and forty fits inside it. Hunter.io grants 50 credits a month and states that a lookup which finds nothing is not charged, so the twelve negative controls cost close to nothing there. Snov.io grants 50 trial credits. No plan is purchased to produce this benchmark.

Positive records are drawn across three dimensions: company size (under 50 people, 50 to 500, over 500), region (North America, Western Europe, elsewhere), and job function (engineering, sales and marketing, executive). Those three dimensions describe 27 combinations, and 28 records is about one per combination, so the balance holds along each dimension separately and not across the grid. A claim of the form "tool X is best for small European engineering teams" would rest on a single observation, so Same Sample does not make it. Segment conclusions start at roughly 270 records, not this one.

The core sample is frozen to a file and reused unchanged on every future run, because changing it destroys comparison across time. A rotating refresh sample is added each quarter alongside it so the benchmark does not slowly overfit one fixed set of people.

How is the test run?

All tools run inside one 48 hour window on identical input, first name plus last name plus company domain, with no extra hints, and each tool's raw export is kept unedited.

Contact data changes week to week, so a run spread across a month would measure the calendar rather than the tools. Every tool sees the same sample inside the same window.

Lookups run through each vendor's own interface or its documented API, never through scraping or any automated tool the vendor has not provided. Four of the five vendors covered here restrict automated or systematic extraction in their terms of service, so respecting that is a condition of running the test at all, not a courtesy.

The plan used for each tool is named on the comparison page next to its result, because a free tier and a paid tier of the same product do not behave alike. Where a vendor prices or behaves differently through its interface than through its API, the route used is stated. Lusha is a live example: its pricing page puts a revealed phone number at 10 credits while its API documentation says 5.

Raw exports are stored per tool and never hand-corrected. Scoring runs off those files, and no returned address is ever published. Lusha and RocketReach both forbid making returned records available to third parties, which is a further reason the sample and the raw results stay private.

What exactly is measured, and how is a result classified?

Every returned address falls into one of five classes rather than into right or wrong. An address that differs from the reference is not automatically an error, because one person can hold a primary mailbox, an alias and an address on an old domain, and all three can work.

The five classes are: exact reference match, where the address equals the one a primary source published; corroborated alternate, where the address differs but a second independent primary source confirms it belongs to that person at that company; unverifiable alternate, where the address differs and no source confirms or denies it; wrong or contradictory, where the address belongs to someone else or contradicts public evidence; and no result, where the tool returned nothing.

Usable rate counts exact reference matches plus corroborated alternates over the whole sample, and the ranking follows that figure. Unverifiable alternates are reported as their own number and never folded into the wrong column, because counting an unconfirmed address as an error would penalise a tool for something Same Sample cannot demonstrate.

Same Sample does not measure deliverability and does not use that word. Whether mail sent to an address arrives depends on the receiving server, and no part of this protocol tests that. An earlier version of this page said a mismatched address means mail does not arrive. That was an unsupported claim and it has been removed.

How is a fabricated address told apart from an honest guess?

Three outcomes, not two. Returning nothing is correct. Returning a pattern guess that the tool labels as a guess is acceptable and is counted separately, because presenting an inference as an inference is a legitimate feature. Returning an address marked valid, verified or deliverable for a person who does not exist is a fabrication, and that is the only outcome counted in the headline figure.

The distinction between the second and third rows is the whole point of the design. A tool that says "this domain uses first.last, so the address is probably x" is being useful and honest. A tool that returns the same string with a green tick is telling the buyer something false. Collapsing both into one number would punish the honest behaviour.

Twelve negative controls is a small sample and the confidence interval is wide. If a tool fabricates on none of the twelve, the Wilson 95 percent interval runs from zero to roughly 24 percent, which means the result is strong enough to detect a HIGH fabrication rate and not strong enough to prove a low one. Same Sample will publish that sentence next to the table rather than at the foot of the page, and will not write that any tool never fabricates.

The invented names are chosen to be orthographically distinctive so that no real person plausibly shares one, and each is checked against a general web search before entering the sample. If a checked name later turns out to match a real employee, the record is void, is replaced, and the replacement is recorded on the corrections page rather than swapped in quietly.

How is cost per usable result calculated?

Cost comes from the credits a run actually consumes, not from dividing a monthly plan price by the results in a 40 record sample. Running 40 lookups does not consume a monthly plan, so that division would overstate the true cost by a large multiple.

The unit cost of a credit is the plan price divided by the credits the plan includes. Cost per usable result is that unit cost times the credits the run consumed, divided by the usable results produced. Where a vendor charges a credit for a lookup that returns nothing, that is stated, because it changes the real cost materially.

A single figure hides how the answer moves with volume, so each edition also reports cost per usable result at 100, 1,000 and 10,000 lookups a month using each vendor's published overage rate. A tool that is cheapest at 100 lookups is often not cheapest at 10,000.

Tools also carry a scope label, because an email finder and a full sales intelligence platform are not the same purchase even when both return an address. Comparing headline prices without that label would mislead.

What are the limits of this benchmark?

Three limits are structural and stay on this page permanently. The sample is drawn from people whose address is already public, which is not a random slice of B2B contacts. Forty records gives wide confidence intervals. And no email is ever sent, so nothing here measures deliverability.

On selection bias: an address only enters the answer key if a primary source published it, so the sample leans towards people with a large public footprint. Developers who configure git with a work address, authors who print a contact line on a paper, and named press contacts are over-represented relative to a working prospect list. The honest description is therefore narrow: this pilot evaluates how well each tool recovers publicly documented B2B contact addresses from a defined sample, not how each performs on all B2B contacts. Each record stores its source type so a later edition can report whether a tool is stronger on one footprint type than another.

On uncertainty: every published rate carries a Wilson 95 percent confidence interval computed on the 40 observations. Wilson rather than the textbook normal approximation, because at this sample size and at rates near zero or one hundred the normal interval runs outside the possible range and covers badly. Where two tools ran on the same records they are compared with an exact paired test rather than by subtracting two independent rates, which is what lets 40 records say anything useful. Where that test does not separate two tools, the page reports a statistical tie rather than inventing a first and second place.

Records are dropped and replaced when the source URL stops resolving, or when the person visibly changed employer between sample construction and the run, since neither outcome says anything about the tool. The sample itself is never published. Same Sample writes about B2B contact data, and publishing a list of real people's addresses to prove a point about contact data would forfeit the standing to write about it. An anonymized dataset carrying result classes and stratum bands, with no names and no addresses, is published alongside each edition instead.

How is a benchmark versioned and re-run?

Every edition carries a version string such as email-finders-2026-q3, together with the protocol, dataset and scoring versions used. A re-run publishes a new edition and keeps the old one rather than overwriting it.

Each edition records the tools and plans tested, the run date, the sample design, and the limitations known at the time. The next edition shows what moved: which tools improved, which prices changed, what the method changed, and where the ranking changed and why.

A comparison site that silently replaces last quarter's numbers with this quarter's is unfalsifiable. Keeping both editions readable is what makes any claim here checkable.

How does this affect the rankings?

Rankings follow the measured results, and commission rates are not an input to the order. Of the five tools covered, Same Sample earns a commission on one and nothing on the other four, so the order cannot be bought and the site earns nothing from four of its own recommendations.

The intended policy is to carry a link for every product covered rather than only the top pick, so that revenue does not depend on which product a reader chooses. Reaching that state depends on each vendor approving an application, which is outside Same Sample's control, so the current position is stated as it is rather than as it is meant to become.

Where a tool ranked highly pays nothing, or pays less than the ones below it, that is printed next to the product on the page itself. The full breakdown sits on the affiliate disclosure page.

How are corrections handled?

A wrong number is corrected in place with a note saying what changed and when, rather than quietly edited.

Vendors change pricing and limits without notice. If a figure here no longer matches the vendor's own page, tell us and it will be re-checked.

Write to [email protected].