Data quality

Websites change their pages without warning, so we check every scraper against its live site on a schedule.

Scheduled checks

Each scraper is checked against its live site on a schedule, by a check run (a "canary"). The results will be published on the status page before the API opens. Each check runs the scraper against known inputs and fails when any of these is true:

• the run did not finish successfully;
• it returned fewer rows than expected, or far more;
• rows are missing their key field, or the same key appears twice;
• an important field is empty in too many rows (each scraper has its own fill-rate threshold per field);
• the newest row is older than the scraper's freshness limit, for sources that publish dated items;
• numbered results have gaps, for sources that page through results.

The thresholds are set per scraper from what its source normally returns. Once results are published, a failed check shows on the status page.

What a scraper must pass before the API sells it

From launch, the catalogue the API sells from is generated by a script, not edited by hand. A scraper is in it only when all of these hold at generation time:

1. Its three most recent checks passed, and the newest is at most 36 hours old.
2. Its price covers its measured cost with room to spare. We measure cost from recent check runs, at the 75th percentile, and a scraper whose cost rises too close to its price is taken out.
3. It has a run test, and every row that test produces passed our personal-data scan (see security).
4. No open question about the scraper is waiting on a decision.
5. The code that runs for you is the same build that passed the checks.
6. A weekly smoke run of the scraper's family on the production setup passed within the last 8 days.
7. Every option that saves state between runs, such as "only new results", is declared, so that the state can be kept separately for each customer. Until that works for a scraper, the option is hidden. One customer's saved state never affects another's results.

A scraper that stops meeting any of these drops out of the catalogue at the next generation.

Health, as shown on each scraper

The health label on a scraper's page comes from its latest checks: when the last check passed and whether the recent ones passed. We show numbers such as success rate or typical run time only once we have measured them on our own runs. Until then the page says nothing rather than guess.

"Collected since"

Some scrapers have a history panel: a fixed set of pages we collect on a schedule and keep. "Collected since" is the date of the first collection in that panel. It is when our history starts, not when the source started publishing. A scraper without a panel shows no date.

When we get it wrong

If rows look wrong, write to [email protected] with the run ID. We compare the rows with the source page. You pay only for delivered rows, and a billing error found this way is refunded as credit.