# Website Categorization Bulk Feed — Data Dictionary **Audience:** licensees integrating the Website Categorization bulk domain-categorization feed. This page defines every column, every value domain, and the two timestamps that are most often confused. It assumes no knowledge of our internals. **Download this document:** [`website-categorization-feed-data-dictionary.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-feed-data-dictionary.md) — the same reference as a single file, for your integration team. **Version history:** [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog) — every dated change to the feed's published values and semantics, with the two policy-pack references beside it. If you are integrating the real-time API instead, see the API documentation [API quickstart & error reference](https://website-categorization.whoisxmlapi.com/api/documentation/v4/making-requests) — the feed and the API share a taxonomy but not a schema. --- ## 1. What you receive Each export is a directory containing: | File | Contents | | --------------- | ---------------------------------------------------------------- | | `*.csv.gz` | gzipped CSV, one header row then one row per domain | | `*.parquet` | the same rows, same column order, native types | | `manifest.json` | row counts, per-file SHA-256, schema version, and lane semantics | Both data files carry **identical rows in identical column order**. Choose whichever suits your loader; you do not need both. ### 1.1 Where and when it arrives Delivery is an S3 bucket you read with your own read-only key. The bucket, the key and the commands come with your license. | Item | Value | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Full snapshot | weekly, starting on Sundays and lately taking about 30 hours to run: sixteen partition folders `exports/snapshot__p00/` … `_p15/`, each with `snapshot_pNN.csv.gz`, `snapshot_pNN.parquet` and `manifest.json`. The set is **complete only when `exports/snapshot_/run_manifest.json` exists** (usually on Monday, UTC) — ingest nothing before it appears ([§10](#10-manifestjson)). | | Daily delta | 10:00 UTC every day except Sundays, when the snapshot holds the export lock: `exports/delta__100000Z/` with `delta.csv.gz`, `delta.parquet` and `manifest.json`, landing within about an hour on weekdays; the Monday delta waits for the weekly set's run marker, so it lands after the set does. The folder is named by the **start of the window it covers**, so the delta published on the 10th is `delta_2026-09-09_100000Z`. A window opens at the prior run's cutoff, so after a skipped run the next spans two days; to resume after a set, see below. Apply every delta row as an upsert by `domain` and never filter a delta by its own timestamps: a row can qualify through a category-side event the feed does not show (an abstain, or a label withdrawn by suppression), so a small share of every delta has no timestamp inside the window: its `classified_at` is NULL and its `status_observed_at` is older than the window. | | Documentation | this dictionary, the changelog and both policy-pack references sit beside each set's `run_manifest.json`, each stamped with the revision it was published from. | | Older folders | snapshot folders dated on or before 2026-08-03 predate the completeness marker and are not whole sets; ignore them. | **Resuming after a full set.** After loading a set, apply the delta whose window contains the whole of the set's export — it opened before the set began and was read after its `run_manifest.json` appeared — then every later delta in order. For the weekly set that is normally the delta dated the day before it — `delta_2026-10-17_100000Z` after the 2026-10-18 set — which runs only after the set is complete, so it never carries an older version of a row than the set holds. Skipping it loses what changed while the set was being written. A row can still reach you one delta after its crawl (a new domain crawled before a delta's read but first labelled after it, which ships in the next delta), and a full set reconciles anything a delta did not carry. From `delta_2026-10-30_100000Z`, a delta also re-reads crawls stamped in the 30 minutes before its window opens, so a crawl stamped just before a cutoff but saved just after it is not missed; one saved more than 30 minutes after its stamp still can be, until the next full set. The cost is repetition: a row can repeat one the previous delta carried, with its `status_observed_at` up to 30 minutes older than the window and its `classified_at` older too, or NULL. An upsert by `domain` absorbs the repeat. Verify every downloaded file against the SHA-256 in its `manifest.json` ([§10](#10-manifestjson)). **Which schema version you are reading.** The 2026-09-06 set reports `feed_schema_version` 1.13.0; every delta from 2026-09-08 and the 2026-09-13 set report 1.13.2 with the same 38 columns. The first 1.14.0 artifact is the **Thursday 2026-09-17** snapshot set — an off-cadence full set published the day the exporter deploys — and it is a full replace ([Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)). Every artifact from the **Tuesday 2026-09-29** exporter deploy reports **1.15.0**: the same 44 columns in the same order, with one published value moving — `confidence_tier` is capped at `MEDIUM` for the name-only LLM lane ([§5](#5-confidence-and-tier)). Those rows are written on 2026-09-30, so the first artifacts carrying any of them are the **daily deltas from 2026-09-30** and the **Sunday 2026-10-04** set; a delta between the deploy and the load reports 1.15.0 with no such row in it yet. Every artifact from the **Friday 2026-10-09** exporter deploy reports **1.16.0**: the same 44 columns in the same order, with one published value moving again — `confidence_tier` is capped at `MEDIUM`, and at `LOW` where the calibrated tier is `DOUBTFUL`, for the teacher-rescored provenance suffixes ([§5](#5-confidence-and-tier)). Those rows already exist, so unlike 1.15.0 this re-rates rows you already have: the deltas from 2026-10-10 report 1.16.0 (the first is in folder `delta_2026-10-09_100000Z`), but the re-rating reaches you only with the **Sunday 2026-10-11** full set ([Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)). Every artifact from the **Tuesday 2026-11-03** exporter deploy reports **1.17.0**: columns 1–44 unchanged, and two columns **appended** — IAB Content Purpose = News for the domains on a curated list of known news publishers ([§14](#14-iab-content-purpose-columns-4546)). The first 1.17.0 artifact is the **Wednesday 2026-11-04** delta (folder `delta_2026-11-03_100000Z`) and the first 1.17.0 set is the **Sunday 2026-11-08** set. ### The column order is the contract The 46 columns below ship in exactly the order listed (38 at `feed_schema_version` 1.13.x; columns 39–44 arrive with artifact 1.14.0 and columns 45–46 with artifact 1.17.0), in both the CSV header and the Parquet schema. **This order is the licensable contract.** A reorder or a rename is a breaking change and will not happen without a major version change and notice. New columns are only ever **appended** to the end, so: - reading by column **name** is always safe; - reading by **position** is safe for every column that existed when you integrated; - a consumer that stops reading after column _N_ is unaffected by later additions. Check `feed_schema_version` in `manifest.json` to know which columns and manifest keys to expect. It is a semantic version. The MINOR component increments on purely additive changes — new columns, new manifest keys, or a new manifest block — with no change to the published row population. The PATCH component increments when the published **row population** changes with no column change: no column is added, removed, renamed or retyped, and no manifest key is removed or given a new meaning. A PATCH release may **add** keys inside an existing `manifest.json` object, so validate the manifest permissively — treat its objects as extensible rather than as a closed set of properties — but it never changes the column schema your reader is built against. Every such change is listed in the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog). --- ## 2. Column reference Types are the Parquet physical types. In CSV every value is text — see [§7 Nulls and encoding](#7-nulls-and-encoding). | # | Column | Type | Meaning | | --- | ---------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | `domain` | `string` | The name the row describes: normally the registrable domain, but not always — see the note below the table. Lowercase ASCII with internationalised names in punycode (`xn--…`) — from 1.14.0; artifacts before it also carry Unicode-form names, most beside their punycode twin (see §7). Unique — no `domain` value repeats within an export. | | 2 | `tier1` | `string` | IAB Content Taxonomy v3.1 tier-1 name, e.g. `Technology & Computing`. The broadest category. Columns 2–5 carry the **single-label convention** — see the note below the table. | | 3 | `tier2` | `string` | IAB v3.1 tier-2 name. Null when the label stops at tier 1. | | 4 | `tier3` | `string` | IAB v3.1 tier-3 name. Null when the label stops higher. | | 5 | `tier4` | `string` | IAB v3.1 tier-4 name. Null when the label stops higher. | | 6 | `iab_id` | `string` | Canonical IAB leaf id for the tier path, e.g. `IAB-596` or `IAB-Rm3SiT`. An **opaque** identifier — numeric on most rows, alphanumeric on a small minority. Match it whole; never parse structure out of it. | | 7 | `confidence` | `float32` | Score for the label in `[0, 1]`, 4 decimal places. **Its meaning depends on `model_name`** — see [§5](#5-confidence-and-tier). | | 8 | `confidence_tier` | `string` | Discrete band: `HIGH` / `MEDIUM` / `LOW`, or `PARKED`. Through 1.13.x it is the stored `confidence` cut; from 1.14.0 it is **derived at export** under the policy named in `confidence_band_policy`. See [§5](#5-confidence-and-tier). | | 9 | `tier` | `int16` | Provenance/priority of the label, `0`–`3`. **Not** a taxonomy depth. See [§5](#5-confidence-and-tier). | | 10 | `model_name` | `string` | Component that produced the label, e.g. `ensemble-5en-3nonen`. **Determines what `confidence` means** — see [§5](#5-confidence-and-tier). | | 11 | `model_version` | `string` | Version of that component. May carry a `+variant` suffix — **do not match with `=`**, see [§5](#5-confidence-and-tier). | | 12 | `taxonomy_version` | `string` | Taxonomy the tier names belong to, e.g. `iab_v3.1`. | | 13 | `classified_at` | `string` | ISO-8601 UTC. **When we processed the domain** — see [§4](#4-the-two-timestamps). | | 14 | `language` | `string` | Language detected on the crawled page, e.g. `en`. Null where not detected or not recorded. | | 15 | `domain_status` | `string` | The status lane. One of six values — see [§3](#3-domain_status--the-six-lanes). | | 16 | `openrtb_cat` | `list` | OpenRTB/AdCOM raw category ids for the tier path. **Array-valued** — see [§6](#6-openrtb_cat--an-array-column). | | 17 | `status_observed_at` | `string` | ISO-8601 UTC. **When the crawl observed `domain_status`** — see [§4](#4-the-two-timestamps). | | 18 | `last_known_tier1` | `string` | Historical tier-1 label. Populated only on some non-active rows — see [§8](#8-the-last_known_-columns). | | 19 | `last_known_tier2` | `string` | Historical tier-2 label. | | 20 | `last_known_tier3` | `string` | Historical tier-3 label. | | 21 | `last_known_tier4` | `string` | Historical tier-4 label. | | 22 | `last_known_iab_id` | `string` | Historical IAB leaf id. | | 23 | `last_known_classified_at` | `string` | ISO-8601 UTC. When that historical label was produced — its own as-of date. | | 24 | `calibrated_confidence` | `float32` | **Calibrated** estimate of the probability that a strict blind judge, reading the page, assigns the same `tier1`, `[0, 1]`, 4 dp. On average it understates how often a label is acceptable, though not in its top band. See [§5](#5-confidence-and-tier). | | 25 | `calibrated_confidence_tier` | `string` | Band for `calibrated_confidence`: `PROBABLE` / `TOSSUP` / `DOUBTFUL`. See [§5](#5-confidence-and-tier). | | 26 | `calibration_segment` | `string` | Which calibration curve was applied, e.g. `en\|bf`. See [§5](#5-confidence-and-tier). | | 27 | `site_purpose_tier1` | `string` | **Reserved** (the site-purpose axis). NULL on every row through 1.13.x — see [§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection). | | 28 | `site_purpose_tier2` | `string` | Reserved. NULL through 1.13.x. | | 29 | `site_purpose_tier3` | `string` | Reserved. NULL through 1.13.x. | | 30 | `site_purpose_tier4` | `string` | Reserved. NULL through 1.13.x. | | 31 | `site_purpose_iab_id` | `string` | Reserved. Canonical leaf id of the purpose label, same format as `iab_id`. NULL through 1.13.x. | | 32 | `site_purpose_confidence` | `float32` | Reserved. The purpose model's own probability — not comparable to `confidence` or `calibrated_confidence`. NULL through 1.13.x. | | 33 | `site_purpose_status` | `string` | **Never NULL.** `labeled` / `undefined` / `not_available`. Through 1.13.x: `undefined` on parked / inactive / dead rows and parked-detector rows, `not_available` elsewhere. See [§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection). | | 34 | `site_purpose_relation` | `string` | Reserved. `same_path` / `same_tier1` / `different_tier1` between the two axes. NULL through 1.13.x. | | 35 | `site_purpose_model_version` | `string` | Reserved. Provenance of the purpose label. NULL through 1.13.x. | | 36 | `policy_group` | `string` | The **policy-pack group**, e.g. `gambling`. At 1.12.0–1.13.x the `policy_v1` rollup of the tier path; from artifact 1.14.0 the pack is `policy_v2`, which reads `safety_flags` FIRST and the tier path after, and retires the group `news` in favour of `current_events`. NULL only on a suppressed row or where neither axis resolves — from 1.14.0 a flagged row carries a group even where `tier1` is NULL. See [§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection). | | 37 | `policy_basis` | `string` | Which axis `policy_group` was projected from: `primary` (the tier path), `site_purpose`, or — from artifact 1.14.0 — `safety` (the brand-safety axis, read first). Always `primary` at 1.12.0–1.13.x; NULL where `policy_group` is NULL. | | 38 | `policy_pack_version` | `string` | `policy_v1@2026.09.2` at 1.12.0–1.13.x, `policy_v2@2026.09.1` from artifact 1.14.0 — the pack and mapping version the row was projected with; constant within an artifact, NULL where `policy_group` is NULL. | | 39 | `safety_status` | `string` | **Never NULL** (artifact ≥ 1.14.0). Whether a brand-safety filter reached a decision about this domain: `flagged` / `assessed_clean` / `not_assessed`. See [§13](#13-the-brand-safety-axis-columns-3941). | | 40 | `safety_flags` | `list` | (artifact ≥ 1.14.0) The flagged block-signal themes, e.g. `["gambling"]`. Empty list, never NULL, on a row that carries a category; NULL on a suppressed row. See [§13](#13-the-brand-safety-axis-columns-3941). | | 41 | `safety_detail` | `string` | (artifact ≥ 1.14.0) A compact JSON object, one entry per filter family that ran, carrying that filter's status, score, scale, theme and version — a family reporting `not_assessed` is retained, so a populated detail is not evidence that anything assessed the domain. NULL on a suppressed row, and on a row where no verdict was materialized at all. See [§13](#13-the-brand-safety-axis-columns-3941). | | 42 | `policy_confidence_basis` | `string` | (artifact ≥ 1.14.0) What the row's number IS: `model_calibrated` / `model_raw` / `filter` / `rule` / `curated` / `carry_over` / `name_only` (artifact ≥ 1.15.0) / `none`. Closed enum; if you validate it against a strict enum, add `name_only` before ingesting a 1.15.0 artifact. NULL only where the category block is suppressed. See [§11](#policy_confidence_basis-and-policy_group_set-columns-4243). | | 43 | `policy_group_set` | `list` | (artifact ≥ 1.14.0) `policy_group` plus every group the pack declares it is blocked WITH, in pack order — one list to match instead of reconstructing the union. NULL exactly where `policy_group` is. | | 44 | `language_primary` | `string` | (artifact ≥ 1.14.0) The BCP-47 **primary subtag** of column 14, e.g. `zh` for `zh-CN` / `ZH_TW` / `zh-Hans-CN`. Derived; column 14 stays raw. NULL where no language was observed. | | 45 | `content_purpose_id` | `string` | (artifact ≥ 1.17.0) `1020` — the IAB Descriptive Vector id of **Content Purpose > Informational > News** — on the domains of a curated list of known news publishers; NULL on every other row. NULL means **unknown, never "not news"**. Not a Content Taxonomy id: it never appears in `iab_id` or `openrtb_cat`. See [§14](#14-iab-content-purpose-columns-4546). | | 46 | `content_purpose_source` | `string` | (artifact ≥ 1.17.0) Which rule set column 45: `curated` (that curated list) — the only value the feed emits; `schema_org` is **reserved**. NULL exactly where column 45 is NULL. See [§14](#14-iab-content-purpose-columns-4546). | **What `domain` holds.** Normally the registrable domain, but not on every row: some rows are a host below a registrable domain (`blog.example.com`) or a bare public suffix. Look a name up as written first, and fall back to its registrable domain's row only when there is no exact match: a host is labelled from its own pages, and it often carries a different `tier1` from its parent. Columns 24–26 were **appended** 2026-07-30 (`feed_schema_version` 1.6.0). Column 7 `confidence` is unchanged in both meaning and position. Columns 27–38 were **appended** 2026-09-01 (`feed_schema_version` 1.12.0); nothing before them changed. Four optional manifest keys for the shippable-key coverage denominator were added 2026-09-03 (`feed_schema_version` 1.13.0, [§10](#10-manifestjson)); no column changed. The first **PATCH** bump, `feed_schema_version` **1.13.1**, was specified to ship with the first Sunday export after its deploy: parked-flagged brand-safety filter rows ship status-only and the `demoted_label_suppression` manifest block gains counters ([§10](#10-manifestjson), [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)); no column changed and no manifest key was removed. **No artifact reports 1.13.1**: 1.13.2 superseded it before any export carried it — the 2026-09-06 set reports 1.13.0 and every export from 2026-09-08 reports 1.13.2, which carries the 1.13.1 change as well. The rule behind that sequence applies to every future bump: a daily delta exported between a deploy and the next Sunday snapshot is stamped with the new version too — both kinds are stamped from the same constant — so consume a full snapshot at the new version before relying on those deltas, because rows the new build masks do not cross the delta cutoffs. The second PATCH bump, `feed_schema_version` **1.13.2**, ships with the first ≥ 1.13.2 export of either kind: every delta now carries the hand-vetted `curated/*` rows as well as the exclusion that already kept a competing row out ([§5](#curated--the-hand-vetted-rows), [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)); no column changed and no manifest key was removed. For the curated rows the delta between deploy and snapshot needs no such caveat — they are not keyed on a cutoff, so they cross it and arrive with the first ≥ 1.13.2 delta. The 1.13.1 suppression change that 1.13.2 inherits is cutoff-keyed like any masking change, so a 1.13.0 snapshot plus 1.13.2 deltas still retains the categories it withdraws: consume a full ≥ 1.13.2 snapshot (the 2026-09-13 set) before treating your copy as reflecting it. Columns 39–44 are **appended** at `feed_schema_version` **1.14.0**, whose first artifact is the **Thursday 2026-09-17** snapshot set (specified 2026-09-10; no export before that date carries them); columns 1–38 are unchanged in name, position and type. 1.14.0 is a MINOR bump rather than a PATCH because it does both things at once: it adds columns AND it moves the published row population — the language-NULL guard ([§10](#10-manifestjson)) starts suppressing categories on rows that carried one at 1.13.2, column 8 becomes derived rather than the stored cut, and the feed's policy pack moves to `policy_v2`. Read the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog) entry for 1.14.0 before ingesting the first artifact at this version, and **ingest a full snapshot at 1.14.0 before relying on 1.14.0 deltas**. Columns 45–46 are **appended** at `feed_schema_version` **1.17.0**, whose first artifact is the **Wednesday 2026-11-04** delta (announced 2026-11-02; nothing published before the Tuesday 2026-11-03 exporter deploy carries them); columns 1–44 are unchanged in name, position and type, and no row moves in or out of the feed. Unlike 1.14.0, **the deltas are enough** to receive them: the only rows that carry a value are the curated rows, and those ride every delta whole ([§14](#14-iab-content-purpose-columns-4546)). The full-snapshot obligation 1.16.0 set for its band re-rating still stands, and the 2026-10-11 set was the first to meet it. > **What `tier1..tier4` mean.** The stored label follows the ratified **single-label > convention**, which is **purpose-first**: it names the site's primary purpose — what the > site is _for_ — rather than its topic, industry or audience, and a small set of rules > (R2, R3, R5, R8 and R9) settles the recurring cases: a gambling home is > `Attractions > Casinos & Gambling` (the harmful subset ships as `Sensitive Topics` — see > [§5](#5-confidence-and-tier)); a software, SaaS or hosting vendor is `Technology & Computing` > whoever it sells to; a multi-brand store is `Shopping` while a brand's own store takes its > vertical; consumer banking, BNPL, investing and pensions are `Personal Finance`; a content > platform takes the content type it carries, not its delivery mechanism. The convention is > set by our labeling rubric; subject-matter and taxonomy-gap rules sit on top of it — news outlets take > `Politics`, since v3.1 has no News tier-1 — so the result is neither a pure topic label nor > a pure purpose label. The reserved `site_purpose_*` columns (27–35) were added so a > separately modelled business-purpose answer could be carried without changing this one; > eight of them stay NULL and `site_purpose_status` keeps its documented values — see > [§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection). --- ## 3. `domain_status` — the six lanes `domain_status` is a **crawl-observed status**, and it is **orthogonal to the content category**. It answers "what kind of thing is at this domain right now", not "what this site is for". There are exactly six values. They are closed — no other value ships. | Value | Meaning | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `active` | A live site with real content — including one that answered our conditional request with `304 Not Modified`, confirming the content we hold is current (from the 2026-09-17 set). **This is the only lane whose content category, where one is present, is a current one** — but the lane does not guarantee one; see below. | | `parked` | An affirmative parking signal — parking nameserver, marketplace landing page, for-sale page, or a capture that extracted to no readable text — the content fingerprint of a parking-provider redirect stub — except, from 2026-07-31, on a domain in popularity bucket `0` or `1` (§10; see `blocked`). | | `inactive` | The domain resolves but serves no real content and shows no parking signal: placeholder, coming-soon, under-construction. **DNS-dead domains are `dead`, not this.** | | `blocked` | **A live site we could not read on this crawl** — anti-bot challenge, `403`/`429`/`451`/`5xx`, timeout (including a DNS timeout or server failure), connection or TLS error, or, on a domain in popularity bucket `0` or `1` (best rank 1,000,000 or better on Tranco, Majestic or Umbrella; §10; not applied when `rank_ladder` is `absent`), a page that gave us no readable text in a capture made **from 2026-07-31** — usually one rendered entirely by JavaScript, a redirect or a bot interstitial. **This is not parking and not death**, with one exception: a minority of those empty captures are hosting placeholders or parking pages our crawler could not recognize (more often in bucket `1` than in bucket `0`; see the changelog). The lane's share of the feed moves with the crawler's re-crawl cadence for blocked sites; the changelog records material moves. A `304 Not Modified` answer to our conditional request is **not** `blocked`: the site confirmed the content we hold is current, so from the 2026-09-17 set the row is laned as a successful crawl is — `active` where a label is on file, `uncategorizable` where no label exists, and `parked` or `blocked` by the rule above where the content it confirms extracted to no readable text. | | `dead` | The name does not resolve (a definitive NXDOMAIN or no-address answer, in crawls from 2026-09-23), or `404`. Not a website. | | `uncategorizable` | Either crawled successfully but no label could be assigned, or the underlying crawl status was not recognized. This is a deliberate fail-loud bucket, never a silent default. | **Known issue (October 2026): some live sites ship as `parked` or `inactive`.** A site whose page our crawler captured empty — often one that builds its page with JavaScript — can carry the parking-stub fingerprint and ship `parked`, with its label withheld (see `parked_stub_suppression` in [§10](#10-manifestjson)). From the 2026-11-03 deploy, on a domain ranked in the top 1M (popularity bucket `0` or `1`) such a capture made from 2026-07-31 ships `blocked` instead, and the 2026-11-08 set is the first to carry every such row. Such a capture made before 2026-07-31 stays `parked` until a crawl fetches the page again (a `304 Not Modified` answer keeps the earlier capture), and below that rank such rows stay `parked`. Separately, some live sites were laned `inactive` by an earlier placeholder rule that misread them, and they stay `inactive`, with their label in `last_known_*`, until the crawler reads them again. Do not cull a domain on `parked` alone, nor on the real-time API's `is_parked`, which is `true` for `inactive` answers too. Three points that matter for integration: 1. **`blocked` is not `parked`.** A blocked domain is a real, live website that defended itself against our crawler. Treating `blocked` as parked or dead will discard live sites. This separation is a deliberate design invariant. 2. **`uncategorizable` is a fail-loud bucket, not a category.** It means "we will not guess". It covers both a clean crawl that produced no label and an inventory status our mapping does not recognize. It is never used to quietly absorb an unknown into `active`. 3. **Only `active` rows carry a current content category — but not every `active` row carries one.** On `parked`, `inactive` and `dead` rows every content column (columns 2–8 and 10–14, plus `openrtb_cat`, the reserved `site_purpose_*` label columns and the three `policy_*` columns) is NULL **by design**. That is not missing data — see [§9](#9-why-a-non-active-row-has-no-category). The implication runs one way only: `active` says the crawl found a live site, not that a label was produced or kept for it. A live site can reach the export with every content column NULL — because it was never labelled, or because one of the suppression guards withheld the label it had (the stub, no-evidence, template and demoted-label guards described below and in [§10](#10-manifestjson)). Test the columns you need (`tier1 IS NOT NULL`); do not infer them from `domain_status = 'active'`. `blocked` rows are the exception worth knowing: they **keep** the category from a previous successful crawl, because the site is presumed live and unchanged. That label is real but was not re-verified on this crawl. **Do not drop `blocked` rows as category-free** — after `parked`, this is the largest non-active lane. Most of those carried labels are `DOUBTFUL`, as most labels on `active` rows are, and a model-adjudicated read of the 2026-10-18 set found them less often acceptable than labels on `active` rows, on a small sample: filter them on `calibrated_confidence_tier` rather than keeping or dropping the lane wholesale. `uncategorizable` rows likewise keep whatever label is already on file, if there is one: where the lane was reached because the crawl status was not recognized, an existing label is retained rather than blanked. **From artifact 1.15.0 a `blocked` or `uncategorizable` row may carry a tier-1 category that was never read from the page:** the name-only LLM lane (`model_name = llm-name/consensus_v1`, `policy_confidence_basis = name_only`; see [§5](#5-confidence-and-tier) and [§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection)) labels head domains (Tranco rank ≤ 100,000) the crawler cannot read from the domain name, where two model families agree. The torso tranche (rank 100,000–1,000,000) this lane first reserved was measured and withheld for now — see the 1.15.0 notice in the changelog — and nothing ranked past 1,000,000 is ever labelled this way. The lane the crawl decided is unchanged by it: the label is additional information about the domain, not evidence that the site was read. (When we checked on 2026-09-30, a small number of these rows, whose crawl produced no label from the page, would ship in the lane a label read from the page would give them: some as `active`, with the label, and others as `parked`, without it. From `delta_2026-10-02_100000Z` and the 2026-10-04 set they keep the lane the crawl decided; the changelog's 2026-10-01 and 2026-10-02 entries have the detail. To find a name-only label, test `policy_confidence_basis` or `model_name`, not `domain_status`.) On `parked`, `inactive` and `dead` rows such a label is withheld like any other, with no `last_known_*` block. **One exception applies on every lane, including `blocked` and `uncategorizable`:** rows whose latest capture extracted to no readable text (see `parked_stub_suppression` in [§10](#10-manifestjson)) ship with NULL category columns wherever they land. No label can be checked against such a capture — see the no-content note in [§5](#5-confidence-and-tier) — so it is withheld rather than retained, with no `last_known_*` block either. Where the crawl itself succeeded and the label is not a name-only one (which never decides the lane; see above), such a capture ships `parked` (it is read as a parking-provider redirect stub), except that one made from 2026-07-31 on a domain in popularity bucket `0` or `1` (§10) ships `blocked` (read as a live site we could not read; a minority are hosting placeholders or parking pages). An export without the buckets (`rank_ladder` `absent`) makes no such exception. So a `blocked` row with a NULL category is either a site that was never labeled or one of these rows; in both cases the NULL is deliberate. ### `curated/*` rows: `active` is asserted, not crawl-observed Rows whose `model_name` begins with `curated/` (see [§5](#5-confidence-and-tier)) are the one place where `domain_status` is **not** derived from a crawl. They always carry `active`, because a human placed the domain on the curated list and that list is the same first-tier lookup the real-time API answers from — so the feed and the API give you the same answer for those domains. The list includes CDN, DNS and API zones that serve no homepage (`akamaiedge.net`, `cloudfront.net` and `googleapis.com`, for example): their `active` is the curated assertion, not a sign that a website answered. Two consequences, stated plainly rather than hidden: - **The lane is a statement about the curated entry, not about the last crawl.** These are disproportionately the sites that defend hardest against crawlers, which is precisely why they were curated; deriving their lane from a crawl would bury a human-verified answer under a `blocked` verdict. - **`status_observed_at` on these rows still dates the last crawl attempt**, and is NULL where the domain has no crawl record at all. It is the one column on a `curated/*` row that reports observation rather than assertion, so do not read it as dating the `active` lane. Curated rows are a small share of the feed. If you need to exclude or specially handle them, filter on `model_name LIKE 'curated/%'`; `manifest.json` publishes the count as `row_count_curated`. **`www.` hosts:** a curated domain ships **once**, under its bare (no-`www.`) name. The `www.` spelling of a curated domain is not published at all — not as a curated row and not as a model-produced one — so it cannot contradict the bare host. If your lookup key can carry a `www.` prefix, strip it before matching, which is what our real-time API does with the requested domain. --- ## 4. The two timestamps The feed carries two dates and they mean different things. This is the single most common misreading, so it is stated plainly: | Column | What it dates | | -------------------- | ------------------------------------------------------------------ | | `classified_at` | When **we processed the domain** and produced or served the label. | | `status_observed_at` | When **the crawler observed** the `domain_status` on that row. | **Neither is "when the website's content changed."** Website Categorization does not publish a content-change date, and neither column should be used as a proxy for one. - `classified_at` records our processing, not the site's. In our own code comments: _"classified_at only records when webcat served or cached the verdict."_ A recent `classified_at` does **not** mean the page was recently re-fetched. - `status_observed_at` is the crawl-attempt timestamp behind `domain_status`. It is populated on every row the crawler has observed, in every lane, and the exceptions are visible rather than silent: a row whose inventory carries a status but no attempt timestamp ships with it null and is counted, per lane, in the manifest's `row_count_by_status_age` `null` bucket (artifact ≥ 1.14.0), and a curated row carries it only where the inventory has a row for that domain. Use it to judge how old a `parked` / `blocked` / `dead` verdict is. - On rows where a historical label is present, `last_known_classified_at` dates **that label**, and is generally older than both of the above. - **How old the status observations are** is published in the manifest rather than left to a scan (artifact ≥ 1.14.0): `row_count_by_status_age` buckets every row, per lane, by `status_age_reference_date` minus the UTC calendar day of `status_observed_at`, and `stale_fraction_active_gt_90d` is the `active` lane's share observed more than 90 days before that reference — see [§10](#10-manifestjson). The reference is the artifact's **key date** (a snapshot's date, a delta's cutoff day), never the time the export happened to run, so a re-published artifact reports the same ages. - `classified_at` is usually the later of the two. Outside the top 1M, a re-crawl re-labels a row only when its content hash changed. --- ## 5. Confidence and tier ### `confidence` and `confidence_tier` `confidence` is a float in `[0, 1]`, published to 4 decimal places. `confidence_tier` is its discrete band: | `confidence_tier` | Condition | | ----------------- | ------------------------- | | `HIGH` | `confidence >= 0.9` | | `MEDIUM` | `0.7 <= confidence < 0.9` | | `LOW` | `confidence < 0.7` | `confidence_tier` is NULL wherever `confidence` is NULL — that is, on every row with no current content category. #### From artifact 1.14.0 the band is derived, not the raw cut **Through artifact 1.13.x** the table above is the whole story: `confidence_tier` is that cut over the stored `confidence`, passed through verbatim. **From artifact 1.14.0** `confidence_tier` is a band **derived at export under a named policy** over three inputs — the **writer** (`model_name`, column 10), the **label** (`tier1`, column 2) and the **calibrated tier** (`calibrated_confidence_tier`, column 25) — and from artifact 1.16.0 over a fourth, the **provenance suffix** of `model_version` (column 11). The reason is the rest of this section. `confidence` is not one scale, so a single cut applied to all of them published a `HIGH` meaning "six gambling keywords appeared on the page" beside a `HIGH` meaning "the classifier's own probability for this label was 0.93". The policy re-scores nothing; it decides which of the four documented words each row's band is entitled to. `manifest.json` says which policy produced an artifact, in `confidence_band_policy` ([§10](#10-manifestjson)). A manifest **without** that key is the pass-through. **What does not change.** The value domain of this column is unchanged: `HIGH` / `MEDIUM` / `LOW`, plus `PARKED`. No other column moves with it — `confidence` (column 7), the whole `calibrated_*` block, `tier1`–`tier4`, `iab_id`, `model_name` and `policy_group` are exactly what they would have been under the old cut. Because `confidence` is untouched, the old band is rebuildable from it on any row; the recipe, and the boundary arithmetic it needs, are in the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog). **What each writer can band.** One row per `model_name` family, in the same order as the scale table below. `PARKED` and NULL are decided before any of this: a stored `PARKED` is the write side's parked flag rather than a confidence, and a row with no band has none to re-rate, so both pass through on every family. | `model_name` family | Bands reachable from artifact 1.14.0 | | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Trained classifiers** — the ensemble family (`ensemble-*`, `multitask-ensemble-*`, any `model_version`) | `HIGH` / `MEDIUM` / `LOW`, with two corrections. `tier1 = Business and Finance` **never bands `HIGH`**: `MEDIUM` where `calibrated_confidence_tier` is `TOSSUP` or `PROBABLE`, `LOW` otherwise — so that population reaches `MEDIUM` / `LOW` only. On every other label the 0.9 / 0.7 cut stands, except that a row the cut put in `LOW` bands `MEDIUM` where its `calibrated_confidence_tier` is `TOSSUP` or `PROBABLE`. That is a promotion only: `HIGH` and `MEDIUM` are never demoted by it. **From artifact 1.16.0** a third correction applies, keyed on `model_version` rather than on the label: where its provenance suffix (`split_part(model_version, '+', 2)`) is `teacher_bf1`, `nocontent_bf1` or `teacher_purpose_v2`, the row **never bands `HIGH`** — `MEDIUM`, and `LOW` where `calibrated_confidence_tier` is `DOUBTFUL` (rule 4, `suffix_caps`, policy `v4` — the pin the 1.16.0 release flips to). Those rows carry a teacher's re-scoring of the classifier's call, or its "nothing to read" marker, rather than the classifier's own reading. A cap only: nothing is promoted by it, and the route stamps `ml_en` / `ml_non_en` and the bare base version are outside it. | | **Trained classifiers** — `cascade/student`, `cascade/teacher_fallback`, other model names | `HIGH` / `MEDIUM` / `LOW`. The 0.9 / 0.7 cut, unchanged: neither correction above applies, because both are keyed on the ensemble writer names. | | **LLM teacher** — `cascade/teacher` | `HIGH` / `MEDIUM` / `LOW`. The cut over the model's self-reported number, unchanged. | | **Sensitive-content filter** — `pipeline/st_filter`, `cascade/st_filter` | **`LOW` only** (rows written before the annotate flip — legacy). Its score is a sensitive-vs-not probability, not a probability for the tier path, so it can never entitle a row to `HIGH` or `MEDIUM`. `cascade/st_filter` **is** covered, for the same reason: the rule matches five whole `model_name` values — both spellings of this family, both spellings of the gambling filter, and `pipeline/adult_filter` — and never matches as a prefix. Those five are not the same set as `filter_model_names` in [§10](#10-manifestjson), which is the demoted-label guard's and names the three `pipeline/` filters plus `pipeline/vendor_rule`. | | **Gambling filter** — `pipeline/gambling_filter`, `cascade/gambling_filter` | **`LOW` only** (rows written before the annotate flip — legacy), for the same reason — its `confidence` is a keyword density. `cascade/gambling_filter` is covered too, on the same rule. | | **Adult-content filter** — `pipeline/adult_filter` | **`LOW` only** (rows written before the annotate flip — legacy), for the same reason — its `confidence` is a rule score. | | **Curated lists and rules** — `curated/*`, `tld_rule/*`, `schema_rule/*`, `human-feedback` | `HIGH` / `MEDIUM` / `LOW`: the ordinary cut over the fixed editorial trust value, unchanged. A `.edu` row at `0.90` still bands `HIGH`, and a curated entry that carries no stored band takes the same cut over its own `confidence`. A `human-feedback` row — a reviewer's accepted verdict, written at a fixed `1.0` — takes that same cut and bands `HIGH`: no rule of the policy is keyed on that writer. | | **Legacy upstream fallback** — `v3_historical_fallback` | `HIGH` / `MEDIUM` / `LOW`. The cut over the carried-over upstream score, unchanged. | | **Vendor-rule flip** — `pipeline/vendor_rule` | `HIGH` / `MEDIUM` / `LOW`: the classifier's own band, carried across the flip unchanged. It is neither an ensemble writer nor one of the five brand-safety filter names, so no rule reaches it. | | **Name-only LLM lane** — `llm-name/*` (artifact ≥ 1.15.0) | `MEDIUM` / `LOW` only. The ordinary cut over the lane's confidence, then **capped at `MEDIUM`** by band-policy rule 5 (`name_only_never_high`, policy `v3` — the pin the 1.15.0 release flips to): the label was inferred from the domain name by two language models that agreed, never from the page, so the lane never publishes `HIGH`. Exclude it with `WHERE COALESCE(policy_confidence_basis, '') <> 'name_only'` — the `COALESCE` is load-bearing, because status-only rows publish that column as NULL and an unguarded `<>` or `NOT LIKE` drops them too. | The three `LOW`-only families above describe the pipeline as it runs today, where a fired brand-safety filter **replaces** the classifier's label. That will change: the filters will annotate their verdict beside the classifier's own label instead of overwriting it, and rows written after that change will carry the classifier's label, `model_name` and band, leaving the rule to match only the rows written before it. The [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog) will carry the date — nothing in this section moves silently. **Blockers: key on `policy_group` and `model_name`, never on the band.** Neither of those two keys survives the annotate flip described in the paragraph directly above unchanged, and the order matters if you are building an allow/block list: 1. **`policy_group` (column 36) follows the primary label.** It is the projection built for exactly this purpose and the right key to start from, but it is a pure function of the stored tier path ([§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection)) — so it is stable only insofar as that label is, and **the annotate flip moves it**. A page a brand-safety filter used to relabel keeps the classifier's label from the flip onward, and the group follows the label: a gambling-positive page stored today as `Attractions > Casinos & Gambling` (group `gambling`) ships under whatever the classifier called it — for example `Video Gaming`, which projects to `games`. So if you block on `policy_group = 'gambling'` today, **expect that set to shrink at the flip** by the pages the classifier files elsewhere, and re-key on `safety_flags` when it ships. 2. **`model_name` (column 10) identifies filter-decided rows _today_, and is not a durable block signal.** It tells you which component decided the row, which is what makes it usable now; but after that same annotate flip a filter-flagged row carries the _classifier's_ `model_name`, and a list keyed on the `pipeline/*` names stops finding it. Use it, and plan to migrate off it. 3. **`safety_flags` is the successor key.** The filter's verdict stays on the row whatever label the row keeps, so it is the one signal the flip does not move. It arrives with the brand-safety columns at artifact **1.14.0** ([§13](#13-the-brand-safety-axis-columns-3941)). The pack projection follows it: under `policy_v2` [`website-categorization-policy-pack-v2.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v2.md) `policy_group` is read from the flagged safety themes _before_ the label, which puts a flagged row back in its block-critical group with `policy_basis` = `safety` when the classifier's label is **not itself in a sensitive group** — the `Video Gaming` row above reads `gambling` / `safety`. Where the label already resolves to a _different_ sensitive group, the label answers instead (`policy_basis` = `primary`): a `gambling`-flagged page stored as `Sensitive Topics > Adult & Explicit Sexual Content` reads `adult` / `primary`. The one sensitive group that does not defer is the theme-unknown bucket `sensitive_other`, which a theme-specific verdict refines (`gambling` on a bare `Sensitive Topics` label reads `gambling` / `safety`). So **`policy_group = 'gambling'` still does not capture every gambling-flagged row even under `policy_v2`** — key on `safety_flags` itself if you want all of them. Below artifact 1.14.0 there is no `safety_flags` column and `policy_group` reads the label and nothing else; from 1.14.0 both ship, and the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)'s 1.14.0 entry names them. `confidence_tier` is not, and has never been, a safety signal: under the old cut a filter row's `HIGH` said only that a rule fired hard, and under the new policy that same row bands `LOW` while its label, its group and its writer are untouched. A blocker keyed on `confidence_tier IN ('HIGH','MEDIUM')` therefore loses rows at the 1.14.0 transition that it never had a reason to be keyed on in the first place. The changelog entry for 1.14.0 records the transition [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog). #### The scale depends on `model_name` — check column 10 first `confidence` is written by whichever component produced the label, and **the number does not mean the same thing on every row.** `model_name` (column 10) tells you which component that was. Nine families ship. **Only the first is a model probability for the label**; the vendor-rule flip (last row) carries that same probability across unchanged, and every other family is something else: | `model_name` | What `confidence` measures | Range | | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | | **Trained classifiers** — `ensemble-5en-3nonen`, `cascade/student`, `cascade/teacher_fallback`, other model names | The model's own probability for the tier-1 label. Uncalibrated — the rest of this section describes these rows. | `[0, 1]` | | **LLM teacher** — `cascade/teacher` | The language model's **self-reported** confidence, not a calibrated output distribution — and where it returned a tier-2 label, the **average of its tier-1 and tier-2 self-reports**, so it describes the path rather than tier 1 alone. | `[0, 1]` | | **Sensitive-content filter** — `pipeline/st_filter`, `cascade/st_filter` | A **binary** brand-safety model's probability that the page is sensitive content. It scores sensitive-vs-not, **not** the tier path. Always `tier1 = Sensitive Topics`. | ≥ the firing threshold, `0.50` today | | **Gambling filter** — `pipeline/gambling_filter`, `cascade/gambling_filter` | **Not a probability.** A keyword-density measure: gambling keyword matches ÷ (2 × the firing minimum), capped at `1.0`. Always `Attractions > Casinos & Gambling`, or `Sensitive Topics > Spam or Harmful Content` where illegal-gambling markers were found. | `[0.5, 1.0]` | | **Adult-content filter** — `pipeline/adult_filter` (artifact ≥ 1.10.0) | **Not a probability.** A multilingual rule layer: `0.95` on an adult-platform marker in the domain or title, `0.90` on an explicit title term, else adult-keyword density capped at `1.0`. Always `Sensitive Topics > Adult & Explicit Sexual Content`. | `[0.5, 1.0]` | | **Curated lists and rules** — `curated/*`, `tld_rule/*`, `schema_rule/*`, `human-feedback` | **Not a probability.** A fixed trust value attached to the list entry or rule that matched, chosen when it was written. It does not vary with the page and was not measured against outcomes. A `human-feedback` row carries the fixed `1.0` an accepted reviewer verdict is written at. | rule-defined, `[0, 1]` | | **Legacy upstream fallback** — `v3_historical_fallback` | A confidence **carried over from the previous-generation (v3) WhoisXML categorization** for that domain, crosswalked into IAB v3.1. It is that older system's score, not this pipeline's, and was produced under a different methodology. | upstream-defined, `[0, 1]` | | **Vendor-rule flip** — `pipeline/vendor_rule` (real-time API from 2026-09-01) | The trained classifier's own probability, **carried across unchanged** — the rule rewrites a `Business and Finance` label to `Technology & Computing` on vendor sites; it does not re-score. Always `tier1 = Technology & Computing`, `tier2`–`tier4` NULL. | `[0, 1]`, the classifier's own | | **Name-only LLM lane** — `llm-name/consensus_v1` (artifact ≥ 1.15.0) | The **lower of two language models' self-reported probabilities** that the domain name alone identifies the site's tier-1 category; a row ships only where the two models agreed on the tier-1, each at ≥ 0.7. **Not a page-derived probability** — no content was read. Band capped at `MEDIUM`; basis `name_only`. | `[0.7, 1]` | The other eight families are self-reports, rules, thresholds, lookups, carry-overs and name-only inferences — several of them decide a row outright rather than asking the trained classifier. Website Categorization publishes each component's own score unchanged rather than rescaling them onto a common axis, so **`confidence` is not comparable across `model_name` values.** If you rank or threshold on this column across the whole feed, you are mixing eight scales. **Nine families, eight scales — the counts differ by one on purpose.** The vendor-rule flip does not score anything: it republishes the trained classifier's own probability under a rewritten label, so it rides the first family's scale rather than adding one of its own. Nine `model_name` families ship; eight mutually incomparable scales sit underneath them. If you encounter a `model_name` not listed above, treat its `confidence` the same way: a component-internal score, not a probability, and not comparable to other rows. #### The sensitive-content filter's video-streaming bypass The sensitive-content filter has one documented exception, and it changes which rows you will find under `Sensitive Topics`. Video-streaming catalogues carry `rated` / `mature` / `explicit` metadata **about** the titles they list, and the filter reads that as the page itself being sensitive. So when all three of the following hold, the filter's verdict is discarded and the page falls through to normal classification: 1. the filter fired, **and** 2. its score is in the narrow band **`[0.50, 0.65)`** — at `0.65` and above the filter stays authoritative even on a streaming domain, **and** 3. the domain matches a video-streaming pattern: an explicit list of well-known streaming brands, or a generic `*video[s].(com|net|tv|io|co)` suffix. Both places the filter can short-circuit a page apply the same rule, so it holds for `pipeline/st_filter` and `cascade/st_filter` alike. What that means when you read the feed: a bypassed row carries the classifier's ordinary label and `model_name`, **not** `*/st_filter` and **not** `tier1 = Sensitive Topics` — so it also does not project to the `sensitive_other` policy group (see [§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection)). Counting `sensitive_other` or `Sensitive Topics` rows therefore undercounts what the filter fired on, by exactly this class. The band and the pattern list are operational settings tuned against measured false positives, not part of the schema contract. #### `pipeline/vendor_rule` — the vendor-rule flip The trained classifier over-generalizes software, SaaS, hosting, IT-services and telecom **vendor** sites into `Business and Finance`. A deterministic post-classifier corrects the measured part of that: when the classifier has labeled a page `Business and Finance` in one of three vendor-heavy tier-3 cells (`Business I.T.`, `Technology Industry`, `Telecommunications Industry`) **and** the page text carries high-precision vendor-identity signals — a vendor describing its own product or service — the row is rewritten to `tier1 = Technology & Computing` with `tier2`–`tier4` NULL and `iab_id = IAB-596`. Field by field: | Column | On a `pipeline/vendor_rule` row | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `model_name` | `pipeline/vendor_rule` — filter on it to find every flipped row. | | `model_version` | the classifier's version (`v1`, `v1+ml_en`, …): the flip carries the classifier's decision rather than replacing it. | | `confidence`, `confidence_tier` | the classifier's own probability and band, unchanged — a `HIGH` row stays `HIGH`. | | `calibrated_*` | NULL. The calibration map was fitted on the classifier's own labels; the flip changes the label, so the map does not apply. | | `tier`, `classified_at`, `language`, `domain_status` | as on any classifier row. | The flip is one-directional: it can only remove a `Business and Finance` label from those three cells, never add one, so banks, finance news and every other `Business and Finance` row are untouchable regardless of text. The rule was measured against a reserved holdout before it was enabled, and the trade it makes — vendor pages corrected against legitimate `Business and Finance` pages moved — is deliberate; the figures are held in our internal evaluation record rather than published here, and the change is dated in the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog). The rule has been enabled in the real-time API since 2026-09-01 23:00 UTC, and the bulk categorization lanes have applied it since 2026-09-02 (see the 2026-10-21 entry in the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)). Filter on `model_name = pipeline/vendor_rule` to find every flipped row. It was **not** applied by the September 2026 re-categorization pass (changelog, 2026-09-02). #### `model_version` can carry a `+variant` suffix `model_name` names the layer that decided the row; `model_version` (column 11) names the version of that component, and may carry a `+` suffix identifying which variant **within** the layer produced it. Four vocabularies share that slot and their values do not overlap: | Suffix | Where it comes from | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `+ml_en`, `+ml_non_en` | Bulk classification: which of the two language-routed ensembles handled the page. | | `+`, e.g. `+groq` | Real-time API classification: which component answered. Mirrors the row's internal routing name. | | `+teacher_bf1`, `+nocontent_bf1`, `+teacher_purpose_v2` | A re-scoring pass over an existing label, its "nothing to read" marker, and the purpose pass that follows it (the 1.16.0 notice in the feed changelog). | | `claude-sonnet-5+gpt-5.2+p:9cbd883f` | The name-only LLM lane (`llm-name/consensus_v1`): the two language models that agreed and the prompt version that asked them. | Read the version as everything before the first `+`, and the variant as everything after it. A name-only row is the exception: every `+`-separated part of its value is provenance (two model names and a prompt version), and none is a version of ours. A row can legitimately carry no suffix — that means no variant was recorded, not that a variant is missing. > **Do not match `model_version` with `=`.** A consumer filtering > `model_version = 'v1'` silently drops every `v1+ml_en` and `v1+ml_non_en` row. Compare > on the base version — split on the first `+` — or match with a prefix. The suffix is provenance, not quality: it says which variant ran, never how reliable the result is. From artifact 1.16.0 one band policy reads it all the same: the three re-scoring suffixes cap the published band (§10, `confidence_band_policy` `v4`). The two bulk routes in particular report `confidence` on scales that are **not** comparable to each other, which is the effect described above. The curated-and-rules family is worth a concrete example, because its values look exactly like model probabilities and are not. A `.edu` domain matched by the TLD rule carries `confidence` of exactly `0.90` on every such row — `confidence_tier = HIGH` by construction, because someone decided `.edu` implies Education, not because that row was scored. Only rules deliberately marked strong enough to replace the model emit a row at all, so in practice `tld_rule/*` rows carry `0.90` or `0.85` and come from a short list of education, military and pharmacy TLDs; weaker rules feed the model as a hint and never appear in this column. These numbers are editorial judgements about the rule, expressed on the same `[0, 1]` axis as a model score. #### `curated/*` — the hand-vetted rows The curated family is worth its own paragraph, because it is the only one where the row did not come from a classifier at all and where several columns therefore mean something slightly different. A curated row exists because a person put that domain on a list; `model_name` names the list batch (`curated/manual_curation_2026-05-31`, `curated/tranco_probe_2026-06-29`, …). Field by field: | Column | On a `curated/*` row | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_name` | `curated/`. The batch tag usually carries the date the entry was curated. This is the same string the real-time API returns for the domain. | | `model_version` | the constant `curated` — the curated list has no released versions. Never carries a `+variant` suffix. | | `tier` | `3` (human-reviewed) on every curated row, which is also why they win conflicts. See [`tier`](#tier). | | `confidence` | the fixed editorial trust value written with the list entry. **Not a probability**, exactly as the table above says for this family. | | `confidence_tier` | the ordinary band of that value (`HIGH` / `MEDIUM` / `LOW`), computed from the same thresholds as any other row. | | `classified_at` | the date the entry was curated, where the batch tag records one; otherwise the export's own key date (a snapshot's date, a delta's `since` cutoff date). | | `language` | always NULL — no page language is recorded for a curated entry. | | `domain_status` | always `active`, by assertion — see [§3](#3-domain_status--the-six-lanes). | | `calibrated_*` | always NULL. The calibration map is fitted on classifier outputs; there is no fitted curve for an editorial trust value, and the feed will not invent one. | | `last_known_*` | always NULL — nothing is being re-presented as historical. | **Delivery:** curated rows ship in **every snapshot and every delta** (artifact ≥ 1.13.2; before 1.13.2 they were in snapshots only). A delta carries the whole curated list, not a changed subset — the list is not keyed on your `--since` cutoff — and it still never carries a competing row for a curated domain, so applying deltas with your usual upsert-by-domain both delivers the curated answer and cannot overwrite it with a model-produced one. The same rows arrive in every delta; that is deliberate, and harmless under upsert-by-domain. **Apply curated rows by domain, not by `classified_at` order.** A curated row is dated when the entry was **curated**, which is routinely _earlier_ than the model-produced row it replaces. If your loader keeps the newest `classified_at` per domain rather than the newest row received, it will discard the curated answer and keep the model's. Match on `domain` and take the row from the most recent artifact. This does not relax the snapshot-ingest obligation ([§10](#the-snapshot-ingest-obligation-contract-term)): removals still travel only in a full per-partition replace. The gambling family is the one most likely to mislead, because its band assignment is pure keyword density. It fires only when **two** conditions both hold — currently at least **3 keyword matches in total** (repeats counted) **and** at least **3 distinct keywords**. Repetition alone is never enough: a page saying one gambling term ten times does not fire, because only one distinct keyword matched. The published `confidence` is then computed from the **total** match count only, as total ÷ (2 × the total-match minimum), capped at `1.0`. So for a row that has already cleared the distinct-keyword requirement: | Total keyword matches | `confidence` | `confidence_tier` | | --------------------- | ------------ | ----------------- | | 3 | `0.5` | `LOW` | | 4 | `0.6667` | `LOW` | | 5 | `0.8333` | `LOW` | | 6 or more | `1.0` | `LOW` | From artifact 1.14.0 the band reads `LOW` on every one of these rows (the family table above says why; a stored `PARKED` flag still passes through, as on every family); through 1.13.x the raw cut banded `0.8333` `MEDIUM` and `1.0` `HIGH`. So a `confidence` of `1.0` on a `*/gambling_filter` row means **"six or more gambling keyword matches appeared on the page"** — not "90%+ likely to be correct". Do not compare it against an ensemble row's `0.9`, and do not read `0.5` there as a coin flip: `0.5` is the _minimum_ a firing row can carry, i.e. the rule fired at exactly its threshold. Every threshold named above — the sensitive-content filter's `0.50`, and the gambling filter's total-match and distinct-keyword minimums, which are tuned independently of each other — is an operational setting, not part of the schema contract. Any of them can be retuned without a schema change, which is another reason to treat these rows' `confidence` as a component-internal score rather than a comparable quantity. The sensitive-content filter carries a second operational setting worth knowing, because it decides **whether a page is looked at at all**: a page whose body text exceeds **1,500 characters is not scored by the filter**, and no `*/st_filter` row can be produced for it. It is skipped outright rather than truncated and scored. Both the real-time and the bulk lanes run the filter at `0.50` with the same 1,500-character limit. The consequence for a consumer is the same as for the bypass above: the absence of an `*/st_filter` row is not evidence that the filter cleared the page. What the rest of this section says about `confidence` describes ensemble-classified rows. **It does not describe rows decided by any of the other seven families.** None of those has been ground-truthed on delivered rows; the one independent read among them is the vendor-rule flip's reserved-holdout measurement, which scores the rule's flips rather than the accuracy of shipped rows. No share is published for how many rows each `model_name` accounts for — group on column 10 in the delivery you actually received rather than assuming one. #### The bands are not a reliability ranking **What this subsection describes, and what it does not.** The measurements behind it were made on bands cut the old way — the stored 0.9 / 0.7 cut, passed through — so they describe the column as artifact 1.13.x and everything before it ships it. They are also exactly the evidence behind the derived band described above: the `HIGH` band they measure is the one that still contains rule-filter rows whose score was never a label probability. From artifact 1.14.0 the composition of all three bands moves, so **read what follows as the diagnosis, not as a description of a 1.14.0 artifact.** A re-measurement of the derived bands is under way and is not published until it has been adjudicated. The guidance underneath it — take `HIGH`, or take everything — is unchanged in the meantime, precisely because nothing here has been re-measured yet. The table above is a **definition**, not a claim about accuracy. Two independent measurements against the production corpus — a stratified sample re-weighted to the feed's true language-and-confidence mix, and a separate sample of well-known domains — agree on the conclusion, and neither is published on this page: **the bands do not deliver the ranking their names suggest.** Feed-wide, `MEDIUM` and `LOW` are statistically indistinguishable, so the tier system's real information is `HIGH` vs everything else. On well-known domains the picture is sharper and worse for `MEDIUM`, which measures _below_ `LOW`, driven by the non-English over-confidence described below. One population needs naming here: rows whose crawl captured no title and no body text — overwhelmingly parked-domain redirect stubs the crawler recorded as successful. **On these rows the category was derived from the domain name alone and should not be relied on.** They concentrate in `MEDIUM`. As of artifact version 1.5.0 the dominant class of these — a single parking-provider redirect stub, identified by a constant content fingerprint — ships on the `parked` lane with **no content category** (see `parked_stub_suppression` in [§10](#10-manifestjson)), so the measured composition describes the corpus as measured rather than what current exports deliver. A small residual of other no-content shapes remains in-band; the accuracy conclusion applies to the content-bearing remainder either way. So the two most natural ways to use this column are both unsafe: - **Do not filter to `confidence_tier IN ('HIGH','MEDIUM')`** expecting a quality gain — `MEDIUM` is at best indistinguishable from `LOW`, is _worse_ than `LOW` on well-known domains, and is the band most contaminated by no-content rows. - **Do not drop `LOW`** expecting to shed the unreliable tail — it is the largest band and no less accurate than `MEDIUM`. `HIGH` is the only band that behaves like its name. If you want a single rule: **take `HIGH`, or take everything — but do not treat `HIGH` + `MEDIUM` as a quality filter.** #### Why the bands misbehave On ensemble-classified rows `confidence` is the classifier's own probability for the tier-1 label, and it is **not calibrated**: a `confidence` of 0.75 does not mean a 75% chance of being right. Three properties of the current pipeline drive it. - **A large no-evidence population.** A material share of feed rows carries no crawled text at all (title and body both empty — dominated by parked-domain stubs recorded as successful crawls). The classifier still emitted a category and an ordinary-looking confidence for them, from the domain name alone, and those rows concentrate in `MEDIUM`. From artifact version 1.5.0 the dominant class of these rows — the constant-fingerprint parking stub — ships on the `parked` lane with no content category (see `parked_stub_suppression` in [§10](#10-manifestjson)), so current exports deliver fewer of them than the corpus as measured contained. - **Two models, two scales.** Pages are routed by language to one of two model ensembles, and the two report confidence on scales that are not comparable: at comparable accuracy the English ensemble reports markedly lower confidence than the multilingual one. Because the bands are cut at fixed thresholds, a non-English row reaches `HIGH` far more often than an English one, and most `HIGH` rows are non-English. The band sorts partly by language, not by reliability. - **One category dominates the middle.** `Business and Finance`, one of the largest categories in the feed, carries systematically higher confidence than its accuracy supports and concentrates in the middle of the scale. Most other categories run the other way — `Technology & Computing`, for instance, is accurate well above the confidence it reports. - **A known-inflated leaf: `Web Hosting`.** A graded audit of the stored rows carrying the tier-4 `Web Hosting` label found the great majority of that class to be placeholder inflation: the crawl captured a hosting provider's parking page, a stock server default, or an empty page, and the label described the HOST's boilerplate rather than the domain. The class runs DOUBTFUL-heavy, so the calibrated bands already de-weight it. From artifact 1.10.0 the template-provable share ships status-only (see `boilerplate_template_suppression` in [§10](#10-manifestjson)); the residue drains through re-crawls under the fixed categorize-time detector, so expect the shipped `Web Hosting` population — and with it the overall categorized share — to shrink toward honest numbers over the following months. - **A known-unsupported class: ensemble labels with no `language`.** A minority of categorized rows per partition carried an ensemble label with `language` NULL behind a captured content hash — rows the no-evidence guard deliberately spared on the theory that a hash proved content. A read of the class itself found a large share of it dead or contentless at fetch, and a blind judge upheld very few of the re-scored labels among the survivors, against a control of same-lane labels the re-score had left unchanged: the labels are bare-domain guesses, not content reads. **From artifact 1.14.0 the class ships status-only** — category columns NULL, no `last_known_*` block, `domain_status` unchanged (see `language_null_suppression` in [§10](#10-manifestjson) and the changelog [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)). Two populations are exempt and ship unchanged: rows whose label projects to a block-critical policy group (held under the same exposure rule as `demoted_label_suppression.block_critical_held`, until an in-group read clears the group) and rows in the top 1M by popularity rank. Expect the categorized share to fall on the first ≥ 1.14.0 snapshot, and a delivery you receive before that snapshot still carries the class, disclosed under this same heading. Filter-derived rows (`pipeline/*`), rule-derived rows and curated rows carry NULL `language` by design and are never affected. #### What `confidence` does not cover The measurements behind this section were made against the production corpus and are not published on this page. What they cannot tell you applies regardless: - The reference labels behind them are **LLM-generated, not human-adjudicated**. They measure agreement with an independent judge. - They cover **tier 1 only**. `confidence` does not describe `tier2`–`tier4`, and full-path correctness is lower than tier-1 correctness. - The no-content population is **not** ground-truthed at all — a judge shown only a domain name would guess exactly like the classifier did. Treat those rows' categories as unreliable, full stop. On ensemble rows, treat `confidence` as an **uncalibrated model score**, and `confidence_tier` as a **band of that score** — not as a probability of correctness. On rows from the other eight families it is not the classifier's judgment of the label: either another component's own internal number, or — on `pipeline/vendor_rule` rows — the classifier's score carried under a label the rule chose; see the family table above. Either way, `confidence_tier` is a band of whatever `confidence` holds, and neither column is a probability of correctness. > **There is now a calibrated alternative.** Since `feed_schema_version` 1.6.0 the > feed also carries `calibrated_confidence`, which corrects for both defects above > and _is_ meant to be read as a probability: the probability that a strict blind judge, > reading the page, assigns the same `tier1`. That makes it a strict-match estimate, not > the chance that a label is usable. If you want a reliability signal, use > that column — see > [§5.3 below](#calibrated_confidence-calibrated_confidence_tier-and-calibration_segment). > `confidence` is retained unchanged for consumers already integrated against it. #### One non-ordinal token is reachable: `PARKED` Parse defensively. Besides the three ordinal bands, this column has a fourth possible value, `PARKED`: - **Where it comes from.** Our storage layer writes `PARKED` into this column instead of a band when its classifier flagged the domain as parked, and the export publishes the stored value unchanged. That holds under **both** band policies: from artifact 1.14.0 the export derives the ordinal bands rather than passing them through (see [above](#from-artifact-1140-the-band-is-derived-not-the-raw-cut)), but `PARKED` is the parked flag and not a confidence, so no rule may re-rate it and none does — the token survives the change exactly as a NULL does. The one thing that removes it is a suppression guard nulling the row's whole content block, this column included; from artifact 1.13.1 that covers most of the rows that would otherwise carry the token (see the last bullet). - **Where it can appear.** On the `parked`, `inactive` and `dead` lanes the whole content block — this column included — is nulled, so `PARKED` cannot survive there. It is reachable only on a category-keeping row — `active`, `blocked` or `uncategorizable` — i.e. when the classifier's parked flag disagrees with the crawl-observed lane. The two are produced by different systems, so they can disagree. - **It is rare, and it is reachable.** A small number of rows carry it, and every one of them was written by a rule filter rather than by the bulk ensemble — which is the disagreement the previous bullet describes: the classify-time record carried the parked flag (set by the crawl's parked verdict or by the classify-time boilerplate detector — the write side merges the two) while the crawl-observed lane still reported `active`. Treat its incidence as a property of the current pipeline that can move between exports, not as a schema guarantee in either direction, and parse for the token regardless of how few rows carry it. - **From artifact 1.13.1, most of those rows stop shipping a category.** They are the category-keeping subset of the population the demoted-label guard was widened to cover — parked-flagged filter rows already on the `parked`, `dead` or `inactive` lanes are covered too, and lose their `last_known_*` history rather than a visible category. A parked-flagged filter row now ships status-only, so `confidence_tier` reads NULL rather than `PARKED` (see `demoted_label_suppression` in [§10](#10-manifestjson) and the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)). The token stays reachable on two populations — a row whose label projects to a block-critical policy group, held deliberately (counted in `demoted_label_suppression.block_critical_held`), and any parked-flagged row whose `model_name` is outside both the ensemble prefixes and `filter_model_names`, such as `cascade/st_filter`. Keep parsing for the token. If you do encounter it, treat `PARKED` as **"no ordinal confidence band"** — not as a fourth ordinal step. Do not sort or compare it against `HIGH`/`MEDIUM`/`LOW`. > **Note if you also use the API.** The v4 API returns no `confidence_tier`. The legacy > `/categorize` route exposes a field with the same name, and it carries **two** vocabularies > depending on how the answer was produced. A live > classification returns a routing tier — `definitive` / `probable` / `uncertain` / > `unclassifiable` — which has nothing to do with this column. An answer served from the > stored corpus (`routed_to: "hetzner-cache"`) returns **this** column's vocabulary, > `HIGH` / `MEDIUM` / `LOW` / `PARKED`, for the same domain the feed carries. The two > vocabularies do not overlap. > > **Branch on `routed_to`, not on `is_cached`.** `routed_to == "hetzner-cache"` is the only > value that carries this column's vocabulary; **every other value — the live cascade and > its gates and filters, and null — carries the routing one**. `is_cached` is **not** the > discriminator, because a live classification answered from the API's own response cache > also comes back with `is_cached: true` while carrying the routing vocabulary. A consumer > that branches on `is_cached` reads the wrong vocabulary on every cached live answer. > > With the API read-path change in the 1.14.0 release, the API applies > the **same published band policy** to those cached rows as the export does, so a domain's > band does not depend on which surface you read it from; **until that change is deployed > the API serves the stored band**. Read the band field rather than re-deriving one from the > API's `confidence`: the response rounds that number to four decimals, which is a > presentation precision and not the value the policy was evaluated on. ### `calibrated_confidence`, `calibrated_confidence_tier` and `calibration_segment` **Appended 2026-07-30.** These three columns are the fix for everything §5 says about `confidence`. `calibrated_confidence` is an estimate of **the probability that a strict blind judge, reading the page, assigns the same `tier1`** — one number with one meaning on every route, which is what `confidence` is routinely mistaken for and is not. The target is strict, so on average — though not in its top band — the value understates how often a label is acceptable to a reader: read it as a reliability ranking and a strict-match estimate, not as the chance that a label is usable. #### How it is produced The raw score is passed through an isotonic (monotone) map fitted on blind-judged rows (calib-map-v2, 2026-08-13: three stratified samples, design-weighted to the feed population). **What the judge scores changed in v2**: the map now estimates P(the `tier1` label matches what a blind LLM judge assigns to the page **as fetched today**) — v1 judged against stored crawl-time text. This is the stricter, buyer's-eye question (it counts content drift and parked domains against the label), and calibrated values are systematically lower than v1's as a result — see the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog). The map is fitted **per segment**, because the miscalibration is not uniform: it is the interaction of the language route with one category. The segment applied to each row is published in `calibration_segment`: | `calibration_segment` | Rows | | --------------------- | -------------------------------------------- | | `en\|bf` | English page, `tier1 = Business and Finance` | | `en\|rest` | English page, any other category | | `nonen\|bf` | Non-English page, `Business and Finance` | | `nonen\|rest` | Non-English page, any other category | | `__global__` | Fallback — no language recorded (see below) | #### What it buys you The gain is **comparability across segments**. Within one segment the map is monotone, so it does not reorder rows — a higher raw score stays higher. What it fixes is that a `0.80` on an English page and a `0.80` on a non-English page did not previously mean the same thing. After calibration they do. Concretely, the pathology §5 describes: the same raw `Business and Finance` score calibrates to a markedly lower probability on the non-English route than on the English one (calib-map-v2; the non-English `bf` curve only lifts off at high raw scores). Same stored score, very different reliability — previously indistinguishable. #### What it estimates — and what it deliberately does not `calibrated_confidence` estimates the probability that a strict blind judge assigns the same **`tier1`**. It makes no claim about the deeper tiers of the path. This matters for how you key policy on the label, and it was measured directly — a content-grounded audit of delivered rows by two independent LLM judge passes. A language model, not a person, re-ruled most of the first pass's wrong verdicts, settled only its correct-versus-wrong disagreements with the second judge, and marked one more row correct that neither judge had called correct; the disagreements over depth and over whether a page could be judged at all were not adjudicated. The figures are not published on this page; their shape is: - **Tier-1 accuracy tracks the bands as designed** — `PROBABLE` above `TOSSUP` above `DOUBTFUL`, which is what the column is for. - **Full-path accuracy is materially lower**, and the classifier does not cut a path where the page evidence runs out: below `tier1` it keeps the most likely child at each tier whenever that child holds more than a small fixed share among its siblings, so nearly every path runs on until the taxonomy has no deeper category. In that model-judged sample, a wrong deeper tier under a right tier-1 was the dominant error shape — not a wrong tier-1. **Guidance:** for allow/block policy, key on `tier1` (or `tier1` + `tier2`) together with `calibrated_confidence_tier`. Treat `tier3`/`tier4` as descriptive detail — routing, reporting, analytics — rather than as policy keys. As always, state the denominator when you quote a number: accuracy on your own traffic depends on your traffic's mix, and the representative test is running your own domain list. #### The bands | `calibrated_confidence_tier` | Condition | | ---------------------------- | -------------------------------------- | | `PROBABLE` | `calibrated_confidence >= 0.75` | | `TOSSUP` | `0.50 <= calibrated_confidence < 0.75` | | `DOUBTFUL` | `calibrated_confidence < 0.50` | These are deliberately **not** `HIGH`/`MEDIUM`/`LOW` and deliberately not cut at `0.9`/`0.7`. The vocabularies are disjoint so the two columns can never be confused, and the cut points mean something on a probability scale: `0.50` is "better than an even chance", `0.75` is "three in four or better". Unlike the raw bands, these do rank by reliability — that is the point of the column. #### Every segment has a ceiling, and three of the five sit below `PROBABLE` An isotonic curve cannot return a value above the highest point it was fitted to, so each segment has a hard ceiling: the maximum `calibrated_confidence` a row in that segment can be given, **at any raw `confidence`**. Three of the five ceilings sit below the `PROBABLE` cut (`calibrated_confidence >= 0.75`), which means those rows can never reach the top band no matter how confident the classifier was. Under the map in force (v2, fitted 2026-08-13) the highest band each segment can reach is: | `calibration_segment` | Highest band reachable | | --------------------- | ---------------------- | | `en\|bf` | `TOSSUP` | | `en\|rest` | `PROBABLE` | | `nonen\|bf` | `DOUBTFUL` | | `nonen\|rest` | `TOSSUP` | | `__global__` | `PROBABLE` | The ceiling values themselves are a property of the fitted curve and are not published on this page; the artifact carries full precision, and a refit moves them. Read that as a property of the segment, not of the row: **only `en|rest` and `__global__` rows can carry `calibrated_confidence_tier = 'PROBABLE'` at all.** If your filter is "`PROBABLE` only", you are also filtering to those two segments, and you will see no `Business and Finance` row in either language and no non-English row outside them. `calibration_segment` (column 26) is published so this is checkable per row rather than inferred. A refit moves these ceilings; `confidence_calibration_map_version` in `manifest.json` says which map produced a given export. Two segments are worth spelling out further: - **`nonen|bf` cannot reach `PROBABLE`.** That segment's fitted curve (v2) tops out well below the `PROBABLE` cut, so a non-English `Business and Finance` row is capped below the top band at _any_ raw score. This is deliberate: it is the single most over-confident population in the corpus, and the map declines to certify it. - **`__global__` is a genuine fallback, and weaker.** Rows with no recorded language (both `bf` and non-`bf`) had too few judged rows at fit time — below the 50-row per-segment floor — so they take the corpus-wide curve instead. That curve is dominated by language-recorded rows, so it may **under-correct** the over-confidence language-less rows are most likely to have. Treat `calibration_segment = '__global__'` as the least reliable calibrated value in the feed. It is published precisely so you can find those rows; today it is the only case that produces it. #### When these columns are NULL All three are always NULL together — you will never see one populated without the other two. They are NULL in three situations. **The row carries no current content category.** That is the three suppressed lanes `parked` / `inactive` / `dead`, plus every empty-capture row (see [§9](#9-why-a-non-active-row-has-no-category)). A row with no category has nothing to be right or wrong about, so it gets no probability of correctness. In particular the empty-capture rows this section warns about ship **no** calibrated value, rather than a calibrated version of a label nothing on the page supported. > **`blocked` is not in that list.** A `blocked` row **keeps** its category > ([§3](#3-domain_status--the-six-lanes)) and therefore keeps its calibrated value > too — the site was live but unreadable on this crawl, so the label from the > previous crawl still stands and the calibrated figure describes that label. So if > you are asserting an invariant in a loader, do **not** key it on > `domain_status <> 'active'`. > > The safe assertion is the **one-way** one: > > ```sql > -- holds on every row > tier1 IS NULL => calibrated_confidence IS NULL > ``` > > The converse does **not** hold, so do not assert it. A row can carry a category > and still be uncalibrated — see the next two situations. `calibrated_confidence` > is populated on a subset of categorized rows, not on all of them. **The label came from a component or model version the map was not fitted for.** As described above, `confidence` is not one scale across the feed — its meaning depends on `model_name`. The map was fitted on the bulk-ensemble corpus only: | `model_name` | Calibrated? | | ---------------------------------------------------- | ----------- | | `ensemble-5en-3nonen` | yes | | `*/st_filter`, `*/gambling_filter`, `*/adult_filter` | no — NULL | | `pipeline/vendor_rule` | no — NULL | | anything else | no — NULL | This matters most for the gambling filter, whose `confidence` is a keyword-density measure and **not a probability at all**. Running it through a map fitted on ensemble probabilities would publish a meaningless "probability that `tier1` is correct" — so we publish nothing. The same applies to `model_version`, in **both** of its halves: - The `+variant` **suffix**, against an explicit allow-list the map carries. The map in force (v2, fitted 2026-08-13) declares five entries, and rows carrying any of them are calibrated: no suffix at all, `ml_en`, `ml_non_en`, `nocontent_bf1` and **`teacher_bf1`**. The first three are routing markers — same model, same scale. `nocontent_bf1` records that a pass fetched the row and found nothing classifiable; it rewrites no score. `teacher_bf1` **is** a re-scoring pass, and its rows were out of scope until the 2026-08-13 refit deliberately fitted them in — so, contrary to what earlier revisions of this page said, `v1+teacher_bf1` rows **do** carry `calibrated_*` values today. Any suffix outside those five is uncalibrated until a refit adds it. - The **base version** (`v1` in `v1+ml_en`). `confidence` is a property of the trained weights, so a **retrained model** shipped as a plain base bump (`v2`) is out of scope until the map is refitted for it. `confidence` itself keeps flowing unchanged; the reason for the NULLs is visible in `model_version` (column 11) and in the manifest's `row_count_calibrated`. So `calibrated_confidence` is populated only for rows whose `model_name`, whose `model_version` base, **and** whose `model_version` suffix the map was each fitted for. If a model upgrade lands before a matching refit, expect these columns to go NULL for the affected rows while `confidence` keeps flowing — that is the intended, visible behaviour, and `row_count_calibrated` in `manifest.json` will show it. All of these rules are allow-lists: a component, pass or model version added later stays uncalibrated until the map has been fitted for it. That is deliberate — an uncalibrated row is honest, a wrongly-calibrated one is not. **The score is unusable.** A missing or non-finite `confidence` cannot be calibrated. This should not occur on a category-bearing row and has not been observed, but it is enumerated so the list is exhaustive. A row that carries a `tier1` but a NULL `calibration_segment` is therefore telling you it was left uncalibrated for one of the last two reasons. `manifest.json` publishes `row_count_calibrated` so you can see how much of an export carries these columns without scanning it. #### Caveats — the same ones as the raw column Calibration corrects the _scale_; it cannot manufacture evidence. - It describes **tier 1 only**, like `confidence`. It says nothing about `tier2`–`tier4`. - The reference labels are **LLM-generated, not human-adjudicated**. - The language half of the segment key is read from the `language` column, which is the language observed on the crawled page. The classifier's own routing decision is made separately and the two can disagree on individual rows. The map was fitted against the same column the feed publishes, so calibration is self-consistent — but `calibration_segment` describes the published `language`, not a guarantee about which model ran. - `manifest.json` records which map produced these values in `confidence_calibration_map_version`. A refit changes calibrated values with no raw column changing; pin that key if you need to know two snapshots are comparable. ### `tier` `tier` is an integer `0`–`3` describing **how the label was produced** — its provenance and conflict-resolution priority. It is **not** taxonomy depth (that is the `tier1`..`tier4` name columns). | `tier` | Meaning | | ------ | -------------------------------------------------------------------------------- | | `0` | **Parked-default.** A status-only row. There is no content classification on it. | | `1` | Bulk classification — the standard automated path. | | `2` | Cascade classification — escalated to a stronger model. | | `3` | Human-reviewed. | A higher `tier` wins conflicts. Every row on the `parked`, `inactive` and `dead` lanes carries `tier = 0`. --- ## 6. `openrtb_cat` — an array column `openrtb_cat` holds the OpenRTB / AdCOM raw category ids for the row's tier path, ordered root → leaf. The taxonomy those ids belong to is published once per export as `cattax` in `manifest.json` (value `9` = IAB Content Taxonomy 3.1). **This column is array-valued, and its physical representation differs by file format:** | Format | Representation | | ---------- | ---------------------------------------------------------------------- | | `.parquet` | native `list` | | `.csv.gz` | a **JSON-encoded array in a single cell**, e.g. `["596", "627", "52"]` | > **Integration warning — this will surprise your loader.** No comparable > categorization feed on the market ships an array-valued column, so most CSV > pipelines are not built to expect one. In the CSV the cell contains a JSON document > that must be parsed, not split on a delimiter, and the standard CSV quoting rules > apply to the embedded quotes. Budget for this explicitly. `openrtb_cat` is **NULL** — not an empty array — whenever the tier path does not resolve to ids. An empty array would read as "resolved to zero categories", which is a different statement. --- ## 7. Nulls and encoding | Format | How NULL appears | | ---------- | --------------------------------------------------- | | `.csv.gz` | an **empty field** (nothing between the delimiters) | | `.parquet` | a **true null** | **Absence is always null, never a sentinel.** No column uses a placeholder to stand in for a missing value: you will not find `"N/A"`, `"Unknown"`, `-1`, or an empty string meaning "we don't know". If a fact is absent, the field is null. `language` (column 14) is the one column whose value comes from the page itself: the crawled page's `lang` attribute, passed through unchanged. A page that declares `lang="null"` therefore ships the four-character string `null`. That is the page's word, not our placeholder — treat any such token as "no usable language". The 1.14.0 language-NULL guard ([§10](#10-manifestjson)) keys on a genuinely absent value and leaves such rows alone. One value looks like a sentinel and is not. `confidence_tier` can carry the non-ordinal token `PARKED` (see [§5](#5-confidence-and-tier)). That is a positive statement from the classifier, not a stand-in for a missing confidence — where confidence is genuinely absent, `confidence_tier` is null like everything else. Columns 45–46 are the plainest case of null meaning absent. `content_purpose_id` is null on every row outside a curated list of known news publishers — which is nearly every row, news sites included — so a null there is **unknown, never "not news"** ([§14](#14-iab-content-purpose-columns-4546)). Other encoding notes: - Files are UTF-8. The CSV is RFC 4180 quoted. - From 1.14.0 (first artifact the 2026-09-17 set), `domain` is lowercase ASCII, with internationalised names in punycode (`xn--`): a source row whose name arrives in Unicode form is withheld by the exporter rather than shipped beside its punycode twin, and the withheld count is published in the manifest as `row_count_excluded_non_ascii_domain`. Artifacts before 1.14.0 still carry those rows, most of them beside the same name in punycode, so one registrable domain appears twice. To apply the 1.14.0 rule to an earlier artifact, drop every row whose `domain` is not ASCII. - All timestamps are ISO-8601 with a UTC offset, with or without six fractional digits: `classified_at` reads `2026-09-09T18:46:49.941343+00:00` on model-written rows and `2026-05-08T00:00:00+00:00` on curated rows, while `status_observed_at` carries the crawl's own precision on every lane. Parse every timestamp column with a format that accepts an optional fraction. They are transported as **strings** in both formats — including in Parquet — for portability across loaders. - `confidence` and `calibrated_confidence` are written with 4 decimal places in CSV so they match the Parquet `float32` value exactly. --- ## 8. The `last_known_*` columns Columns 18–23 carry a **historical** content label: one this domain held at an earlier point, which is no longer presented as current. **They are populated only on `parked`, `inactive` and `dead` rows** — never on `active` (where the label _is_ current, in columns 2–6) and never on `blocked`. ### Coverage: sparse, and read it off your own delivery This is a sparse column block, and how sparse it is moves between exports: on the large majority of non-live rows it is empty, and the non-live population per partition grows over time, so a figure measured on one export is out of date on the next. **Read coverage off the delivery you actually received.** Every export publishes its own `row_count_with_last_known_category` and `row_count_by_domain_status` in `manifest.json`. That pair is exact for the artifact in your hands and needs no scaling; a figure quoted in documentation never will be, which is why none is quoted here. The reason coverage is low at all is structural, not a gap we intend to close: **the great majority of non-live domains were already parked, inactive or dead the first time we crawled them**, so there was never a live label to retain. A historical label exists only where we observed the domain live, labelled it, and later observed it non-live. **Do not treat this as a parked-wide feature.** If your integration depends on knowing what a parked domain used to be, it will be unpopulated on the overwhelming majority of rows — verify against your delivery's own count before building on it. **Scope of a partition measurement, stated precisely.** Both feed tables are `HASH(domain)` on the same 16-way modulus, so domains are distributed uniformly across partitions, but the lane mix is not: in the 2026-10-18 set the `blocked` and `dead` shares and the `last_known_*` count differed noticeably from one partition to the next. Scaling one partition by sixteen gives an estimate, and only for the export it was taken from. The count grows over time as more domains are categorized while live and later observed non-live, so a scaled figure ages the moment the next snapshot is built. Sum the sixteen partition manifests of the delivery in front of you (or read the run-level `run_manifest.json`, [§10](#10-manifestjson)) when the exact feed-wide number matters. Where a `last_known_*` value **is** present it is dated by `last_known_classified_at` and is a statement about the past. It is never a claim about the domain's current content or its current owner. --- ## 9. Why a non-active row has no category On `parked`, `inactive` and `dead` rows the content columns are NULL deliberately. A parked page, a coming-soon placeholder and a dead domain are not "sites about something" — running a content classifier over a parking lander would produce a label describing the parking service, not the domain. Our position is that the honest classification of a parked domain **is** `domain_status = parked`, dated by `status_observed_at`. Categorization vendors generally take the same approach. Two consequences to encode in your integration: 1. **A NULL category is not a claim that no category exists.** It is a statement that we are not asserting one for this domain in this state. 2. **Do not blend lanes when measuring accuracy.** A content-accuracy figure computed over this feed is computed over `active` rows only. Every export states this explicitly — see below. --- ## 10. `manifest.json` Every export directory contains a manifest. The keys most relevant to a consumer: | Key | Meaning | | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `feed_schema_version` | Semantic version of the artifact (columns + manifest keys). Pin against this. | | `schema_version` | The frozen column-contract identifier, `feed_v1`. | | `columns` | The column list, in contract order. | | `row_count` | Total rows — **blended across all six lanes**. | | `row_count_by_domain_status` | Rows per lane. Sums to `row_count`. | | `content_accuracy_denominator_lane` | Which lane is answerable. Always `active`. | | `content_accuracy_denominator_row_count` | The answerable row count — **the correct denominator for any accuracy figure**. | | `row_count_with_last_known_category` | Rows carrying a historical label ([§8](#8-the-last_known_-columns)). | | `confidence_calibration_map_version` | Which fitted map produced the `calibrated_*` columns (artifact ≥ 1.6.0). A refit changes those values with **no raw column changing**, so pin this key if you need to know two snapshots' calibrated values are comparable. See [§5](#5-confidence-and-tier). | | `confidence_band_policy` | Which band policy produced column 8 (artifact ≥ 1.14.0). `v1` is the raw pass-through — the stored `confidence` cut, verbatim — and a manifest that does **not** carry this key is `v1`. `v2` is the derived band described in [§5](#from-artifact-1140-the-band-is-derived-not-the-raw-cut), `v3` (artifact ≥ 1.15.0) is `v2` plus rule 5, which caps the name-only lane at `MEDIUM`, and `v4` (artifact ≥ 1.16.0) is `v3` plus rule 4, which caps at `MEDIUM` — and at `LOW` where `calibrated_confidence_tier` is `DOUBTFUL`, a stored `MEDIUM` included — any ensemble row whose `model_version` provenance suffix (`split_part(model_version, '+', 2)`) is `teacher_bf1`, `nocontent_bf1` or `teacher_purpose_v2`. The band is derived rather than tracked, so a delta whose policy differs from the artifact it extends is **reported as a chain gap**, and re-publishing an existing artifact key under a different policy is refused — that refusal is ours, at publish time, and is a different mechanism from the gap. A chain gap is a flag, not a refusal: whether a policy step actually re-rates any row you already hold depends on the rule, so the changelog entry for that policy is what tells you whether you need a new full snapshot or the deltas carry it. **For `v2` → `v3` the deltas are enough** — rule 5 is keyed on the `llm-name/*` writer alone, so no row written before that lane's first load re-bands. **For `v3` → `v4` they are not**: rule 4 is keyed on `model_version`, so rows published in every set since 1.14.0 re-band, and the change reaches you only by ingesting the full Sunday 2026-10-11 set — see that release's changelog entry. The check compares each artifact with its **immediate predecessor in the chain**, not with the last snapshot, so the **first** delta after the 1.15.0 deploy reports the policy change and the deltas after it — which extend a `v3` artifact — are quiet. That same first delta also crosses `feed_schema_version` 1.14.0 → 1.15.0 and reports that step separately: expect **two** reports on that one delta, then silence. Both are expected, and the silence that follows is not a sign the check stopped running. The first delta after the 1.16.0 deploy, `delta_2026-10-09_100000Z`, does the same: it reports `v3` → `v4` and 1.15.0 → 1.16.0, and the deltas after it report neither step. Pin this key beside `confidence_calibration_map_version` — under `v2` a refit of the calibration map moves band membership as well as calibrated values. | | `row_count_calibrated` | Rows carrying a `calibrated_confidence` ([§5](#5-confidence-and-tier)). Lets you see the calibrated share of an export without scanning it. | | `row_count_curated` | Rows from the hand-vetted curated list (artifact ≥ 1.7.0) — `model_name` beginning `curated/`. See [§5](#curated--the-hand-vetted-rows). | | `status_only_lane` | Whether this artifact carries the status-only lane ([§9](#9-why-a-non-active-row-has-no-category)). On a snapshot: the whole lane — every inventory domain the lane can ship that has no category row (the parked, placeholder and never-categorized population), as distinct from category-lane rows a guard held to status-only (those are counted by the suppression keys below); never-shippable dead inventory (`inventory_never_shippable_by_rank_bucket`) produces no row. Its size is not published as one key: `row_count_by_domain_status` gives the `parked` / `inactive` / `dead` counts that make up nearly all of it — read them off your own delivery rather than any fixed feed-wide figure. On a delta (artifact ≥ 1.9.0): the lane's rows whose inventory moved since the cutoff — the baseline still arrives only with a snapshot. Part of the artifact's identity: a re-publish at the same key with this toggled is refused rather than silently changing the artifact. | | `curated_lane` | Whether this artifact carries curated rows at all (artifact ≥ 1.7.0). `true` on snapshots, and on deltas from artifact ≥ 1.13.2; `false` on a delta below 1.13.2, which carried the curated exclusion but not the rows. `row_count_curated` is how many actually shipped. | | `rank_ladder` | `present` or `absent` (artifact ≥ 1.7.0) — whether popularity-rank buckets were available when this export ran. `absent` means the bucket keys below are not in this manifest; it is a stated value so their absence is never ambiguous. Mostly operational telemetry, but two row-level rules read the buckets: the language-NULL head exemption (`language_null_suppression`) and the empty-capture `blocked` lane (§3, `parked_stub_suppression`). With `rank_ladder` `absent`, neither applies. | | `row_count_by_rank_bucket` | Present only when `rank_ladder` is `present`: exported rows per popularity bucket, keyed `"0"` top-100k and `"1"` top-1M on the crawler's rank ladder, and `"3"` the rest; `"2"` is reserved and not emitted. The ladder takes a domain's best rank across the lists it ranks on, so bucket `"0"` holds more than 100,000 rows. Counts rows backed by a crawl-inventory record, so it is drawn from the same population as the next key and **does not necessarily sum to `row_count`** — use `row_count_by_domain_status` for a breakdown that does. | | `inventory_total_by_rank_bucket` | Optional: the size of our crawl inventory in each bucket, when the (expensive) inventory scan was enabled for the run. | | `coverage_by_rank_bucket` | Present only when both of the two keys above are: the ratio of the first to the second, per bucket, to 4 decimals. It counts presence in the feed, labelled or not, so it is not a labelled-coverage figure. | | `row_count_dead_status_by_rank_bucket` | (artifact ≥ 1.13.0) Present only when `rank_ladder` is `present`: of the rows in `row_count_by_rank_bucket`, those whose crawl status is in our dead set — in practice `not_found` (`404`) and, from 2026-09-23, a definitive DNS failure (NXDOMAIN or no address; changelog 2026-09-28) — they ship because a category or curated row exists for them. | | `inventory_dead_status_by_rank_bucket` | (artifact ≥ 1.13.0) Optional, from the same inventory scan as `inventory_total_by_rank_bucket`: inventory keys per bucket whose crawl status is in the dead set (uncrawled keys, which have no status yet, are NOT dead). Absent when the scan could not read the flag on every row. | | `inventory_never_shippable_by_rank_bucket` | (artifact ≥ 1.13.0) Present only when both keys above are: dead-status inventory minus the dead-status rows that shipped, floored at 0 — keys that cannot produce a row until a re-crawl succeeds (our heal lane retries `not_found` on a 180-day backoff, and a definitive DNS failure on a 14-day one). Slightly overstated when a key heals between the inventory scan and the row stream of the same export. | | `coverage_shippable_by_rank_bucket` | (artifact ≥ 1.13.0) Present only when the key above is: `row_count_by_rank_bucket` over (`inventory_total_by_rank_bucket` minus `inventory_never_shippable_by_rank_bucket`), to 4 decimals, capped at 1.0 (the heal skew above can push a bucket over). A bucket whose keys are all never-shippable is omitted, not published as 0. The existing `coverage_by_rank_bucket` is unchanged; this is the stricter-denominator companion. | | `status_age_reference_date` | (artifact ≥ 1.14.0) The date every row's `status_observed_at` was aged against for the three keys below: a snapshot's own date, a delta's cutoff day (UTC). It is the artifact's **key**, never the time the export ran, so a re-published artifact reports the same ages. `null` only on an export produced outside the scheduled lanes, in which case the three keys below are absent rather than clock-dated. | | `row_count_by_status_age` | (artifact ≥ 1.14.0) Rows per lane per status age, where age is `status_age_reference_date` minus the UTC calendar day of `status_observed_at`: `le_7d` (≤ 7 days), `le_30d` (8–30), `le_90d` (31–90), `gt_90d` (> 90) and `null` (no `status_observed_at`). Each row lands in exactly one bucket, so every lane's five buckets sum to its `row_count_by_domain_status` entry and the whole map sums to `row_count`. The crawler revisits domains in cohorts, so a bucket can move sharply between two consecutive days' references. | | `row_count_by_status_age_by_rank_bucket` | (artifact ≥ 1.14.0) Present only when `rank_ladder` is `present`: the same five-bucket split per popularity bucket, over the inventory-backed rows `row_count_by_rank_bucket` counts — so each bucket's five ages sum to that key's entry, and the map does **not** necessarily sum to `row_count`. | | `stale_fraction_active_gt_90d` | (artifact ≥ 1.14.0) The `active` lane's `gt_90d` count over the lane's whole row count (`content_accuracy_denominator_row_count`), to 4 decimals; `null` when the export has no `active` rows. A one-number freshness read for the answerable lane — compare it across your own deliveries rather than against a fixed threshold, for the cohort reason given under `row_count_by_status_age`. On a **delta** the `active` lane is the changed-row set plus the curated registry (carried whole on every delivery), not the corpus — a row whose crawl moved inside the window ages to zero or below against the cutoff day, while re-classified rows and curated rows carry arbitrary crawl ages — so a delta's value is structurally fresh-skewed. Compare snapshots with snapshots and deltas with deltas, never one against the other. | | `policy_pack` | Which policy pack produced the `policy_*` columns (artifact ≥ 1.12.0): `{"id": "policy_v1", "version": "2026.09.2", "sha256": "…"}` on every 1.12.0–1.13.x artifact and `{"id": "policy_v2", "version": "2026.09.1", "sha256": "…"}` from 1.14.0. `version` is the **mapping** version — a pack edit changes published groups with no column change, so pin it to know two snapshots' groups are comparable; `sha256` is the pack file's digest. `version` and `sha256` null means the pack was unavailable, so the four pack-derived columns — `policy_group`, `policy_basis`, `policy_pack_version` and `policy_group_set` — are NULL on every row because nothing could project them, not because nothing resolved. `policy_confidence_basis` (column 42) is **independent** of pack availability — it is derived from `model_name` and the calibrated value — so it stays populated beside a null pack, as does `calibrated_confidence`; do not read a populated basis as evidence the pack loaded. See [§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection). | | `row_count_with_policy_group` | Rows carrying a `policy_group` (artifact ≥ 1.12.0). Lets you see the projected share of an export without scanning it; 0 beside a null `policy_pack.version` is the degraded case. | | `site_purpose_model` | Which model populated the reserved `site_purpose_*` columns (artifact ≥ 1.12.0): `null` until a purpose pass runs, then `{"name": …, "version": …}`. | | `row_count_by_site_purpose_status` | Rows per `site_purpose_status` value — `labeled` / `undefined` / `not_available` (artifact ≥ 1.12.0). `labeled: 0` is the honest statement that the axis is reserved, not populated. | | `manifest_keys_version` | (artifact ≥ 1.14.0) The version of the manifest's KEY SET, independent of `feed_schema_version`. Pin this if what you parse is the manifest rather than the columns: a MINOR bump adds a key, a MAJOR one removes or re-means one. An artifact carrying no such key predates 1.14.0. `1.0.0` from 1.14.0; `1.1.0` from 1.17.0, which added `content_purpose_list` and `row_count_by_content_purpose_source`. | | `row_count_by_confidence_tier` | (artifact ≥ 1.14.0) Rows per `confidence_tier` — `HIGH` / `MEDIUM` / `LOW` / `PARKED` / `null`. All five keys are always present, so a `0` is a measured zero. `null` counts rows carrying NO band and nothing else: should a row ever ship a value outside the four documented bands, it is counted under that value's own key rather than folded into `null`, so the map still sums to `row_count` and the anomaly is visible rather than disguised as an absence. Sums to `row_count`. | | `row_count_by_writer_family_x_tier` | (artifact ≥ 1.14.0) The same distribution split by which family wrote the label: `ensemble` / `filter` / `vendor_rule` / `rule` / `curated` / `carry_over` / `name_only` (artifact ≥ 1.15.0) / `other` / `unknown`. `confidence` is not one scale across those families ([§5](#5-confidence-and-tier)), so a blended band distribution hides whether `HIGH` is classifier rows or filter rows. One family per row, so the inner maps sum to `row_count`. Every inner map carries the same key set as `row_count_by_confidence_tier`, including any out-of-vocabulary key described there, so the two can be diffed cell by cell. The `curated` family here is the WRITER family, not the curated lane: it also counts `human-feedback` rows (a reviewer's accepted verdict), where `row_count_curated` above counts only rows whose `model_name` begins `curated/` — so the two keys need not agree, and `curated` here is the larger of the two whenever any reviewer verdict shipped. | | `row_count_by_policy_group` | (artifact ≥ 1.14.0) Rows per `policy_group`, with **every group the pack declares present**, `0` where this artifact has none — so "no gambling-shaped rows here" and "the pack stopped mapping gambling" are distinguishable. `{}` means the pack did not load. Sums to `row_count_with_policy_group`. | | `row_count_policy_group_uncalibrated` | (artifact ≥ 1.14.0) Rows that resolved a `policy_group` but carry no `calibrated_confidence` — the population a group filter would be thresholding on a raw, scale-incomparable number. Bounded above by `row_count_with_policy_group`. | | `row_count_by_policy_group_by_writer_family` | (artifact ≥ 1.14.0) Which writer families populate each group. A group whose rows are all `filter` should be read as a filter verdict, not as a classifier's opinion. Each inner map sums to that group's `row_count_by_policy_group` entry. | | `row_count_by_safety_status` | (artifact ≥ 1.14.0) Rows per `safety_status` — exactly `flagged` / `assessed_clean` / `not_assessed`. Sums to `row_count`. | | `row_count_by_safety_flag` | (artifact ≥ 1.14.0) Flag **occurrences** per theme. A row carries zero, one or several flags, so this is the one key in this block that does **not** sum to `row_count`. | | `row_count_by_language_primary` | (artifact ≥ 1.14.0) Rows per `language_primary`: the 30 largest subtags by name, everything else summed under `other`, and rows with no observed language under `null`. `other` and `null` are not language codes, and a row whose `language_primary` IS the literal string `null` (a page whose `lang` attribute said so) counts under `other`, never under `null` — that bucket means no language was observed. `null` is held OUT of the top-30 ranking, so a partition rich in languages can never rank the no-language bucket into `other`. Sums to `row_count`; at run level the map is re-capped after summing — the re-cap ADDS its tail to the `other` the partitions already carried, so the total stays exact — while a subtag that fell into `other` on some partitions is under-attributed by name. Because the partitions' summed `other` takes part in that re-cap's ranking, the run-level map usually names 29 subtags, not 30. | | `content_purpose_list` | (artifact ≥ 1.17.0) Which news-publisher list set columns 45–46: `{"path": …, "sha256": …, "row_count": …}`. `sha256` is the list file's digest and `row_count` the number of domains it names. Pin `sha256` to know two artifacts were tagged from the same list: an edit to the list changes the tags with no column change. On a delta the `curated` count in `row_count_by_content_purpose_source` equals this block's `row_count` — not the manifest's top-level `row_count` — unless a listed domain did not ship as a curated row (a takedown, for example); a partition manifest counts only its own partition's tagged rows. The snapshot run marker carries the set's list as `{sha256, row_count}` and is not written for a set tagged from two lists. See [§14](#14-iab-content-purpose-columns-4546). | | `row_count_by_content_purpose_source` | (artifact ≥ 1.17.0) Rows per `content_purpose_source` — `curated` / `schema_org` / `null`. All three keys are always present, so a `0` is a measured zero: `schema_org` is reserved and reads `0` on every 1.17.0 artifact, and `null` counts every untagged row. Sums to `row_count`; the run marker sums it across the sixteen partitions. | | `row_count_tier1_changed` | (artifact ≥ 1.14.0, **deltas and the snapshot run marker**) Qualifying tier-1 transition **events** inside the window, from our category-transition ledger — first labels excluded. On a delta the window is that delta's own; on a snapshot's `run_manifest.json` it is the ledger window between the previous snapshot's run marker and this one, read with ONE query over the whole interval rather than summed over the deltas inside it — so an overlap or a gap between two consecutive delta windows does not move this number. Counted per event, not per domain: a domain relabelled `A` → `B` → `C` inside one window contributes twice, so the key is not de-duplicated by domain and is not a fraction of `row_count`. **A planned re-score announces itself here:** a delta's `row_count` says how many rows moved, not whether they moved between categories. On a run marker a `0` beside a non-null `tier1_transitions_reason` is that reason, not a measured zero — see the reason key below. | | `top_tier1_transitions` | (artifact ≥ 1.14.0, **deltas and the snapshot run marker**) The twenty largest `{from, to, count}` tier-1 moves in the same window as `row_count_tier1_changed`, largest first. Each `count` counts **events** on the same terms as `row_count_tier1_changed`, so one domain can appear in two pairs. The tail is not published, so these do **not** sum to `row_count_tier1_changed`. | | `policy_group_transitions` | (artifact ≥ 1.14.0, **snapshot run marker only**) The moves behind `top_tier1_transitions`, each projected through the CURRENT policy pack at the full tier path it was recorded at: `{from, to, count}` over `policy_group`s, largest first, capped at twenty. Full paths from the 2026-11-08 set; before it, the projection read each move's tier-1 label alone, so a move under a deeper pack rule was counted in its bare tier-1 label's group — `Attractions > Casinos & Gambling` as `travel`, for example ([Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog)). A move whose two paths land in the SAME group is dropped deliberately — a move a group filter cannot see is not churn a blocker needs to act on — so these do not sum to `row_count_tier1_changed` either, and the moves behind one tier-1 pair can split across several group pairs. A move that keeps its tier-1 label is not counted here at all, even when a deeper tier changes its group. Projected over the moves behind the twenty published pairs, so the tier-1 tail those drop is dropped here too. `[]` when the pack did not load, and also when the verified 16-partition set was projected with a pack this build is not running — the groups would otherwise come from a different mapping than the marker's own `policy_pack` key names, which a re-run recovering a marker after a pack change could do. **Deltas do not carry this key**, and a delta's `top_tier1_transitions` cannot rebuild it: a tier-1 pair does not say which path each move took. | | `tier1_transitions_window` | (artifact ≥ 1.14.0, **deltas and the snapshot run marker**) `{since, until}` for the churn keys above, or `null` when no churn was measured (a run marker also says why, in `tier1_transitions_reason`). `null` is not a zero. Half-open on the ledger's own `occurred_at`: a row exactly at `since` is counted, a row exactly at `until` is not. On a DELTA, `until` is read when the export starts while the NEXT delta's `since` is the cursor captured when this one's run began (its 10:00 UTC start, to the minute), so two consecutive delta windows OVERLAP by the time between the two and a ledger row landing in that gap is counted by both. Normally that is the export's start-up latency: at the observed churn rate, a row or two per delta. When a weekly set is still running at 10:00 UTC on a day other than Sunday, the delta waits for it to finish instead of skipping the day, and the overlap is that whole wait — its events are counted by both deltas, while the run marker's count is unaffected. The row stream overlaps by the same convention on purpose, being an idempotent upsert. On a RUN MARKER, `until` is the marker's own `created_at` and `since` is the previous snapshot marker's, so two consecutive snapshot windows tile the interval exactly — no gap, no double count — and re-publishing a date RETAINS BOTH of the first marker's bounds — its `created_at` and the `since` it published — so a recovery re-run can neither move a boundary the next snapshot already abuts nor narrow the interval it told you these counts cover, even if a marker for a date inside that interval appears in between: both bounds, and the counts over them, are reproducible. The window rides the marker rather than being inferred from the weekly cadence. | | `tier1_transitions_since` | (artifact ≥ 1.14.0, **snapshot run marker only**) The lower bound of the window above, published as a bound in its own right: the previous snapshot marker's `created_at`, or `null` when this run located no previous marker to measure from. On a measured marker it is exactly `tier1_transitions_window.since`. On a marker that published a `tier1_transitions_reason` instead of a window it is the interval that run _would_ have measured over — for every one of those reasons, including a re-run made with no ledger configured at all — which is what makes the bound survive a re-run that could not measure, so the recovery re-run after it reuses the interval you were told these counts cover rather than re-deriving one. It is not itself a claim that anything was measured: read the counts off `tier1_transitions_window` and `tier1_transitions_reason` as before. | | `tier1_transitions_reason` | (artifact ≥ 1.14.0, **snapshot run marker only**) `null` when the window above was measured; otherwise why it was not — and the value tells you how to read the counts. `predecessor marker missing` / `no 1.14.0 predecessor`: there is no earlier ≥ 1.14.0 snapshot to measure from (the first snapshot of the chain), and the three churn keys are published as `0` / `[]` / `[]` — the absence of a predecessor, **not** a measurement of no churn. `ledger not configured` / `ledger engine could not be opened` / `ledger read failed` / `predecessor marker unreadable` / `predecessor marker is not older than this run` / `a later snapshot's window is already published` / `later snapshot marker unreadable` / `successor scan exhausted before this run's upper bound`: the measurement could not be made, and the three churn keys are **absent** rather than reported as zero. The last three are the out-of-order case: a snapshot whose date is recovered _below_ one already published would have to measure over an interval that later snapshot already covered, so it withholds rather than announce the same churn twice — and it withholds the same way when the check for such a snapshot ran out of budget before reaching the day this run's own window closes on. Its `tier1_transitions_since` is `null` too, and the later snapshot's own bounds and counts are untouched. Either way the marker is written and the completeness contract below is unchanged. | | `partition` | (snapshots) Which of the sixteen partition folders this manifest describes, `p00` … `p15` — the same token as the folder suffix. Absent on deltas. | | `parquet_codec`, `parquet_row_group_size` | How the Parquet twin was written: the compression codec (`zstd`) and the row-group size (1,000,000 rows). Loader hints only; the CSV has no equivalent. | | `openrtb_taxonomy_loaded` | `true` when the AdCOM mapping behind `openrtb_cat` loaded for this export. `false` means the `openrtb_cat` cells of this artifact are empty for a tooling reason, not because no mapping exists — treat the column as unavailable on that artifact. | | `unmapped_crawl_status_counts` | Crawl statuses the lane mapping did not recognise, with their row counts; `{}` when none. Non-empty is the fail-loud signal behind the `uncategorizable` lane in [§3](#3-domain_status--the-six-lanes): those rows went there rather than being guessed. | | `row_count_excluded_non_ascii_domain` | (artifact ≥ 1.14.0) Rows the exporter withheld because the source `domain` was not ASCII — a Unicode-form internationalised name, which would otherwise ship beside its punycode twin as a second row for one registrable domain. Always present; `0` is a measured zero; summed in the run marker. The count includes names whose punycode form is not in the feed yet; those return once the inventory ingest normalises them. | | `cattax` | AdCOM taxonomy id for `openrtb_cat` ids (`9` = IAB Content Taxonomy 3.1). | | `status_lane_notice` | Machine-readable restatement of the lane semantics in [§3](#3-domain_status--the-six-lanes). | | `parked_stub_suppression` | The empty-capture override (artifact ≥ 1.5.0; named for the parking-provider stub that dominated it then): `content_hash` names the fingerprint of a capture that extracted to no readable text; `row_count_overridden` counts the rows whose category this export withheld for it. A row whose crawl succeeded ships `parked`, or `blocked` where that capture is from 2026-07-31 and the domain is in popularity bucket `0` or `1`, when the export carries the buckets (a row whose only label is a name-only one keeps the lane its crawl decides, §3); one whose latest crawl reports a fresher non-content status (`dead`, `blocked`, …) keeps that lane. All of them carry no content category and no `last_known_*` block. It does not trend to zero: an empty capture of a live site produces the same fingerprint, so the count holds roughly steady from week to week (see the known issue in [§3](#3-domain_status--the-six-lanes)). | | `no_evidence_suppression` | The no-evidence guard (artifact ≥ 1.8.0): ensemble-family rows (`model_name_prefixes`) whose category was produced with no captured content (`content_signal`: `content_hash` is NULL — no crawl ever captured page content — with `language` NULL/empty as a guard conjunct) ship status-only — category columns NULL, no `last_known_*` block, `domain_status` unchanged. `row_count_overridden` counts the rows this export suppressed. Rows whose label had captured content behind it, and rule-derived rows (`tld_rule/*`, `schema_rule/*`, `curated/*`, `v3_historical_fallback`), are never affected. Consume the first ≥ 1.8.0 set as a **full per-partition replace** — the same replace model removals already require; a delta stream cannot retro-apply the mask to rows that don't otherwise change. Trends to 0 as re-crawls land. | | `boilerplate_template_suppression` | The boilerplate-template guard (artifact ≥ 1.10.0): ensemble-family rows (`model_name_prefixes`) whose `content_hash` is on the published `content_hashes` list — 60 vetted templates at introduction: hosting-provider parking/coming-soon pages, registrar reserved/for-sale pages, stock web-server and panel defaults, expiry pages, and similar — ship status-only: category columns NULL, no `last_known_*` block, `domain_status` unchanged. The crawl captured the HOST's boilerplate rather than owner content, so the label described the hosting provider, not the domain (most visibly as spurious `Web Hosting` leaf labels). The full hash list is published in the block so the artifact is self-describing about which rows the guard can touch; `row_count_overridden` counts the rows this export suppressed. Rule-derived rows are never affected. Same transition rule as 1.8.0: consume the first ≥ 1.10.0 set as a **full per-partition replace**. Trends to 0 as re-crawls land — new content changes the hash and releases the row. | | `demoted_label_suppression` | The demoted-label guard (artifact ≥ 1.11.0): ensemble-family rows (`model_name_prefixes`) — and, from 1.13.1, rows whose `model_name` is exactly one of `filter_model_names` (`pipeline/st_filter`, `pipeline/gambling_filter`, `pipeline/adult_filter`, `pipeline/vendor_rule`; matched whole, never as prefixes) — whose label the WRITE side branded parked-demoted (`signal`: the category row's `is_parked` flag is TRUE — set since 2026-08-22 by the classify-time placeholder detector, or by the crawl's own parked verdict carried on the record being classified) ship status-only: category columns NULL, no `last_known_*` block, `domain_status` unchanged. This closes the gap the vetted-hash guard above cannot: the detector generalizes across template variants while the hash list is exact-match, so a demoted row whose template is not (yet) on the vetted list previously shipped its fabricated category. Rows the hash guards already suppress are counted under those blocks, never here — the four `row_count_overridden` counts are disjoint (five from 1.14.0, with `language_null_suppression` below). Curated rows are never affected, and neither is any rule-derived row outside `filter_model_names` (`tld_rule/*`, `schema_rule/*`, `v3_historical_fallback`). Same transition rule as 1.8.0/1.10.0: consume the first ≥ 1.11.0 set — and the first ≥ 1.13.1 set — as a **full per-partition replace**. Trends to 0 as re-crawls under the fixed detector replace the demoted labels' ground truth. **From 1.13.1** the block also publishes the block-critical hold. `block_critical_held` counts filter rows deliberately NOT suppressed because their `policy_group` is block-critical — counted there and never in `row_count_overridden`; on the category-keeping lanes (`active`, `blocked`, `uncategorizable`) such a row keeps its category and `PARKED` tier, and on the parked/dead/inactive lanes, where the lane already blanks the current category, it keeps its `last_known_*` history. Beside it: `block_critical_groups` (the groups actually held in that run), `block_critical_group_policy` (the eight-group rule — `adult`, `drugs`, `gambling`, `gambling_games`, `harmful_content`, `hate`, `piracy`, `weapons`), `unresolved_group_held` (rows held fail-closed because their group could not be resolved — non-zero beside a `policy_pack` block whose `version` and `sha256` are null means the pack did not load and the hold ran blind for that export; the block itself is always present) and `policy_group_delta` (per-group `before`/`after` counts, over rows on category-keeping lanes only). | | `language_null_suppression` | The language-NULL guard (artifact ≥ 1.14.0): ensemble-family rows (`model_name_prefixes`) whose label carries `language` NULL/empty behind a captured content hash (`content_signal`: `content_hash` NOT NULL, `crawl_status` NOT NULL, `language` NULL/empty — the population the no-evidence guard above deliberately spares, so the two counters are disjoint by construction) ship status-only: category columns NULL, no `last_known_*` block, `domain_status` unchanged. Suppressed on a **judged** basis, not a re-score — `judged_basis` restates it: a blind judge upheld very few of the re-scored labels in the class, against a control of same-lane labels the re-score had left unchanged, and a large share of the sample was dead or contentless at fetch. The candidate class is every row matching `content_signal` that no sibling guard already suppresses. `row_count_overridden` counts the rows this export actually suppressed: that class minus the three exempt populations below (block-critical group, rank ≤ 1M, unresolved group). The changed count and the three exempt counts partition the class, so summing the four reconciles to it; rows a sibling guard already suppresses are counted under that guard's block, never here. Three populations ship **unchanged** and are counted apart, never in `row_count_overridden`: `block_group_exempt_row_count` (rows whose label projects to a block-critical policy group — `block_critical_group_policy` lists the eight; `block_group_exempt_groups` the ones actually held in that run — kept under the same exposure rule as `demoted_label_suppression.block_critical_held` until an in-group read clears the group), `head_rank_exempt_row_count` (rows in the popularity buckets listed by `head_rank_exempt_buckets`, i.e. the top 1M; a row whose rank is unknown — `rank_ladder` `absent`, or no inventory record — is not exempt), and `unresolved_group_exempt_row_count` (rows held fail-closed because their group could not be resolved; non-zero beside a `policy_pack` block whose `version` and `sha256` are null means the pack did not load and the guard was inert for that export). Filter-derived (`pipeline/*`), rule-derived and curated rows are never affected. Same transition rule as 1.8.0/1.10.0/1.11.0: consume the first ≥ 1.14.0 set as a **full per-partition replace**. Trends to 0 as re-crawls attach a language to these rows or retire them. | | `files` | Per-file name, size, and SHA-256 — verify your download against these. | | `created_at`, `export_dir` | When the export completed, and its directory name. | **On accuracy figures:** `row_count` is blended, so it is the wrong denominator. `content_accuracy_denominator_row_count` is published precisely so you do not have to know which lane is answerable — it is the `active` count, and non-active lanes are excluded from it by construction. ### The run-level `run_manifest.json` — is the snapshot COMPLETE? A full snapshot is delivered as **sixteen** partition directories (`snapshot__p00/` … `_p15/`), each with its own `manifest.json` — which proves that one partition, not the set. Listing the prefixes for a date cannot tell you 14/16 from complete, and a fan-out interrupted near UTC midnight can even leave a recovered partition under the _next_ day's date. So each completed snapshot run also publishes **one run-level marker**: ``` exports/snapshot_/run_manifest.json ``` It is written **only after** all sixteen partition manifests for that date are present and coherent (each parseable, matching its partition, with a positive `row_count`, and all on one `feed_schema_version`). It carries `type` (`snapshot_run`), `date`, `feed_schema_version`, `partitions_expected` (16), `partitions` (each partition's `manifest_key` and `row_count`), and `row_count_total`. From artifact 1.17.0 the sixteen must also have been tagged from one news-publisher list, which the marker names as `content_purpose_list` — `{"sha256": …, "row_count": …}`, without the partitions' `path` — so a set whose partitions were tagged from two lists gets no marker until it is re-published ([§14](#14-iab-content-purpose-columns-4546)). From artifact 1.7.0 it also rolls the per-partition keys up to run level, when and only when every one of the sixteen partitions carries them: `row_count_curated_total`, `rank_ladder`, `row_count_by_rank_bucket`, `inventory_total_by_rank_bucket`, `coverage_by_rank_bucket`, `row_count_dead_status_by_rank_bucket`, `inventory_dead_status_by_rank_bucket`, `inventory_never_shippable_by_rank_bucket` and `coverage_shippable_by_rank_bucket`, and (artifact ≥ 1.14.0) the status-age keys — `status_age_reference_date`, `row_count_by_status_age`, `stale_fraction_active_gt_90d` (re-derived from the summed `active` map, never averaged) and, with a present ladder, `row_count_by_status_age_by_rank_bucket` — which roll up only when every partition aged its rows against the **same** reference date. A key missing from even one partition is **omitted** from the roll-up rather than summed over a subset — a distribution over 14 partitions is not a distribution over the corpus. These keys never affect whether the marker is written; the completeness contract above is unchanged. From artifact 1.14.0 the marker also carries the churn block — `tier1_transitions_window`, `tier1_transitions_since`, `tier1_transitions_reason`, `row_count_tier1_changed`, `top_tier1_transitions` and `policy_group_transitions` — read with ONE query over our category-transition ledger for the interval between the previous snapshot's marker and this one, never assembled from the deltas published in between. It is telemetry on the same terms as the roll-up keys: it can never decide whether the marker is written, and a run that could not measure it publishes `tier1_transitions_reason` instead of a number. `tier1_transitions_window` is the half-open interval the three counts cover — `since` is the PREVIOUS snapshot's marker `created_at`, `until` is this marker's — so two consecutive snapshots tile the timeline and no ledger event is counted by both. `since` is established once, on the run that first publishes the date, and a re-run reuses it (see below). That previous marker has to be the one this chain published for its own date: it must sit at that date's key, declare that `date`, and enumerate that date's partition manifest keys under the same prefix. An object that fails any of those (including a marker for a different snapshot restored to the key) bounds nothing — the counts are withheld and `tier1_transitions_reason` says `predecessor marker unreadable`, never a number over a window we could not establish. `tier1_transitions_reason` is `null` exactly when the three counts are measured; otherwise it names the case, and two of them (`predecessor marker missing`, `no 1.14.0 predecessor`) publish an honest `0 / [] / []` because the chain simply starts there. Tiling needs the chain to be published in date order, and a recovery can break that: a snapshot date marked for the first time _after_ a later date was already published closes its window at the moment it is marked, which is past the later snapshot's own `created_at`, so its interval would contain that snapshot's entirely. Such a run publishes `a later snapshot's window is already published` and withholds the counts — with `tier1_transitions_since: null`, so a re-run of it decides the same way — rather than announce churn you have already been given. The later snapshot's marker, its bounds and its counts are not touched, and the recovered date still gets its completeness marker. If a later date's marker cannot be read at all, the run says `later snapshot marker unreadable` and withholds for the same reason. That check walks forward one date at a time and is bounded at five weeks, so a recovery more than five weeks late runs out of candidates before it reaches the day its own window would close on; it then says `successor scan exhausted before this run's upper bound` and withholds too, because the days it could not ask about may hold a window you have already been given. All three withhold with `tier1_transitions_since: null`, so a re-run of the date decides afresh rather than inheriting a bound that would skip the check. **Use it as the completeness gate:** treat a dated snapshot as complete iff its `run_manifest.json` exists, and read the sixteen manifest keys from it rather than globbing the prefix. Its absence on a current delivery means the set is still incomplete or being recovered — do not ingest a partial set. Snapshot dates published before this marker existed (up to and including 2026-08-02) have none; absence is only meaningful on deliveries after that. The marker describes the set **as of its `created_at`**, which is the date's FIRST published stamp: a re-run retains it rather than re-stamping the clock, because it is also the churn window's upper bound and the next snapshot's lower bound. The window's own `since` is retained from that same marker for the mirror-image reason — it is the interval you were told these counts cover — so a re-run re-derives **neither** bound. Without that, a marker for a date _inside_ the published interval appearing between two runs of the same snapshot (a re-published older date, an archive copy-back, a run recovered out of order) would move `since` forward on the re-run and shrink the counts beside it, with `until` unchanged and no reason given. Both values are only retained from an object that passes the same identity test the previous marker above has to pass — this date, and this date's partition manifest keys under the same prefix — so a marker for another set restored onto the key supplies neither bound; the run stamps its own clock, measures its own window, and the marker is still written. A re-run that cannot read that object back fails instead — the marker already on the key is left exactly as it is — so a published boundary never moves under a successor that has already been delivered. The retained `since` is read from `tier1_transitions_since`, which a marker carries whether or not it managed to measure over it — and whatever stopped it, a failed read, a timed-out one or no ledger configured on the run at all — so a run that could not measure does not release the bound and hand the run after it back to deriving one: a re-run publishes the bound it was handed, and a first run of a date publishes the one it had validated before its read failed or timed out (a first run with no ledger to read never locates a predecessor, so it publishes `null`); either way the interval survives the failure. A date whose marker carries `tier1_transitions_since: null` — no previous marker was located at all — has no bound to retain, so a later re-run of it measures normally. In the rare case that a date is re-published after its marker was written, the marker is rewritten when the re-publish completes; per-file integrity is always the `sha256` values inside each partition's own `manifest.json` — verify your downloads against those regardless of the marker. ### The snapshot-ingest obligation (contract term) Deltas alone cannot keep you compliant. Removals — GDPR takedowns and suppressions — retract a row, and only a full per-partition replace expresses a retraction; the status-only lane's baseline is likewise snapshot-borne. (Curated rows are not: from artifact ≥ 1.13.2 they ride every delta as well — [§5](#curated--the-hand-vetted-rows) — which changes nothing about this obligation, since neither removals nor the status-only baseline travel in a delta.) The licence therefore carries this term: > Licensee shall ingest each full snapshot within **14 days** of publication, > and shall apply it as a per-partition replace rather than a merge. Deltas > alone do not convey removals; a Licensee consuming only deltas will retain > records WhoisXML API has withdrawn. The 14 days are set against the platform's 30-day end-to-end erasure target: a withdrawn record leaves the next weekly snapshot (that set starts within seven days and is usually complete the following Monday, UTC), and your 14-day ingest window puts the worst case at about 22 days — inside the target, with margin for our own publishing latency. ### Replacing a set, and which partition a domain is in Deltas are not partitioned: each one mixes rows from all sixteen partitions and carries no partition column. A full set withdraws a row by no longer carrying it, so replacing one partition at a time removes a withdrawn row only if you know which partition it belongs to — rows a delta added included. Two ways to apply a set: - **Replace all sixteen partitions of a set together.** Load the complete set (its `run_manifest.json` exists) in place of everything you hold, delta-applied rows included, then resume with the deltas as [§1.1](#11-where-and-when-it-arrives) says. - **Or replace one partition at a time**, first deleting every row you hold that routes to that partition, delta-applied rows included; once it is reloaded, re-apply that partition's rows from the delta [§1.1](#11-where-and-when-it-arrives) names and from every later one. A domain's partition is PostgreSQL's hash-partition routing of the `domain` value exactly as published: 1. `h = hashtextextended(domain, 0x7A5B22367996DCFD)`: PostgreSQL's 64-bit hash of the name's UTF-8 bytes under the seed every hash-partitioned table uses, read as an unsigned 64-bit integer; 2. `hash_combine64(0, h)`, which for a single key is `(h + 0x49A0F4DD15E5A8E3) mod 2^64`; 3. that value `mod 16`, which is the `NN` of `snapshot__pNN`. In PostgreSQL itself the partition is `((hashtextextended(domain, 8816678312871386365) & 15) + 3) & 15` for a `text` column under a deterministic collation (the default). Anywhere else, this standard-library Python reproduces it exactly: ```python def feed_partition(domain: str) -> int: """The partition (0-15, the NN of snapshot__pNN) that holds `domain`.""" m32 = 0xFFFFFFFF def rot(x, k): return ((x << k) | (x >> (32 - k))) & m32 def mix(a, b, c): a = ((a - c) & m32) ^ rot(c, 4) c = (c + b) & m32 b = ((b - a) & m32) ^ rot(a, 6) a = (a + c) & m32 c = ((c - b) & m32) ^ rot(b, 8) b = (b + a) & m32 a = ((a - c) & m32) ^ rot(c, 16) c = (c + b) & m32 b = ((b - a) & m32) ^ rot(a, 19) a = (a + c) & m32 c = ((c - b) & m32) ^ rot(b, 4) b = (b + a) & m32 return a, b, c def final(a, b, c): c = ((c ^ b) - rot(b, 14)) & m32 a = ((a ^ c) - rot(c, 11)) & m32 b = ((b ^ a) - rot(a, 25)) & m32 c = ((c ^ b) - rot(b, 16)) & m32 a = ((a ^ c) - rot(c, 4)) & m32 b = ((b ^ a) - rot(a, 14)) & m32 c = ((c ^ b) - rot(b, 24)) & m32 return a, b, c # 1. hashtextextended(domain, 0x7A5B22367996DCFD): PostgreSQL's 64-bit lookup3. key = domain.encode("utf-8") seed = 0x7A5B22367996DCFD a = b = c = (0x9E3779B9 + 3923095 + len(key)) & m32 a = (a + (seed >> 32)) & m32 # the seed is hashed first, as a 12-byte block b = (b + (seed & m32)) & m32 a, b, c = mix(a, b, c) while len(key) >= 12: a = (a + int.from_bytes(key[0:4], "little")) & m32 b = (b + int.from_bytes(key[4:8], "little")) & m32 c = (c + int.from_bytes(key[8:12], "little")) & m32 a, b, c = mix(a, b, c) key = key[12:] tail = key.ljust(11, b"\0") # the last 0-11 bytes; the low byte of c is never written a = (a + int.from_bytes(tail[0:4], "little")) & m32 b = (b + int.from_bytes(tail[4:8], "little")) & m32 c = (c + (int.from_bytes(tail[8:11], "little") << 8)) & m32 a, b, c = final(a, b, c) h = (b << 32) | c # 2. hash_combine64(0, h), then 3. mod 16. return ((h + 0x49A0F4DD15E5A8E3) & 0xFFFFFFFFFFFFFFFF) % 16 ``` Check it on your own delivery before relying on it: every `domain` in `snapshot__p07.parquet` returns `7`. --- ## 11. The reserved `site_purpose_*` block and the `policy_group` projection Columns 27–38 (artifact ≥ 1.12.0) add two things: a **reserved second axis** and a **projection** over the label. Neither changes any existing column. ### The reserved `site_purpose_*` block (columns 27–35) The stored label (`tier1..tier4`) is the primary label under the purpose-first single-label convention (§2). The `site_purpose_*` columns are reserved for a **separately modelled** business-purpose answer, produced by a different model against the same taxonomy — nothing populates them today — in the same IAB v3.1 vocabulary (`Shopping` for a store, `Attractions > Casinos & Gambling` for a sportsbook), with its own confidence, its own provenance (`site_purpose_model_version`) and a `site_purpose_relation` to the primary label (`same_path` / `same_tier1` / `different_tier1`). At 1.12.0 **no purpose pass has run**: every one of these columns is NULL on every row except `site_purpose_status`, which is **never NULL** and tells you why the rest is empty: | `site_purpose_status` | Meaning at 1.12.0 | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `not_available` | Not assessed. Every live row until a purpose pass runs. Do **not** read this as "no purpose". | | `undefined` | The domain has no separate business purpose: the `parked`, `inactive` and `dead` lanes, and rows the existing parked detectors flagged (the parking-stub fingerprint, except on a popularity-bucket-`0` or `1` capture from 2026-07-31; the parked-demoted label). Derived from the same status evidence as `domain_status`, never from a model. Sensitive-filtered rows are _not_ undefined — they have a purpose and read `not_available` until labeled. | | `labeled` | The purpose columns are populated. Does not occur at 1.12.0; `site_purpose_model` in the manifest is `null` until it does. | The columns are reserved now so that one schema bump, not two, carries both features and you build your table once. `row_count_by_site_purpose_status` in the manifest shows the split without a scan. When a purpose pass lands, the purpose columns are **tracked** like the category columns (a purpose-label change alone produces a delta row). These nine columns **keep their names and positions**: they will not be renamed or repurposed, under the [§1 column-order contract](#the-column-order-is-the-contract). Eight stay NULL and `site_purpose_status` keeps its documented values until a purpose pass is announced in the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog). A second axis, if one ever ships, arrives as **new columns appended** at the end of the then-current schema, never by changing what these hold. ### `policy_group`, `policy_basis`, `policy_pack_version` (columns 36–38) `policy_group` is the row's label rolled up into one of the pack's 47 **policy groups** — the filtering vocabulary (`adult`, `gambling`, `piracy`, `drugs`, `weapons`, `hate`, `dating`, `social_networking`, `shopping`, …) that DNS-filter, secure-web-gateway and endpoint policies already use. `policy_basis` says which axis it was projected from (`primary` = the tier path; `site_purpose` once that axis is labeled, which takes precedence; `safety` from artifact 1.14.0, which takes precedence over both). `policy_pack_version` is the pack and mapping version the row was projected with. The full group list, each group's definition and the IAB nodes it covers are in [`website-categorization-policy-pack-v1.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v1.md) and, from artifact 1.14.0, [`website-categorization-policy-pack-v2.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v2.md). **Which pack, by artifact version.** `policy_v1@2026.09.2` at 1.12.0–1.13.x; `policy_v2@2026.09.1` from **1.14.0**. `policy_v2` changes the projection in two ways, both of them visible in this column: the brand-safety axis is read before the label (§13), and the group `news` is retired in favour of **`current_events`** — the same three current-affairs roots (`Crime`, `Disasters`, `War and Conflicts`), an id that says what they are, and the reason the pack ID moved rather than the mapping version. **A policy keyed on `news` matches nothing from 1.14.0 on**; see the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog). No other path changes group and the group count is unchanged. Why it exists: IAB v3.1 scatters what a blocking policy treats as one thing. Gambling alone lives at `Attractions > Casinos & Gambling`, `Shopping > Lotteries and Scratchcards`, `Sports > Poker and Professional Gambling` and a `Video Gaming` genre — an integration that blocks the `Sensitive Topics` subtree blocks none of them. The three real-money nodes project to `gambling` and the video-game genre (simulated gambling, no real-money play) to `gambling_games` (mapping 2026.09.2); a policy is one equality test per group on this column. How it is computed — and what that means for you: - **Through 1.13.x (`policy_v1`) it was a pure function of the stored label**: the deepest rule matching the tier path won (a label stored as `Attractions > Casinos & Gambling` → `gambling`; one that stopped at `Attractions` → `travel`), it read nothing else about the domain, and `policy_basis` was `primary` on every projected row. - **From 1.14.0 (`policy_v2`) the label is no longer the only input.** The projection reads the **brand-safety axis first** — the flagged themes in `safety_flags` (§13) — then `site_purpose` (still unlabeled, so unreachable today), then the tier path, and `policy_basis` names the axis that answered. It still never changes any other column. Three consequences a policy has to be written against: - A flagged row whose label sits in a **non-sensitive** group takes the theme's group: `gambling` flagged on a row labelled `Automotive` reads `policy_group = gambling`, `policy_basis = safety`. So **`policy_group` need not correspond to `tier1` at all**, and an integration that re-derives the group from the tier path will disagree with the column on exactly these rows. - **A row already in a block-critical group never leaves it.** When the label resolves to a _different_ `sensitive` group and no flagged theme names that group, the **label** answers and `policy_basis` reads `primary`; the verdict stays visible in `safety_flags`. The one exception is the theme-unknown verdict (`sensitive`, theme not identified), which is never more specific than a theme-specific one — an `adult` flag on a bare `Sensitive Topics` label reads `adult` / `safety`. - A flagged theme this pack maps to no group, and one it does not know, are **ignored** — the projection runs over every row and must be total. - It **cannot correct a wrong stored label.** A domain whose category is wrong projects to the wrong group; an adult site stored as `Entertainment > Movies` reads `entertainment`. Per-group precision and recall have **not been measured**, and no accuracy figure attaches to any group. Evaluate on your own domains before blocking. - It expresses only what IAB v3.1 has nodes for. There is **no malware, phishing, botnet or proxy/anonymizer group**, because the taxonomy has no such category; threat intelligence is a separate WhoisXML product. - **When the three columns are NULL.** Through 1.13.x that was exactly where `tier1` was NULL or was not an IAB v3.1 tier-1 name. **From 1.14.0 that is no longer the rule**: a row carrying a flagged theme the pack maps to a group resolves from the safety axis alone, so `policy_group`, `policy_basis` and `policy_pack_version` are **populated (basis `safety`) even where `tier1` is NULL or unresolvable**. From 1.14.0 the three columns are NULL exactly where: - the row is **suppressed** — parked, inactive, dead, and the stub/no-evidence/template/demoted guards on any lane. The whole category block is blanked on those rows, so the policy columns go NULL there whatever the safety axis says. `safety_flags` and `safety_detail` are blanked with them; `safety_status` is **preserved**, and is the only reading that survives on a suppressed row; **or** - **neither axis resolves** on a live row: no flagged theme names a group **and** `tier1` is NULL or is not an IAB v3.1 tier-1 name. A deeper tier that does not resolve (an unknown `tier2`/`tier3`/`tier4` name) falls back to the deepest resolvable prefix — the same rule as `openrtb_cat`, which keeps the ids of the resolvable prefix — so a group read from the _label_ can only ever be the parent's, never a more specific claim than the label supports. None of the three is carried into the `last_known_*` block. - The policy columns are **derived, not tracked.** The mapping is versioned independently of the schema: `manifest.json` carries `policy_pack: {id, version, sha256}`, and a pack edit (say, moving casino-themed video games out of `gambling`) bumps the version and is listed in the changelog with **no column change**. Because it touches every row, a pack version change ships via the next full snapshot, never as a delta: every delta carries the `policy_pack` of the snapshot it extends, a delta whose pack differs from its predecessor is a chain gap, and the exporter refuses to write one. So **the first ≥ 1.12.0 artifact you ingest must be a snapshot**, and after any `policy_pack` change you will receive a snapshot before the next delta. **The pack `id` is half of that identity, not decoration**: `policy_v1@2026.09.1` and `policy_v2@2026.09.1` carry the same `version` and different mappings, so compare `policy_pack` as the `id` + `version` pair — the `@` form the per-row `policy_pack_version` column already uses. Pin that pair if your policy depends on exact membership. ### `policy_confidence_basis` and `policy_group_set` (columns 42–43) Two columns that make the policy block usable without a lookup table. **`policy_confidence_basis` says what column 7 / column 24 IS on this row.** [§5](#5-confidence-and-tier) documents that `confidence` is not one scale — a trained classifier's probability, a filter's internal score, a keyword density and a hand-chosen trust value all live in the same column, and until now the only way to tell them apart was to know the `model_name` families by heart. The value is a **closed enum**: | Value | What the number is | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_calibrated` | A measured probability: `calibrated_confidence` is populated on this row, and it is the number to threshold on. | | `model_raw` | A trained classifier's own, uncalibrated probability. | | `filter` | A content filter decided the row outright; the number is that filter's internal score, not a probability for the tier path. | | `rule` | A deterministic rule decided or rewrote the label (`tld_rule/*`, `schema_rule/*`, `pipeline/vendor_rule`). Provenance, not scale: a `tld_rule/*` row carries the rule's fixed trust value while a `pipeline/vendor_rule` row carries the classifier's own probability unchanged. | | `curated` | A hand-written verdict — a curated list entry, or an accepted `human-feedback` review; a fixed trust value chosen when it was written, never measured against outcomes. | | `carry_over` | Carried over from a previous-generation system rather than produced for this label. | | `name_only` | Two language models' self-reported probability (the lower of the two) that the domain NAME identifies the site's category — no page was read (writer family `llm-name/*`, artifact ≥ 1.15.0). Never `model_calibrated`; band capped at `MEDIUM`. | | `none` | No label to attribute the number to, or no writer recorded. | The **writer family is decided first**: a filter, rule, curated, carry-over or name-only row keeps its family whatever the calibrated column holds, so `model_calibrated` appears only where the row would otherwise read `model_raw`. Two identities hold on any artifact: `count(policy_confidence_basis = 'model_calibrated')` equals `row_count_calibrated`, and every `*_filter` writer reads `filter`. `policy_basis` (column 37) is a different question — which AXIS the group came from — and from 1.14.0 its documented values are `primary`, `site_purpose` and **`safety`**. `role` is reserved for a later release and is not emitted at 1.14.0. If you validate column 37 against a strict enum, add `safety` before ingesting a 1.14.0 artifact; this is the one **breaking value addition** in the release, and it is confined to an opt-in block. **`policy_group_set` is the group plus its declared union.** Blocking "gambling-shaped" content means matching `gambling`, `gambling_games` and `harmful_content`; blocking drugs means `drugs`, `alcohol` and `tobacco`. The pack declares those unions, and this column materializes them per row, own group first, then its partners in pack order: ``` policy_group = "gambling" → policy_group_set = ["gambling", "gambling_games", "harmful_content"] policy_group = "shopping" → policy_group_set = ["shopping"] ``` One `array_contains` replaces a join against the taxonomy endpoint. It is NULL exactly where `policy_group` is NULL, so the two columns can never disagree about whether the projection ran. --- ## 12. Quick reference - 46 columns, fixed order, additive-only. Read by name where you can. Columns 27–35 are the reserved `site_purpose_*` axis (NULL through 1.13.x except `site_purpose_status`); columns 39–44 were appended at 1.14.0 and columns 45–46 at 1.17.0. - `content_purpose_id` (column 45) is `1020`, IAB Content Purpose = News, on the domains of a curated list of known news publishers and NULL everywhere else — a list, not news-site coverage, and NULL is unknown, never "not news". The topic columns are unchanged ([§14](#14-iab-content-purpose-columns-4546)). - `domain_status` has six values. `parked` / `inactive` / `dead` carry no category at all; `active`, `blocked` and `uncategorizable` may. `active` is not a promise of a label — test `tier1 IS NOT NULL` instead ([§3](#3-domain_status--the-six-lanes)). - Want a reliability signal? Use `calibrated_confidence` (a probability), **not** `confidence` (an uncalibrated model score). See [§5](#5-confidence-and-tier). - `calibrated_confidence` is populated on a **subset** of categorized rows. `tier1 IS NULL` implies it is NULL; the converse does not hold — see [§5](#5-confidence-and-tier). - `calibration_segment = '__global__'` marks the weakest calibrated rows. - `blocked` = live but unreadable. Not parked, not dead — and it **keeps** its category and its calibrated value from the previous crawl. A `304 Not Modified` answer is not `blocked` (from the 2026-09-17 set). - `classified_at` = when we processed it. `status_observed_at` = when the crawl saw the status. Neither = when the content changed. - `confidence` is not one scale: on filter rows (`*/st_filter`, `*/gambling_filter`, `*/adult_filter`) and rule/lookup rows (`curated/*`, `tld_rule/*`, `schema_rule/*`, `v3_historical_fallback`) it is a component's own score, not a category probability — check `model_name` before comparing or thresholding ([§5](#5-confidence-and-tier)). - `pipeline/vendor_rule` rows carry the classifier's `confidence` under a label the rule chose (`Business and Finance` → `Technology & Computing` on vendor sites); `calibrated_*` is NULL there ([§5](#pipelinevendor_rule--the-vendor-rule-flip)). - `curated/*` rows are hand-vetted, `tier = 3`, always `domain_status = active` by assertion, always NULL `calibrated_*`, and ride **every snapshot and every delta** (deltas from artifact ≥ 1.13.2) — **apply them by `domain`, never by `classified_at` order**, because a curated row is usually dated earlier than the row it replaces ([§5](#curated--the-hand-vetted-rows)). - `model_version` may carry a `+variant` suffix (`v1+ml_en`) — compare on the base version, never with `=` ([§5](#5-confidence-and-tier)). - `openrtb_cat` is an array — JSON-encoded in CSV, native list in Parquet. - `policy_group` is the policy-pack rollup (`gambling`, `adult`, …), versioned by `policy_pack_version` / `policy_pack.version`. Below 1.14.0 it was the `policy_v1` rollup of the tier path and nothing else, so it was NULL exactly where that path did not resolve; from **1.14.0** the pack is `policy_v2`, which reads `safety_flags` first and renames `news` to `current_events`. So from 1.14.0 a flagged row carries a group **even where `tier1` is NULL**, and the column is NULL only on a suppressed row (`parked` / `inactive` / `dead` and the stub guards) or where neither axis resolves — never read `tier1 IS NULL` as "no group". `site_purpose_status` is never NULL ([§11](#11-the-reserved-site_purpose_-block-and-the-policy_group-projection)). - `last_known_*` is historical and sparse. Read your delivery's own `row_count_with_last_known_category`; figures quoted in documentation age ([§8](#8-the-last_known_-columns)). - NULL is an empty CSV field / a true Parquet null. No sentinels. - A dated snapshot is complete **iff** `exports/snapshot_/run_manifest.json` exists — read the sixteen partition manifests from it instead of globbing ([§10](#10-manifestjson)). - `safety_status` (column 39) is **never NULL** and is kept even on a suppressed row; `safety_flags` / `safety_detail` are NULLed with the category block. `safety_status` and `safety_flags` read **brand-safety** verdicts only ([§13](#13-the-brand-safety-axis-columns-3941)). - `policy_confidence_basis` (column 42) tells you what the number in column 7 / 24 is without knowing the `model_name` families; `policy_basis` (column 37) gains the value `safety` at 1.14.0 — a breaking value addition if you validate it against a strict enum ([§11](#policy_confidence_basis-and-policy_group_set-columns-4243)). - `policy_group_set` (column 43) is the group **plus its declared union** — match one list instead of rebuilding `gambling ∪ gambling_games ∪ harmful_content` yourself. - `language_primary` (column 44) is the BCP-47 primary subtag of column 14. `eng` stays `eng` — it is a valid three-letter subtag and is deliberately **not** folded to `en`, because the two spellings come from different detectors and folding them would claim an equivalence we have not measured. Column 14 stays raw. --- ## 13. The brand-safety axis (columns 39–41) Three columns, appended at artifact **1.14.0**, carrying what our brand-safety filters concluded about a domain — separately from what its category says. Why it is a separate axis: a category answers "what is this site for?", and a blocking policy usually also needs "did anything flag it?". Those are different questions, and they disagree in both directions — a gambling operator labelled `Attractions > Casinos & Gambling` is normally flagged, a news outlet reporting on gambling normally is not, and a parked ex-gambling domain can carry a flag with no category at all. Neither filter is exact: a model-adjudicated read of the 2026-10-18 set found the sensitive-content filter flagging some clearly legitimate sites at the head. Reading the second question off `tier1` or off `model_name` is what customers were doing, and it under-blocks by construction. | Column | Type | Value | | ------------------ | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 39 `safety_status` | `string`, **never NULL** | `flagged` — at least one brand-safety filter raised a block signal; `assessed_clean` — a filter looked and raised none; `not_assessed` — nothing looked at this domain. | | 40 `safety_flags` | `list` | The flagged themes, e.g. `["gambling"]`. On an unsuppressed row `safety_status = 'flagged'` **iff** this list is non-empty, and the list is an empty list rather than NULL. NULL on a suppressed row — which keeps its `safety_status`, so a NULL list is not evidence the row is unflagged. | | 41 `safety_detail` | `string` (JSON object) | One entry per filter family that ran, carrying its `status`, `score`, `scale`, `theme` and `version` — a family reporting `not_assessed` is retained here, so a populated detail is not itself evidence that anything assessed the domain. NULL on a suppressed row, and on a row where no verdict was materialized at all. | **`safety_status` is preserved through suppression, like `site_purpose_status`.** A parked or dead row ships no category, but "a brand-safety filter looked at this domain" stays true whatever the feed does with the label — so column 39 keeps its value while columns 40 and 41 go NULL with the rest of the category block. A `flagged` status on a parked row is a fact about the assessment, not a category claim; the flags that would substantiate it are not published on that row, by design. **What counts as a brand-safety verdict — and what does not.** `safety_status` and `safety_flags` are computed over **brand-safety verdicts only**. The vendor rule is a ROLE classification, not a brand-safety one: it decides whether a `Business and Finance` label belongs on a company that _sells to_ that industry rather than _operating in_ it. Its verdict appears in `safety_detail` under the family `vendor` **and nowhere else** — never in `safety_flags`, never enough to make a row `flagged`, and never enough to move a domain out of `not_assessed`. A `role` basis is reserved for a later release. Shadow themes (measurements whose precision has not been read yet, such as `adult_nonadjacent`) are recorded in `safety_detail` on the same terms: they are not block signals. **Rows written before the verdict store.** A domain's verdicts are stored from the 1.14.0 release onwards. Rows whose last write predates it are **derived from the row's existing `model_name` and `tier2`** and carry a `legacy` version marker in `safety_detail`. That derivation exists so the axis is complete on the first artifact rather than reading `not_assessed` for every pre-existing filter row — a domain last crawled 80 days ago is 90-day-shielded and would otherwise never gain a verdict. For those rows the flag is a re-statement of what column 10 already told you, not new evidence; `safety_detail`'s version marker is how you tell the two apart. **Reading it.** `safety_status = 'flagged'` is the coarse gate. `safety_flags` is the one to key a block list on if you want every flagged row regardless of where its label landed — `policy_group` under `policy_v2` reads the safety axis first, so a flagged row usually carries the theme's group, but a row whose label already resolves to a _different_ sensitive group keeps that group instead. `safety_detail` is diagnostic: read it to understand a verdict, not to build a filter, because its shape is per-family and will gain families. --- ## 14. IAB Content Purpose (columns 45–46) Two columns, appended at artifact **1.17.0**, that say a domain is a news publisher in the IAB's own vocabulary for it. The first artifact carrying them is the Wednesday 2026-11-04 delta (`delta_2026-11-03_100000Z`); the first full set carrying them is the Sunday 2026-11-08 set. **Why a separate pair of columns.** IAB Content Taxonomy v3.1 has no News topic, so a news outlet's `tier1`–`tier4` name what it covers — `Politics` under the single-label convention ([§2](#2-column-reference)), or `Sports`, `Business and Finance` and so on. The IAB Tech Lab Content Taxonomy 3.0 Implementation Guide (section 8, Descriptive Vectors) expresses news as a **Content Purpose**, carried next to the topic and never in place of it: `Content Purpose > Informational > News`, unique id `1020`. These two columns carry that vector. The topic columns are unchanged on every row: filter news on column 45, and keep your topic rules on columns 2–6. | Column | Type | Value | | --------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 45 `content_purpose_id` | `string` | `1020`, or NULL. A Descriptive Vector id, **not** a Content Taxonomy v3.1 id: it never appears in `iab_id` or `openrtb_cat`, and no tier name corresponds to it. | | 46 `content_purpose_source` | `string` | Which rule set column 45: `curated`, or NULL exactly where column 45 is NULL. `schema_org` is a **reserved** value — see below. | **Where the value comes from: a list, and only a list.** In the feed the only source is a curated list of known news publishers maintained by our team; the list is versioned, and every change to it is reviewed before release. Every domain on it is also one of our hand-vetted `curated/*` rows ([§5](#curated--the-hand-vetted-rows)), so the value appears **only on `curated/*` rows**. No model sets these columns and no page is read to set them. That makes this **a list, not news-site coverage**: most news sites in the feed are not on it and read NULL, and will until a later release adds another source. `content_purpose_list.row_count` in any manifest says how many domains the list names ([§10](#10-manifestjson)). **NULL means unknown, never "not news".** A NULL in column 45 says we hold no news-publisher statement about the domain. It is not a statement that the site is not a news outlet, and it must not be used as one — for example, to build an "exclude news" list out of the NULL rows. **`schema_org` is reserved.** The real-time API's opt-in `content_purpose` block (`include=content_purpose`) has a second source, `schema_org`: the site's own schema.org `NewsMediaOrganization` markup, read from the crawled page. The feed does not emit it in 1.17.0, because the export carries no page markup. It is named now so a loader that validates column 46 against a strict enum can accept it ahead of time; a release that starts emitting it will be announced in the [Website Categorization feed changelog](https://website-categorization.whoisxmlapi.com/database/specifications/changelog) first. **The same answer as the API, where both give one.** For a domain on the list, column 45 and the API block's `id` are both `1020`, and column 46 and its `source` are both `curated`. A domain the API tags from its markup alone (`schema_org`) reads NULL here. The legacy v1 response field that is also called `content_purpose` is a different thing: reserved, null in every deployment, and not carried by these columns. **How it reaches you.** Curated rows ride every snapshot and every delta whole, so the deltas are enough: if you consume deltas only, add the two columns as NULL to your table and apply the first 1.17.0 delta — the two columns then hold exactly what a 1.17.0 full set publishes in them. When the list changes, the next delta re-sends every curated row with its current value, so a domain taken off the list reads NULL on your side after that delta. A domain taken off our curated list altogether stops riding the deltas, so your copy keeps its last curated row, tag included, until the domain ships again from our other lanes — in a later delta or the next full set, with columns 45–46 NULL — or a full set no longer carries it ([§10](#the-snapshot-ingest-obligation-contract-term)). **Suppression.** Both columns belong to the content-category block, so they are NULL on any row whose category is withheld ([§9](#9-why-a-non-active-row-has-no-category)). A curated row is always `active` ([§3](#3-domain_status--the-six-lanes)), so this removes no value today. **Counting it.** `row_count_by_content_purpose_source` counts rows per column 46 value, and `content_purpose_list` names the list the values were read from — its `sha256` and its `row_count` ([§10](#10-manifestjson)). On a delta, and on a full set's run marker (which sums the sixteen partitions), the `curated` count equals the list's `row_count` unless a listed domain did not ship as a curated row in that artifact — a takedown withholds it, for example. A partition manifest counts only its own partition's tagged rows. The run marker also names the one list the whole set was tagged from; a set whose partitions were tagged from two different lists is not marked complete.