# Website Categorization Bulk Feed — Changelog Consumer-visible changes to the bulk feed's published values and semantics — including the ones that are **invisible in the schema**: same columns, different meaning or different numbers. Nothing else forces a consumer to notice those, so they are called out here. Read this page beside the data dictionary [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary), which defines every column and every value domain. Entries are newest first. **Download this document:** [`website-categorization-feed-changelog.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-feed-changelog.md) **Also available:** the data dictionary [`website-categorization-feed-data-dictionary.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-feed-data-dictionary.md), the policy pack `policy_v1` group reference [`website-categorization-policy-pack-v1.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v1.md) and the policy pack `policy_v2` group reference [`website-categorization-policy-pack-v2.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v2.md). --- ## How to read a version number `feed_schema_version` in `manifest.json` is a semantic version, and the component that moves tells you what to check. A **MINOR** bump is purely additive to the schema: new columns appended at the end of the contract order, new manifest keys, or a new manifest block — a reader that selects by name, or that stops reading after the column it integrated against, is unaffected by the append itself. A **PATCH** bump changes the published **row population** with no column change at all: no column is added, removed, renamed or retyped, and no manifest key is removed or given a new meaning, though a PATCH may add keys inside an existing manifest object — so validate the manifest permissively, as an extensible object rather than a closed set of properties. A **MAJOR** bump would reorder or rename a column; that is the licensable contract and it does not move without notice. Two bumps below are MINOR because they do both things at once — they append columns **and** move the row population — and each says so in its own entry. One rule cuts across several entries and is worth stating once: **a change that is derived at export rather than tracked per row reaches you only through a full snapshot.** A row whose band, policy group or suppression status moves under a new rule is not a _changed_ row and does not cross a delta cutoff. Every entry that carries that property says **"consume the first ≥ x.y.z set as a full per-partition replace"**, and the exporter holds the same line on its side: a delta whose policy differs from the snapshot it extends is reported as a chain gap and is refused. --- ## Delivery cadence | Artifact | Schedule | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Full snapshot** | Weekly, **starting on Sundays**, and lately taking about 30 hours to run. Delivered as sixteen partition directories; complete when `run_manifest.json` appears, usually on Monday, UTC. | | **Incremental delta** | Daily at **10:00 UTC**, every day except Sunday — the snapshot holds the export lock then — landing within about an hour on weekdays. The next delta spans the gap. The Monday delta waits for the weekly set's run marker, so it lands after the set does. | Export folders are named by the **start of the window** they cover, not by the time the export finished. A dated snapshot is complete **if and only if** `exports/snapshot_/run_manifest.json` exists: read the sixteen partition manifests from that marker rather than listing the prefix, and do not ingest a partial set. The marker's absence on a current delivery means the set is still being written or recovered. Deltas alone cannot keep you compliant. Removals — takedowns and suppressions — retract a row, and only a full per-partition replace expresses a retraction; the status-only lane's baseline is likewise snapshot-borne. The licence carries a term requiring each full snapshot to be ingested within an agreed window and applied as a per-partition replace rather than a merge. See the data dictionary [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) for the exact wording. --- ## 2026-11-03 — three corrections to the data dictionary (wording only, no schema or version change) Nothing published changes with this entry: no column, value or row moves. **Corrected in the data dictionary published beside this entry** [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary): - §3: `uncategorizable` rows keep whatever label is already on file, if there is one. The dictionary no longer adds that usually there is none. - §5: the note that introduces `calibrated_confidence` now says what it estimates, as the column's own section does: the probability that a strict blind judge, reading the page, assigns the same `tier1`, a strict-match estimate rather than the chance that a label is usable. It said only that the column is meant to be read as a probability. - §10, the snapshot-ingest term: the next weekly set starts within seven days and is usually complete the following Monday, UTC, as §1.1 says. It said the set is complete by the following Monday morning. --- ## 2026-11-03 — the run marker's `policy_group_transitions` projects each move's full tier path, with an erratum for the sets from 2026-09-20 through 2026-11-01 (first artifact: the 2026-11-08 set's run marker; no schema or version change) No column, manifest key or `feed_schema_version` changes. From the 2026-11-08 set, the run marker's `policy_group_transitions` projects each move through the full tier path it was recorded at, as data dictionary §10 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) now says. Deltas still do not carry the key. **Erratum.** From the 2026-09-20 set through the 2026-11-01 set, the projection read each move's tier-1 label alone, so a move whose path falls under a deeper rule of the policy pack was counted in the group of its bare tier-1 label. The 2026-09-17 set, the first 1.14.0 set, had no earlier marker to measure from and published `[]`. Deeper rules sit under several tier-1 labels: under `Sensitive Topics`, for example, deeper paths map to groups such as `adult`, `hate` and `weapons`, while the bare label maps to `sensitive_other`. In the 2026-10-18 set's marker, a move to or from `Attractions > Casinos & Gambling` (`gambling`) was counted as `travel`, the group of bare `Attractions`. Other `policy_group_transitions` entries can also have counted part of their moves in the wrong group pair. `top_tier1_transitions` and `row_count_tier1_changed` are unchanged. --- ## 2026-11-03 — for domains ranked in the top million (popularity buckets 0 and 1), a page that gave us no readable text ships as `blocked`, not `parked` (first artifacts: `delta_2026-11-03_100000Z` and the 2026-11-08 set; no schema or version change) No column, manifest key or `feed_schema_version` changes. What changes is the lane of one group of rows: those whose crawl succeeded but whose capture extracted to no readable text — no title, no description and no body text. - **Before:** every such row that held a category read from a page shipped as `parked`, with its category columns NULL, because the capture's content fingerprint was read as the parking-provider redirect stub that produced most of these captures in July 2026. A page rendered entirely by JavaScript, a meta or JavaScript redirect and a bot interstitial produce the same fingerprint, so live sites shipped as `parked`. That contradicted the data dictionary's meaning of `parked`, "an affirmative parking signal". - **Now:** since 2026-07-31 our crawler recognizes the parking-provider redirect stub when it fetches it and records the domain as parked directly. On a domain in popularity bucket `0` or `1` — a best rank of 1,000,000 or better on the Tranco, Majestic Million or Cisco Umbrella list (data dictionary §10 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary)) — a capture made from that date that extracted to no readable text is read as a live site we could not read, and its row ships as `blocked`. Most such pages are live; a minority are hosting placeholders or parking pages our crawler could not recognize. Everywhere else such a row still ships as `parked`: outside the Tranco top million, many of the pages we checked turned out to be parking, for-sale or registrar pages, or hosting placeholders, whose text sits in an image or a script, so an empty capture alone cannot tell them from a live site. A capture made before 2026-07-31 still ships as `parked` until a crawl fetches the page again: a `304 Not Modified` answer keeps the earlier capture, and with it the `parked` lane. An export run without the popularity buckets (`rank_ladder: absent`) moves no row. Data dictionary §3 now says so for both lanes. - **What does not change:** the category stays withheld on these rows wherever they land, with no `last_known_*` block, because no label can be checked against a capture with no text. A domain our crawler recorded as parked (a parking page, a for-sale page, the redirect stub itself) still ships as `parked`. A row whose only label is a name-only one keeps the lane its crawl decides, as since the 2026-10-02 entry below. `parked_stub_suppression.row_count_overridden` counts the same rows as before. - **`site_purpose_status`:** on a `blocked` or `uncategorizable` row in popularity bucket `0` or `1` whose empty capture was made from 2026-07-31 it now reads `not_available` instead of `undefined`, including rows whose lane does not move, because an empty capture is not evidence that a domain has no business purpose. Rows on `parked`, `inactive` and `dead` still read `undefined`, as does a parked-demoted label (data dictionary §11), and `row_count_by_site_purpose_status` in `manifest.json` moves with it. - **What you will see:** rows that qualify move from `parked` to `blocked`; no row outside popularity buckets `0` and `1` changes lane. Live checks of a sample of these pages found mostly live sites, along with pages with no site behind them, redirects to other domains, JavaScript apps that looked like gambling sites (judged without rendering them), very few parking or for-sale pages, and hosting placeholders or blank pages, which are wrong as `parked` too: their lane is `inactive`. - **When:** from the first daily delta and the first Sunday set exported after the deploy that carries this change. The lane is decided when the feed is exported, so a row that moves is not a changed row: a daily delta carries it only when we crawl the domain again within that delta's window, and the first full set after the deploy is the first artifact that carries every such row. A domain's bucket is re-stamped monthly, so a domain that enters or leaves buckets `0` and `1` can change lane without being crawled again; that move, too, reaches you only through a full set. --- ## 2026-11-02 — the 1.17.0 notice: IAB Content Purpose = News in two appended columns (first artifacts: the Wednesday 2026-11-04 delta and the Sunday 2026-11-08 set) Nothing published changes with this entry. It is the **customer notice** for feed 1.17.0, dated ahead of its first artifact. `feed_schema_version` **1.16.0 → 1.17.0**, a MINOR bump on the definition in data dictionary §1 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary): two columns are **appended** after column 44, and columns 1–44 keep their names, positions and types. Reading by name, or by position up to column 44, is unaffected, and no row enters or leaves the feed. | # | Column | Type | Value | | --- | ------------------------ | -------- | ---------------------------------------------------------------------------------------- | | 45 | `content_purpose_id` | `string` | `1020` — IAB **Content Purpose > Informational > News** — or NULL | | 46 | `content_purpose_source` | `string` | `curated`, or NULL exactly where column 45 is NULL; `schema_org` is reserved (see below) | **What it says.** IAB Content Taxonomy v3.1 has no News topic, so a news outlet's `tier1`–`tier4` name what it covers — `Politics`, `Sports` and so on. The IAB Tech Lab expresses news as a **Content Purpose**: a Descriptive Vector carried next to the topic, never in place of it, and column 45 carries its id. No topic column changes: filter news on column 45 and keep your topic rules on columns 2–6. `1020` is not a Content Taxonomy id and never appears in `iab_id` or `openrtb_cat`. **Where it comes from — a curated list, and only that.** The value is set on the domains of a curated list of known news publishers maintained by our team, and on no other row; the list is versioned, and every change to it is reviewed before release. Every domain on the list is also one of our hand-vetted `curated/*` rows, so the value appears only on those rows. The feed reads no page markup, so it cannot tag a news site that is not on the list: **this is a list, not news-site coverage.** Most news sites in the feed read NULL. **NULL means unknown, never "not news".** Do not build an "exclude news" rule out of the NULL rows. **`schema_org` is a reserved value of column 46.** The real-time API's opt-in `include=content_purpose` block has a second source, `schema_org` — the site's own schema.org `NewsMediaOrganization` markup. The feed does not emit it in 1.17.0, because the export carries no page markup. It is named now so a loader that validates column 46 against a strict enum can accept it in advance; a release that starts emitting it will say so here first. For a listed domain the feed and that API block agree (`1020`, `curated`); a domain the API tags from markup alone reads NULL in the feed. **When.** The 1.17.0 exporter deploys **no earlier than Tuesday 2026-11-03 at 13:00 UTC**, after that day's 10:00 UTC delta; if this date moves, we will say so in a dated amendment to this entry. If it deploys then, the first 1.17.0 artifact is **Wednesday 2026-11-04's delta**, in the folder `delta_2026-11-03_100000Z`, and the first 1.17.0 full set is the **Sunday 2026-11-08** set. Nothing published before the deploy carries the two columns. **The deltas are enough for this change.** Curated rows ride every delta whole, so the Wednesday 2026-11-04 delta already carries every tagged row. If you consume deltas only, add the two columns to your table as NULL and apply that delta: the two columns then hold exactly what a 1.17.0 full set publishes in them. A chain check that compares each artifact with the one before it (data dictionary §10) reports the 1.16.0 → 1.17.0 version step on that first delta — it cannot know what a version change carries — so expect that one report; this entry is what it points to. **The 1.16.0 rule still binds.** 1.17.0 carries 1.16.0's band policy unchanged: column 8 and `confidence_band_policy` (`v4`) mean exactly what the 1.16.0 notice below says. That notice asks for a full set at 1.16.0 or later before you rely on ≥ 1.16.0 deltas, and the 2026-10-11 set was the first such set. **Two new manifest keys**, on every partition and delta manifest (data dictionary §10); `manifest_keys_version` moves **1.0.0 → 1.1.0**, and no key is removed or given a new meaning: - `content_purpose_list` — `{path, sha256, row_count}` of the list the values were read from. `sha256` changes whenever the list does; `row_count` is how many domains it names. - `row_count_by_content_purpose_source` — rows per column 46 value: `curated`, `schema_org` (always `0` in 1.17.0) and `null`; sums to `row_count`, and the run marker sums it across the sixteen partitions. On a delta, and on a full set's run marker, the `curated` count equals the list's `row_count` unless a listed domain did not ship as a curated row in that artifact — a takedown withholds it, for example. A partition manifest counts only its own partition's tagged rows. The run marker also carries the set's list as `content_purpose_list` (`{sha256, row_count}`), and a set whose partitions were tagged from two different lists gets no marker — it is not complete until it is re-published from one. --- ## 2026-10-30 — deltas re-read crawls stamped in the 30 minutes before their window, and documentation corrections No column or schema-version change. Which rows a delta carries changes slightly, and the data dictionary published beside this entry [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) corrects three passages. **From `delta_2026-10-30_100000Z`, each delta also re-reads crawls stamped in the 30 minutes before its window.** A crawl is stamped when its page is fetched and saved a little later, so a crawl stamped just before a 10:00 UTC cutoff could be saved after that delta's read and still fall outside the next delta's window, so it reached no delta and only the full set carried it. The look-back delivers any crawl saved within 30 minutes of its stamp. Its cost is repetition: a delta can repeat a row the previous delta carried (its `status_observed_at` falls in the 30 minutes before the window), which an upsert by `domain` absorbs. **Corrected in the data dictionary published beside this entry:** - §1.1: after loading a full set, resume with the delta whose window contains the whole of the set's export — for the weekly set, normally the one dated the day before it — and then every later delta; skipping it loses rows. "Nothing is lost" is withdrawn: a crawl saved after a delta's read could reach no delta, a row can reach you one delta after its crawl, and a full set reconciles anything a delta did not carry. - §2: `domain` is normally, not always, the registrable domain — some rows are a host below one or a bare public suffix. Look a name up as written, then fall back to its registrable domain. - §10: deltas are not partitioned. Replace all sixteen partitions of a set together, or route each row to its partition with the published recipe, and re-apply that partition's delta rows after reloading it. --- ## 2026-10-21 — documentation corrections (no data change) Nothing published changes with this entry: no column, value or row moves. It corrects documentation that a review of the 2026-10-18 set found out of step with the data, and it carries one notice that is late. **Late notice: the bulk categorization lanes apply the vendor rule, and have since 2026-09-02.** The data dictionary said they did not yet and that this changelog would say when they did; it did not say so until now. A row the rule rewrote carries `model_name = pipeline/vendor_rule`, `tier1 = Technology & Computing` with `tier2`–`tier4` NULL, and NULL `calibrated_*`. The rule itself, and the trade it makes, are in data dictionary §5 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) and in the 2026-09-02 entry on the `pipeline/adult_filter` and `pipeline/vendor_rule` families, below. **Corrected in the data dictionary published beside this entry:** - §1.1: how long the weekly set lately takes, when it is complete, when a daily delta lands, and the delta rows with no timestamp inside the window: a small share of every delta, whose `classified_at` is NULL, not old. - §3: a known issue — some live sites ship as `parked` or `inactive`, so neither lane is a cull key on its own; `parked`, not `blocked`, is the largest non-active lane; how to filter the labels `blocked` rows carry; curated infrastructure zones carry an asserted `active`. - §4: no crawl cadence is promised. - §5: what `calibrated_confidence` estimates — the probability that a strict blind judge assigns the same `tier1`, which on average understates how often a label is acceptable; the vendor rule in the bulk lanes; the gambling filter's band, which reads `LOW` on every row from 1.14.0; which rows the language-NULL guard spares; that the v4 API returns no `confidence_tier`; and how to read a name-only row's `model_version`. - §8: the lane mix differs from one partition to the next. - §10: rank buckets `"0"`, `"1"` and `"3"` are emitted and `"2"` is reserved, and `coverage_by_rank_bucket` counts presence; `parked_stub_suppression` does not trend to zero; the run-level language map usually names 29 subtags. - §13: the brand-safety example no longer reads as exact. **Corrected in this changelog:** in the 2026-09-11 entry below, how the delta rows with no timestamp inside the window read, and, under the 2026-09-13 correction of its `blocked`-lane item, a dated note that 30-day eligibility for a re-attempt is not a schedule. --- ## 2026-10-07 — the 1.16.0 notice: teacher-rescored rows never publish HIGH (first artifacts: `delta_2026-10-09_100000Z`, published Saturday 2026-10-10, and the Sunday 2026-10-11 set) Nothing published changes with this entry. It is the **customer notice** for feed 1.16.0, published before the deploy it describes so that a loader can adjust first. `feed_schema_version` **1.15.0 → 1.16.0**, a MINOR bump: no column is added, removed or moved, and the 44 columns keep their order and their types. What moves is the published **value** of one column — `confidence_tier` (column 8) — on one population. **Unlike 1.15.0, this re-rates rows you already have.** The 1.15.0 notice below describes a new lane arriving and says no other row re-bands; that was true of 1.15.0 and is not true of this release. Rows published in every set since 1.14.0 change band here. That is why this is its own release with its own notice rather than an amendment to that one. **When.** The 1.16.0 exporter deploys **Friday 2026-10-09 at 13:00 UTC**. The first artifact that reports `feed_schema_version` 1.16.0 is the daily delta published **Saturday 2026-10-10**, in folder `delta_2026-10-09_100000Z`: a delta folder is named by the start of its window, so this one carries the deploy's date but is cut after the deploy. Every delta after it reports 1.16.0 or later. The re-rating itself does **not** arrive with them. The band is derived at export rather than tracked, so a row whose band moves is not a changed row and does not cross a delta cutoff: no delta carries the re-rating as a whole, and a capped row reaches a delta only when the row changes for some other reason, such as a re-crawl or a re-label. The re-rating reaches you with the **Sunday 2026-10-11** full set, and only by ingesting that set. Nothing published before 2026-10-09 carries any of this. **Ingest the full 2026-10-11 snapshot set before relying on 1.16.0 deltas.** This is the same transition rule the 1.14.0 notice states, for the same reason and with no exception: the first ≥ 1.16.0 artifact you ingest must be the full sixteen-partition set, or your table will carry two bands for one row set with nothing marking which is which. `manifest.json` carries `confidence_band_policy` so you can tell them apart — see below. In practice: skip `delta_2026-10-09_100000Z`, the one delta that reports 1.16.0 before that set, and do not apply it after the set either, because it is older than the set; once the set is loaded, resume with `delta_2026-10-10_100000Z`. ### Which rows re-band Rows whose `model_version` (column 11) carries one of three **provenance suffixes** — the second `+`-delimited field of that column, `split_part(model_version, '+', 2)` — and whose `model_name` (column 10) is an ensemble writer (`ensemble-*`, `multitask-ensemble-*`): | provenance suffix | what wrote it | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `teacher_bf1` | the 2026-07/08 teacher pass that **re-scored** `Business and Finance` rows: a second model's verdict written over the classifier's own call. | | `nocontent_bf1` | rows the same pass marked as having **no judgeable content** — it looked, there was nothing to read, and the classifier's original score was left standing. | | `teacher_purpose_v2` | a purpose-teacher lane that has **not been written yet**. It is named here ahead of its first row so that the cap is in place from that row rather than one release later. | On such a row: - a stored `HIGH` publishes **`MEDIUM`**; - and publishes **`LOW`** where `calibrated_confidence_tier` (column 25) is `DOUBTFUL`; - **a stored `MEDIUM` also publishes `LOW` where that same column is `DOUBTFUL`** — a second, smaller population that leaves `MEDIUM` as well as the one that leaves `HIGH`. If you key on `confidence_tier IN ('HIGH','MEDIUM')` rather than on `'HIGH'` alone, these rows leave your selection too; - a stored `LOW` is **unchanged**, and so is a stored `MEDIUM` at any other calibrated tier — the rule is a cap: it moves a band down or leaves it alone, and it **never promotes** anything; - `PARKED` and a NULL band pass through untouched, as under every policy. **No other row and no other column moves.** A row outside the three suffixes — including the `ml_en` / `ml_non_en` route stamps and the bare base version, which record which routed model produced the label rather than a pass over it — bands exactly as it does today. The value domain of column 8 is unchanged: `HIGH` / `MEDIUM` / `LOW`, plus `PARKED`. `confidence`, the `calibrated_*` block, `tier1`–`tier4`, `iab_id`, `model_name`, `model_version` and `policy_group` carry exactly what they would have carried under the old policy, on every row including these. **No `Business and Finance` row moves.** The teacher pass writes an **ensemble** `model_name`, so a teacher row that kept `tier1 = 'Business and Finance'` has been capped by 1.14.0's ensemble rule since that release. What 1.16.0 moves is the capped-suffix population whose `tier1` is **not** `Business and Finance`. **This is not a loss of rows, labels or groups.** Every row that re-bands still ships, with the same `tier1`–`tier4`, the same `iab_id`, the same `policy_group` and the same `model_name`. If your block list or your routing is keyed on `confidence_tier = 'HIGH'`, it will stop seeing them — and if it is keyed on `confidence_tier IN ('HIGH','MEDIUM')` it will still stop seeing the smaller `MEDIUM`-at-`DOUBTFUL` population, which lands on `LOW`. That is what this notice is for. ### Why A blind re-read on **2026-09-12** put a sample of these rows in front of a judge that reads the live page and compared its verdict with the stored label. This family's labels agreed with a fresh reading of the page **far less often than the threshold we had pre-registered for keeping them at our top confidence word** — so publishing them as that word was not defensible. They are scheduled for re-vetting; until that has happened they publish at `MEDIUM` at most, which is the honest reading of what a re-scoring pass over a label actually establishes. The three suffixes travel together because they are one lane: a re-scoring pass, its "nothing to read" marker, and the purpose pass that follows it. ### How to leave it out, and how to rebuild the band you have To exclude the capped population entirely, on any version of the feed: ```sql WHERE COALESCE(split_part(model_version, '+', 2), '') NOT IN ('teacher_bf1', 'nocontent_bf1', 'teacher_purpose_v2') ``` The `COALESCE` is load-bearing, not defensive. Rows that ship a crawl status but no category — the status-only lane — publish `model_version` as NULL, and in SQL `NULL NOT IN (...)` is NULL, not true. Without it those rows fail the predicate and drop out of a query whose job is to exclude the capped population and keep everything else. To rebuild the band as it was before any of the policies — the raw `confidence` cut — `confidence` (column 7) is untouched on every row, and the policy-v1 reconstruction recipe in the 2026-09-10 1.14.0 notice below still applies unchanged, including its rounding rule. On every row the feed already carried before 1.15.0, that recipe returns the band the feed published through 1.13.x, whatever policy the artifact you are reading was produced under. A name-only row (`model_name` beginning `llm-name/`, the 1.15.0 lane) is the exception: it is stored at `MEDIUM` whatever its `confidence`, so there the recipe can return `HIGH` where policy `v1` returns `MEDIUM`. `confidence_band_policy` in `manifest.json` reads **`v4`** from the first 1.16.0 artifact on (`v1` or an absent key is the raw pass-through, `v2` from 1.14.0, `v3` from 1.15.0). Pin it beside `confidence_calibration_map_version`, as before: the `DOUBTFUL` cell above is decided by the calibration map, so a refit moves which of these rows land on `LOW` rather than `MEDIUM`. **The v4 API returns no `confidence_tier`,** so no v4 answer changes with this release. On the legacy `/categorize` route, an answer served from the stored corpus the feed is exported from (`routed_to: "hetzner-cache"`) carries the `v4` band from the same deploy, so until you load the 2026-10-11 set it can differ from the band in your copy of the feed. --- ## 2026-10-02 — a name-only label no longer changes a row's `domain_status` (first artifacts: `delta_2026-10-02_100000Z` and the 2026-10-04 set; no schema or version change) No column, manifest key or `feed_schema_version` changes. The 1.15.0 notice (the 2026-09-14 entry below) and data dictionary §3 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) say a name-only label leaves `domain_status` as the crawl decided it. The 2026-10-01 entry describes a small set of rows for which the deltas published before this change, starting with `delta_2026-09-30_100000Z`, did not keep that promise: the exporter counted the name-only label as a category when it chose the lane. From the artifacts named above it no longer does: - **Rows that would ship as `active` ship as `uncategorizable`**, still with their name-only label. A name-only label no longer puts a row in `active`, so these rows also leave the manifest's `content_accuracy_denominator_row_count`. - **Rows that would ship as `parked` ship as `uncategorizable`.** The last page captured for each was a parking provider's redirect page, so their category columns stay NULL, as before. Only rows whose sole label is a name-only one and whose latest crawl succeeded change lane; no other row changes lane and no category changes. Deltas carry these rows again only when their `classified_at` or `crawl_attempted_at` crosses a delta's cutoff, as a new label or a new crawl attempt does; this change moves neither, so a consumer that applies only deltas keeps the earlier lane until then. To pick up every corrected lane, replace from the 2026-10-04 set, the first full set carrying name-only rows. --- ## 2026-10-01 — the name-only lane added domains the 2026-09-13 set did not carry, rather than labelling existing rows (wording only, no schema or version change) No column, manifest key or `feed_schema_version` changes. This entry corrects the 1.15.0 notice (the 2026-09-14 entry below) on four points: how many rows the lane labels, which rows those are, the lane a labelled row ships in, and when a page-derived label replaces a name-only one. The rows were written on 2026-09-30 as announced: - **The load wrote fewer labels than the notice expected.** A name-only label is written only where we hold no category row for a domain, labelled or not, and some of the domains on which the two model families agreed had one when the load ran: some were `blocked` or `uncategorizable` rows of the 2026-09-13 set without a category, the rows the notice described, which the lane cannot label (next point), and the others were not in that set but had a category row by the time of the load. - **None of the domains the lane wrote was in the 2026-09-13 set.** The notice said the lane labels today's `blocked` and `uncategorizable` rows. It cannot: the lane writes only where we hold no category row for a domain, and the feed carries such a domain only when its crawl finds it parked or a placeholder, or it is on our curated list. The first artifacts carry them. - **Most of them ship with their name-only label**, nearly all as `blocked`. The rest ship with NULL category columns, as data dictionary §3 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) describes. Nearly all of them are in the feed only because a name-only row now exists for them. Some are domains the crawl found `dead`, `inactive` or `parked`, where a name-only label is withheld like any other; for others the last page we captured was a parking provider's redirect page, which withholds the category on every lane (`parked_stub_suppression` in the manifest, dictionary §10). The filters under _How to leave it out_ in the notice remove every name-only label but not these rows, which look like any other row without a category. - **When we checked on 2026-09-30, a small number of them would ship in the lane a label read from the page would give them, although the notice said the label leaves the lane the crawl decided.** Their crawl reached the site but produced no label from the page, which on its own makes a domain `uncategorizable`: some would ship as `active` with their name-only label, and others as `parked` with NULL category columns, because the last page we captured from them was a parking provider's redirect page. To tell a name-only label from one read from the page, test `policy_confidence_basis` or `model_name`, not `domain_status`. Data dictionary §3 now notes this and points here. - **Some are already being replaced.** When a later crawl reads a domain's page and our classifier labels it, that label replaces the name-only one. The first replacements came within hours of the load. The notice and data dictionary §5 said this happens on the domain's next successful crawl; both now say it this way. The first artifact carrying these rows is the delta in folder `delta_2026-09-30_100000Z`, the 10:00 UTC run on 2026-10-01 (folders are named by the start of their window); its manifest counts the labelled ones under `name_only` in `row_count_by_writer_family_x_tier`. The 2026-09-14 entry now carries a dated amendment pointing here, and its description of the rows the lane labels and its sentence on replacement are corrected in place; nothing else in it changes. --- ## 2026-09-28 — domains whose name no longer resolves now ship as `dead`, not `blocked` (no schema or version change) No column, manifest key or `feed_schema_version` changes. What changes is the lane a domain ships in once our crawler learns that its name no longer resolves, and with the lane, the category. `dead` rows carry every content column NULL by design (data dictionary §3 and §9 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary)), so a row that would have stayed labelled in `blocked` now ships without a current label. Where we keep history for its earlier label, that label appears in the `last_known_*` columns (§8). - **Before:** when a crawl got a definitive DNS answer that the name does not exist (NXDOMAIN) or has no address, we recorded a failed crawl, and the domain shipped in `blocked`. That did not match the data dictionary's meaning of `dead`, which then read "DNS failure / NXDOMAIN / `404`". - **Now:** the crawler records that answer as its own status, and the domain ships as `dead`. DNS answers that are not definitive, such as a timeout or a server failure, still ship as `blocked`. Data dictionary §3 now says so for both lanes. - **When:** in crawls from 2026-09-23. It applies to a domain the next time we crawl it, so a `blocked` row whose `status_observed_at` is before 2026-09-23 may still be a name that no longer resolves. Daily deltas carry the change from `delta_2026-09-22_100000Z`, published 2026-09-23, and the 2026-09-27 set is the first weekly set to carry it. - **What you will see:** the `dead` lane grew sharply in the 2026-09-27 set, and `blocked` grew more slowly than in earlier weeks. In that set, each partition our nightly crawl visited after the change added far more `dead` rows than the partitions it visited just before. Most rows that move from `active` or `parked` straight to `dead` are this change; a `404` could already do that. - **Which domains this reaches.** A domain whose name no longer resolves ships as `dead` only if we hold a category row for it, labelled or not (data dictionary §10), and a `curated/*` row stays `active`, because our curated list asserts that lane rather than a crawl observing it (§3). A domain without a category row does not ship as `dead`, as before, so the manifest's `inventory_dead_status_by_rank_bucket` grows much faster than the `dead` row count. Those keys also count as never-shippable, which is the main reason `coverage_shippable_by_rank_bucket` rose in the 2026-09-27 set. --- ## 2026-09-27 — how deep a path goes, and what the 2026-08-22 audit adjudicated (wording only, no schema or version change) No column change, no manifest change, no `feed_schema_version` change and no value in the feed moves. Data dictionary §5 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) ("What it estimates — and what it deliberately does not") made two statements that were wrong, both now corrected: - **The trained classifier does not cut a path where the page evidence runs out.** The dictionary said labels are emitted "to the depth the evidence supports". Below `tier1` the classifier keeps the most likely child at each tier whenever that child holds more than a small fixed share among its siblings, so nearly every path it writes runs on until the taxonomy has no deeper category; how deep a path goes is not a measure of how much the page supported. The guidance is unchanged: key policy on `tier1` (or `tier1` + `tier2`) and treat `tier3` / `tier4` as descriptive detail. - **No person adjudicated the 2026-08-22 audit.** The dictionary said its two judge passes had "human adjudication of every disagreement". A language model re-ruled most of the rows the first pass judged wrong, settled the rows where the reviewed first pass and the second judge disagreed over correct versus wrong, and marked one more row correct that neither judge had called correct. No verdict was adjudicated by a person, and the judges' other disagreements were not adjudicated, among them disagreements on depth alone and on whether the page could be judged at all. --- ## 2026-09-15 — a second, convention-named copy of every set and delta under `datafeeds/` (withdrawn the same day) **Withdrawn, 2026-09-15.** A second copy of every set and delta's data files and manifests, under a `datafeeds/` sub-folder and named the way the WhoisXML datafeeds server names files, was announced for the exporter deploy of Thursday 2026-09-17 and withdrawn the same day. The WhoisXML datafeeds team asked for a different shape, and that copy is now built and delivered outside this bucket, so no set and no delta under `exports/` carries a `datafeeds/` sub-folder, and there is nothing to exclude when you sync. Only the 2026-09-13 set has one, filled by hand before the change of plan; it stays (nothing under `exports/` is ever deleted) and you can ignore it. Nothing about the documented objects changed: the sixteen partition folders, `run_manifest.json`, the delta folders and every name in the data dictionary stay exactly as they are. --- ## 2026-09-15 — 1.14.0 moves a week earlier: first artifact Thursday 2026-09-17, first Sunday set 2026-09-20 **The date the entries below first announced — the Sunday 2026-09-27 snapshot set — is superseded.** Nothing about the schema, the columns or the corrections changes; only when they reach you. The schema and the corrections it carries were ready ahead of the announced date, and moving the exporter change forward keeps it clear of a separate release the following week. **Amended 2026-09-17: no delta publishes on Friday; the first 1.14.0 delta is Saturday's.** The 2026-09-17 set started right after Thursday's deploy and, like every full set, takes about a day to run, so it still holds the export lock when Friday's 10:00 UTC delta is due. That run is skipped, exactly as Sunday's is every week while the weekly set runs. Saturday's 10:00 UTC delta covers Thursday 10:00 UTC to Saturday 10:00 UTC, so nothing is lost and the chain has no gap. - **The exporter deploys Thursday 2026-09-17**, after that day's 10:00 UTC daily delta. - **The first 1.14.0 artifact is an off-cadence full set, `exports/snapshot_2026-09-17/`.** The run starts immediately after the deploy and the set is complete when its `run_manifest.json` appears — expect that on Friday 2026-09-18 UTC, about a day after the run starts. It is a full per-partition replace exactly like a weekly set: the same sixteen partitions, the same partition and run manifests, nothing special in its shape. Treat it as a weekly set that lands on a Thursday. - **The first 1.14.0 delta is Saturday 2026-09-19's 10:00 UTC run**, in the folder `delta_2026-09-17_100000Z` — named, as always, by the start of its window — covering Thursday 10:00 UTC to Saturday 10:00 UTC. Thursday's delta ran before the deploy, on 1.13.2, so the chain has no gap. - **Sunday 2026-09-20 is the first regular weekly 1.14.0 set.** The Sunday 2026-09-27 set is now an ordinary weekly set with nothing special about it. - **Nothing published before 2026-09-17 carries 1.14.0.** The 2026-09-13 set and every delta through the one that ran at 10:00 UTC on 2026-09-17 (folder `delta_2026-09-16_100000Z`) report `feed_schema_version` 1.13.2. **Ingest a full 1.14.0 snapshot before relying on 1.14.0 deltas.** The rule the 2026-09-10 entry states is unchanged and now points at the 2026-09-17 set: three of the 1.14.0 changes are re-derivations that touch every row, and a delta cannot carry a row that did not move. The dates in the 2026-09-10 entries below have been corrected in place to match. --- ## 2026-09-14 — the 1.15.0 notice: a name-only label lane for the unreadable head (first artifacts: the daily deltas from 2026-09-30 and the Sunday 2026-10-04 set) Nothing published changes with this entry. It is the **customer notice** for feed 1.15.0, dated two weeks ahead so a loader has the whole window to adjust. `feed_schema_version` **1.14.0 → 1.15.0**, a MINOR bump: no column is added, removed or moved; column 42 gains one value, and a small population of domains is added to the feed, most of them with a category. **Amended 2026-09-15, before any 1.15.0 artifact exists: the lane ships the HEAD ONLY.** The torso validation this entry reserved came back below the precision bar we set for shipping it, so rows ranked 100,000–1,000,000 are **not** labelled this way for now. Nothing else in this entry changes: the mechanism, the column values, the dates and the filter are exactly as first published. If the torso ships later it will be its own changelog entry, not a quiet addition to this one. **Amended 2026-10-01: the lane added domains that the 2026-09-13 set did not carry, not labels on existing rows. Most of them ship with a name-only label, and when we checked on 2026-09-30 a small number would ship in the lane a label read from the page would give them, although the `domain_status` row of the table below says the label leaves the lane the crawl decided.** The first delta carrying them is folder `delta_2026-09-30_100000Z`; see the 2026-10-01 entry. The filters under _How to leave it out_ remove every name-only label, but not the rows the lane added that ship without one. Under _What the lane is_ below, the description of the rows it labels and the sentence on replacement are corrected in place; nothing else in this entry changes. **Amended 2026-10-02: the `domain_status` row of the table below holds again from the first artifacts exported after the 2026-10-02 deploy; a consumer that applies only deltas picks up the corrected lanes by replacing from the 2026-10-04 set. See the 2026-10-02 entry.** **When.** The 1.15.0 exporter deploys **Tuesday 2026-09-29 at 13:00 UTC**. The rows described here are written on **Wednesday 2026-09-30**, so the first artifacts carrying them are the **daily deltas from 2026-09-30** and the **Sunday 2026-10-04** full set. Nothing published before 2026-09-29 carries any of this; a delta between the deploy and the load reports 1.15.0 with no name-only rows in it yet. No full-snapshot ingest is required before relying on 1.15.0 deltas: no existing row is re-derived, so every new row moves and every delta carries it. ### What the lane is For domains in the Tranco top 1,000,000 for which we held no category row, mostly sites the crawler cannot read, two independent language-model families label the site from the **domain name alone**, separately. A row ships only where **both agree on the tier-1 category and each puts at least 0.7 on it**; otherwise nothing is written and the lane leaves the domain as it was, which keeps it out of the feed unless its crawl finds it parked or a placeholder, or it is on our curated list. Every row that ships with a name-only label is ranked in the top 100,000 (see the amendments above and the 2026-10-01 entry). The lane is gap-fill only: it never replaces a label that exists, and a page-derived label replaces it once a later crawl reads the page and our classifier labels it. Only the top 100,000 is labelled this way today, and nothing ranked past 1,000,000 ever will be. ### How a name-only row reads | column | value | | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_name` (10) | `llm-name/consensus_v1` | | `policy_confidence_basis` (42) | **`name_only`** — the new value. The closed set becomes `model_calibrated`, `model_raw`, `filter`, `rule`, `curated`, `carry_over`, `name_only`, `none`. | | `tier1` … `tier4` | tier 1 only; `tier2`–`tier4` NULL | | `confidence` (7) | the **lower** of the two models' self-reported probabilities, so always in `[0.7, 1]` — a scale of its own, not comparable with any other family's (data dictionary §5) | | `confidence_tier` (8) | `MEDIUM` or `LOW`, **never `HIGH`**: band policy `v3` = `v2` plus rule 5 (`name_only_never_high`), a cap on this family only. `confidence_band_policy` in `manifest.json` reads `v3` from the first 1.15.0 artifact on; no other row re-bands, because rule 5 is keyed on this family's `model_name` alone. | | `calibrated_confidence` (24) | NULL — the lane is never calibrated | | `domain_status` (3) | unchanged: the lane the crawl decided. A `blocked` row with a name-only label is still `blocked`. | | `last_known_*` (dictionary §8) | never carries a name-only label: when such a row is later withdrawn, no history block is projected from it | | `row_count_by_writer_family_x_tier` (manifest) | gains the `name_only` family | ### The torso The separate validation this entry reserved for the torso (rank 100,000–1,000,000) ran on 2026-09-15, judged by the same blind judge that scores the rest of the feed, and did not clear the bar we set as the condition for shipping it. Those labels are therefore not shipped, and the first artifacts carry head rows only. We would rather withhold a label than ship one below the bar we set. ### How to leave it out To exclude the name-only lane entirely, on any version of the feed: ```sql WHERE COALESCE(policy_confidence_basis, '') <> 'name_only' ``` or, equivalently, on the writer family: ```sql WHERE COALESCE(model_name, '') NOT LIKE 'llm-name/%' ``` The `COALESCE` is load-bearing, not defensive. Rows that ship a crawl status but no category — the status-only lane — publish `policy_confidence_basis` and `model_name` as NULL, and in SQL `NULL <> 'name_only'` and `NULL NOT LIKE 'llm-name/%'` are NULL, not true. Without it those rows fail the predicate and drop out of a query whose job is to exclude the name-only lane and keep everything else. A block list built on `confidence_tier = 'HIGH'` never sees these rows. The real-time API does not serve name-only labels: it keeps answering those domains live, as today. --- ## 2026-09-13 — a `304 Not Modified` answer is not a blocked site (first artifact: the 2026-09-17 set) No column change and no schema-version change: the same rows carry a different `domain_status`. The correction ships with the 1.14.0 exporter (deploy Thursday 2026-09-17), so the 2026-09-17 snapshot set is the first artifact that carries it. - **What was wrong.** The nightly refresh re-reads a site with a conditional request. When the server answers `304 Not Modified`, the crawler records the status `not_modified`: the content we hold is confirmed current and nothing was fetched. The exporter listed that status among the `blocked` ones, so a healthy, unchanged site was published as `blocked` — against §3 of the data dictionary [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary), which defines `blocked` as a site we could not read. - **What changes.** From the 2026-09-17 set a `not_modified` status lanes like a successful crawl: `active` where the row carries a label, `parked` where the content we hold is the parking-provider stub (the stub override in §10 applies as before), `uncategorizable` where no label exists. Nothing else about the row moves: the label, its calibrated value and `status_observed_at` (which dates the `304` answer) are as they were. The real-time API is not affected: its `domain_status` answer comes from the category store, not from this rule, and the category it returns for these rows does not change. - **What to do.** The rows that move did not change in the inventory, so a delta cannot carry them; the first 1.14.0 set is a full per-partition replace anyway (see the 2026-09-10 entry), and ingesting it brings every such row to its corrected lane. Expect `row_count_by_domain_status` to show `blocked` lower and `active` higher, before the week's own movement. - **The 2026-09-11 entry below first attributed the `blocked` lane's growth over the 2026-08-30 → 2026-09-06 sets to paused re-crawls (corrected there).** The mapping corrected here also contributed to that growth, alongside the re-crawl cadence and real anti-bot answers. --- ## 2026-09-11 — one new manifest key and three clarifications from the data audit of the 2026-09-06 set No column change. The key ships with 1.14.0 (first artifact the 2026-09-17 set); the three clarifications describe artifacts you already have. - **Unicode-form domains are withheld and counted.** The 2026-09-06 set carried rows whose `domain` was an internationalised name in Unicode form (`helaförsäkring.se`), most of them beside the same name in punycode (`xn--helafrskring-mcb3w.se`) — two rows for one registrable domain, and the two disagreed on lane or label in many of those pairs. From 1.14.0 the exporter withholds a non-ASCII `domain` and publishes the count as `row_count_excluded_non_ascii_domain` (partition and run manifests; `0` is a measured zero). The withheld rows include names whose punycode form is not in the feed yet; they return once the inventory ingest normalises them, after which the counter reads `0`. A delta cannot retract a row you already hold, so the correction reaches you with the first 1.14.0 set — a full replace. Data dictionary §2 and §7 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary) now state the form: lowercase ASCII, punycode for internationalised names. - **`status_observed_at` can be null on a handful of rows, and the manifest says where.** A handful of rows of the 2026-09-06 set carry a status with no attempt timestamp. The dictionary's §4 promised every row is dated; it now says every crawler-observed row is dated and points at the `null` bucket of `row_count_by_status_age` (1.14.0), which counts the exceptions per lane. A curated row is dated only where the inventory has a row for that domain. - **Deltas are upserts; do not filter them by timestamp.** A small share of every delta has no timestamp inside the delta's window: `classified_at` is NULL and `status_observed_at` is older (_corrected 2026-10-21: first stated with both timestamps older_). Such a row qualified through a category-side event the feed does not show — an abstain, or a label withdrawn by suppression — and it is a real update: applying only the rows whose timestamps fall inside the window keeps withdrawn labels. Stated in dictionary §1.1. - **The `blocked` lane is growing.** It grew set-wide across the 2026-08-30, 2026-09-02 and 2026-09-06 sets. A blocked row keeps its last label and dates it (§3, §9), and `row_count_by_domain_status` in each manifest is the number to watch. _Corrected 2026-09-13: this item first attributed the growth to paused re-crawls of blocked sites and said the share would keep rising until they resumed. That was wrong. Re-crawls of blocked sites are not paused — a blocked domain becomes eligible for a re-attempt after 30 days. The growth comes from the nightly refresh reaching sites it last observed in May 2026, most of which no longer answer; between the 2026-09-06 and 2026-09-13 sets, nearly every `p06` row that moved from `active` carried a `status_observed_at` from May 2026 before the move (your own copies of those two sets show this). Sites answering `304 Not Modified` were also published as `blocked` (see the entry above). The rest is sites that did not answer the re-fetch, and real anti-bot answers._ _Corrected 2026-10-21: "eligible for a re-attempt after 30 days" is eligibility, not a schedule. No re-try schedule is promised, so a blocked row can keep its verdict for months. `row_count_by_status_age` in each manifest shows how old the verdicts are (dictionary §4)._ --- ## 2026-09-10 — 1.14.0: six new columns, a new policy pack, a derived band, and a smaller labelled population `feed_schema_version` **1.13.2 → 1.14.0**, a MINOR bump. Ships with the first ≥ 1.14.0 export — the **Thursday 2026-09-17** snapshot set (exporter deploy Thursday 2026-09-17, after that day's 10:00 UTC delta); nothing published before that date carries any of this. **Ingest a full snapshot at 1.14.0 before relying on 1.14.0 deltas:** three of the changes below are re-derivations that touch every row, and a delta cannot carry a row that did not move. ### Six appended columns (39–44) Columns 1–38 are unchanged in name, position and type — the append-only contract, same precedent as 1.6.0 and 1.12.0. A reader that selects by name or by position is unaffected by the append itself. | # | Column | What it is | | --- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 39 | `safety_status` | Whether a brand-safety filter reached a decision: `flagged` / `assessed_clean` / `not_assessed`. Never NULL. | | 40 | `safety_flags` | The flagged block-signal themes, as a list. | | 41 | `safety_detail` | A compact JSON object, one entry per filter family. | | 42 | `policy_confidence_basis` | What the row's `confidence` / `calibrated_confidence` IS — `model_calibrated`, `model_raw`, `filter`, `rule`, `curated`, `carry_over`, `none`. | | 43 | `policy_group_set` | `policy_group` plus every group the pack declares it is blocked WITH. | | 44 | `language_primary` | The BCP-47 primary subtag of column 14. `zh-CN`, `ZH_TW` and `zh-Hans-CN` all read `zh`; `eng` stays `eng`, deliberately, because the two spellings come from different detectors. Column 14 stays raw. | The brand-safety axis is documented in data dictionary §13, columns 42–43 in §11 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary). ### `policy_basis` gains the value `safety` — a breaking value addition The feed's policy pack moves from `policy_v1` to **`policy_v2`**. The new pack reads the **safety axis first**, then `site_purpose`, then the primary tier path, and `policy_basis` (column 37) says which answered. Its documented values are therefore `primary`, `site_purpose` and now **`safety`**. `role` is reserved for a later release and is not emitted at 1.14.0. **If you validate column 37 against a strict enum, add `safety` before ingesting.** It is confined to an opt-in block, and it is why the pack change ships via a full snapshot rather than a delta: `policy_pack.version` in `manifest.json` moves with it, and a delta whose pack differs from its predecessor is a chain gap the exporter refuses to write. Because the safety axis is read first, some flagged rows now carry the theme's group where they previously carried their label's — a gambling-flagged row labelled `Sensitive Topics` reads `gambling` rather than `sensitive_other`. Nothing about the label, the confidence or the calibrated block moves with it. ### `policy_group`: `news` is retired, `current_events` replaces it `policy_v2` also renames one group, so this is a breaking value change on **column 36**, not only on column 37. The three current-affairs roots — `Crime`, `Disasters` and `War and Conflicts` — published as `news` under `policy_v1`; from 1.14.0 they publish as **`current_events`**. No other path changes group, and the group count is unchanged at 47. **If you filter or validate `policy_group` against a fixed list or an enum, replace `news` with `current_events` before ingesting the first 1.14.0 snapshot** — from that artifact on, a policy keyed on `news` matches nothing. `policy_group_set` (column 43) carries the new id too, and so does `row_count_by_policy_group` in `manifest.json`. The rename is why `policy_v2` is a new **pack id** rather than a mapping bump: a group id never changes meaning inside a pack, so the old id was retired with its pack. The old id was misleading — IAB v3.1 has no news-publisher node, so `news` never contained newspapers or news portals, and a policy that allowed or blocked it allowed or blocked no news site. `current_events` says what the group actually is; there is still no group that contains news publishers. Full definition, the per-group table and the machine-readable diff: [`website-categorization-policy-pack-v2.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v2.md). ### `confidence_tier` is derived (band policy v2) Column 8 is derived at export from the writer (`model_name`), the label (`tier1`) and the calibrated tier (`calibrated_confidence_tier`), instead of being the stored `0.9` / `0.7` cut passed through. The value domain is unchanged — `HIGH` / `MEDIUM` / `LOW`, plus `PARKED`. `manifest.json` carries `confidence_band_policy: "v2"`, and a manifest that does not carry the key at all is `v1`. The notice below carries the per-family detail and the recipe for rebuilding the old band from columns you already receive; read it before the first 1.14.0 artifact lands. ### Fewer rows carry a category: the language-NULL guard is now armed This is the change that makes 1.14.0 a MINOR rather than a PATCH. Rows written by the ensemble family that carry a category with **no page language recorded**, behind a captured content hash, now ship **status-only**: the category block is suppressed and no `last_known_*` projection is made. Those labels were produced from content we cannot show was in a language the model was fitted for, and a judged read put their agreement low enough that suppression is the correct answer rather than a re-score. `manifest.json` carries a `language_null_suppression` block from 1.14.0 on, counting the rows this export changed and the three exempt populations (block-critical groups, the popularity head, and rows whose group could not be resolved) — those are **not** suppressed. Rule lanes are untouched. Expect the labelled population of a partition to fall; **the manifest block of your own delivery is the authority on how much.** ### New manifest keys Every export now publishes the distributions a consumer previously had to scan for, at partition, delta AND run level: `row_count_by_confidence_tier`, `row_count_by_writer_family_x_tier`, `row_count_by_policy_group` (every pack group present, `0` where empty), `row_count_policy_group_uncalibrated`, `row_count_by_policy_group_by_writer_family`, `row_count_by_safety_status`, `row_count_by_safety_flag` and `row_count_by_language_primary`. The four coverage counters that existed per partition (`row_count_with_policy_group`, `row_count_calibrated`, `row_count_by_site_purpose_status`, `row_count_with_last_known_category`) are now summed in the run-level `run_manifest.json` too, so a snapshot's totals no longer require adding up sixteen files. Deltas additionally carry `row_count_tier1_changed`, `top_tier1_transitions` and `tier1_transitions_window` — **a planned re-score announces itself there.** A delta's `row_count` says how many rows moved; those keys say whether they moved between categories, and which moves were the largest. `manifest_keys_version` is new and names the KEY SET's own version, independent of `feed_schema_version`. Pin it if what you parse is the manifest rather than the columns. Every key is described in data dictionary §10 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary). --- ## 2026-09-10 — the 1.14.0 notice: what `confidence_tier` will say, and how to rebuild the old band Nothing published changes with this entry. It is the **customer notice** for the release above: which band each writer family can reach, which rows move, and the recipe for rebuilding the band you have today. Read it now, not on the day the snapshot lands. **Why this entry is dated where it is.** The first artifact carrying any of this is the **Thursday 2026-09-17** snapshot set. The exporter that produces it deploys on **Thursday 2026-09-17**, after that day's 10:00 UTC delta, and everything described here is frozen a week before that. This notice is therefore dated **before** the first 1.14.0 artifact exists — deliberately, because the band is derived rather than tracked and there is no changed-row marker that would tell a loader to look for it. If you are reading this after 2026-09-17, the change has already shipped. **Amended 2026-09-11, before any 1.14.0 artifact exists.** The three brand-safety filter families are written under **five** `model_name` values: `pipeline/st_filter`, `pipeline/gambling_filter`, `pipeline/adult_filter` and the two cascade twins `cascade/st_filter` and `cascade/gambling_filter`, whose `confidence` is the same non-probability as their `pipeline/` spelling. The filter rule below covers all five. A `human-feedback` row — a reviewer's accepted verdict, written at a fixed `confidence` of `1.0` — reads `policy_confidence_basis = curated`, not `model_raw`, and keeps the ordinary `0.9` / `0.7` cut, because no rule of the policy is keyed on that writer. ### What changes in column 8 `confidence_tier` stops being the stored `0.9` / `0.7` cut passed through verbatim and becomes a band **derived at export** from the writer (`model_name`), the label (`tier1`) and the calibrated tier (`calibrated_confidence_tier`). The per-family table — what band each `model_name` family can reach — is in data dictionary §5 [Website Categorization bulk feed data dictionary](https://website-categorization.whoisxmlapi.com/database/specifications/data-dictionary). In one sentence: rows written by the three brand-safety filters band `LOW`; ensemble rows labelled `Business and Finance` never band `HIGH`; and an ensemble row the cut put in `LOW` bands `MEDIUM` when its calibrated tier is `TOSSUP` or `PROBABLE`. **The value domain of column 8 is unchanged** — `HIGH` / `MEDIUM` / `LOW`, plus `PARKED`. No word is added, none is removed, and a `PARKED` or NULL band passes through untouched. `confidence`, the `calibrated_*` block, `tier1`–`tier4`, `iab_id`, `model_name`, `policy_group` and every other column carry exactly what they would have carried under the old cut. ### Which rows move, and in which direction All three rules move rows, and the one with the clearest consequence for a block list — a brand-safety filter's `confidence` is a rule score, so those rows can no longer band `HIGH` — is the **smallest** of the three by row count, not the largest. | Rule | Direction | | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | | The three brand-safety filters band `LOW` only | Rows leave `HIGH` and `MEDIUM`. They project to the `sensitive_other`, `gambling`, `harmful_content` and `adult` policy groups. | | Ensemble `Business and Finance` never bands `HIGH` | Rows leave `HIGH`; those whose `calibrated_confidence_tier` is `DOUBTFUL` also leave `MEDIUM` for `LOW`. This rule moves more rows than the first. | | An ensemble `LOW` row whose calibrated tier is `TOSSUP` or `PROBABLE` bands `MEDIUM` | Rows move **up**. This rule moves the most of the three, and nothing is demoted by it. | **This is not a loss of rows, labels or groups.** Every row still ships, under all three rules, with the same `tier1`–`tier4`, the same `iab_id`, the same `policy_group` and the same `model_name`. What changes is the one column that never carried the information a blocker needs: on the first rule's rows the old `HIGH` meant "a keyword rule fired hard", never "this label is 90% likely to be right". If your block list is keyed on `confidence_tier`, it will lose those rows — which is why the data dictionary says, in as many words, to key on `policy_group` first and to read its ranking of the other keys before relying on them. **Read your own delivery's manifest counts**; the per-partition figures behind this entry are not published here, and one partition's composition is not the feed's. ### Rebuilding the policy-v1 band client-side `confidence` (column 7) is untouched, so the band you have today is reconstructible on every row. Evaluate it exactly as the export does, or you will disagree with us on the boundary: ```sql -- The policy-v1 band, on any row whose confidence_tier is not PARKED and not NULL. CASE WHEN round(confidence::numeric, 4) >= 0.9 THEN 'HIGH' WHEN round(confidence::numeric, 4) >= 0.7 THEN 'MEDIUM' ELSE 'LOW' END ``` **Round to four decimals first — both cut points need it.** The pipeline produces `confidence` rounded to four decimals and the Parquet column is a 32-bit float, so the decoded value is the nearest float32 rather than the decimal you were shown. Both boundary values fall _below_ their cut: `0.9` decodes as `0.899999976…` and `0.7` as `0.699999988…`, so an unrounded `>= 0.9` silently bands every exact-`0.9` row `MEDIUM` and an unrounded `>= 0.7` bands every exact-`0.7` row `LOW`. The `csv.gz` cell is already written at four decimals and needs no rounding; the Parquet column does. Two rows this recipe deliberately does not touch: a `PARKED` band and a NULL band are not cut points and are passed through under both policies. ### The transition rule: the first 1.14.0 set is a full replace The band is derived, not tracked: a row whose band moves under the new policy is not a changed row and does not cross a delta cutoff. **The first ≥ 1.14.0 artifact you ingest must be a snapshot** — the full sixteen-partition set — or your table will carry two bands for one row set with nothing marking which is which. `manifest.json` carries `confidence_band_policy` so you can tell them apart: `v1` (or the key's absence) is the pass-through, `v2` is the derived band. The same release also makes the language-NULL ensemble class ship status-only, which moves the published row population. One full per-partition replace satisfies both. ### The brand-safety columns ride the same release Columns 39–41 — `safety_status`, `safety_flags`, `safety_detail` — ship on the first 1.14.0 artifact. One property of them is worth knowing, because it decides what they are evidence of. A domain's per-filter verdicts are stored from the 1.14.0 release onwards. Rows whose last write predates it are **derived from the row's existing `model_name` and `tier2`** and carry a `legacy` version marker in `safety_detail`. That derivation is deliberate — without it every pre-existing filter row would report `not_assessed` and a blocker keyed on the new columns would under-block on day one — but on those rows it is a re-statement of what column 10 already told you, not new evidence. Values written from a real verdict are distinguishable by their version marker. --- ## 2026-09-09 — ahead of 1.14.0: the transition rule goes on record Nothing published changes with this entry. It puts the **transition rule** for the next MINOR release, 1.14.0, on record before the artifact that needs it exists: column 8 becomes a derived band, the first ≥ 1.14.0 artifact you ingest must be a full sixteen-partition snapshot, and `manifest.json` gains `confidence_band_policy` so the two policies are distinguishable (`"v1"`, or the key's absence, is the pass-through every artifact to date has carried). The key can appear on a 1.13.x manifest ahead of the bump, always reading `"v1"`; under the extensible-manifest rule that carries no change to the artifact. Blockers should key on `policy_group` and `model_name`, never on the band. The full notice is the entry above. --- ## 2026-09-06 — 1.13.2: curated rows on every delta (no column change) `feed_schema_version` 1.13.1 → **1.13.2**, a PATCH bump. Ships with the first ≥ 1.13.2 export — and, unlike 1.13.1, that includes the first **daily delta** after the exporter deploy, not only the first Sunday export after it. Every delta now carries the **whole hand-vetted curated list** — the `curated/*` rows the real-time API answers from — and not only the exclusion that already kept a competing model-produced row out. Until now a curated row shipped in snapshots alone, so a consumer building its table from deltas kept the old model guess for a curated domain until the next weekly snapshot. Those are the most-visible domains in the feed, so the divergence sat exactly where it was least acceptable. Snapshot artifacts are unchanged. **A daily delta exported between the deploy and the Sunday snapshot carries 1.13.2 too** — both kinds are stamped from the same constant — and here that delta is already complete for this change: the curated rows are not keyed on a cutoff, so unlike 1.13.1's masked rows they **do** cross the delta boundary and arrive with the first ≥ 1.13.2 delta. No snapshot is needed to receive them. The snapshot-ingest obligation is unchanged for every other reason: removals and the status-only baseline still travel only in a full per-partition replace. What else changes for you: - **The whole list rides every delta, not a changed subset.** The curated list is a file, not a table, so there is no cutoff to narrow it by. The same rows arrive in every delta; under the upsert-by-domain a delta already requires, re-applying them is a no-op. - **Apply curated rows by `domain`, never by `classified_at` ordering.** A curated row is dated when the entry was **curated**, so it is routinely _older_ than the model-produced row it replaces. A loader that keeps the newest `classified_at` per domain will discard the curated answer — match on `domain` and take the row from the most recent artifact. This was already true of snapshots; deltas make it matter daily. - **`curated_lane` in a delta's `manifest.json` now reads `true`** (it read `false` before), and `row_count_curated` reports how many curated rows that delta carried rather than always `0`. Both keys already existed; neither is given a new meaning. - **A delta's `row_count` is no longer 0 on a quiet night** — it is at least the curated count. To ask "did anything change in the source?", read `row_count - row_count_curated`. --- ## 2026-09-05 — 1.13.1: parked-flagged brand-safety filter rows ship status-only (no column change) `feed_schema_version` 1.13.0 → **1.13.1**, the feed's first PATCH bump: the published row population changes, no column is added, removed, renamed or retyped, and no manifest key is removed. Ships with the first ≥ 1.13.1 snapshot set — the first Sunday export after the exporter deploy. Parked-flagged brand-safety filter labels — `pipeline/st_filter`, `pipeline/gambling_filter`, `pipeline/adult_filter` and `pipeline/vendor_rule` — now ship status-only, matching the treatment parked-branded **ensemble-family** labels have had since 1.11.0. Parked-flagged rows outside both those families (`cascade/st_filter`, for one) are not touched by this guard at all. Affected rows publish `site_purpose_status = undefined` with NULL category columns and an empty `last_known_*` block; the domain's status lane is unchanged. Until now such a row asserted a current category and "no current content" at the same time. Those rows sit on the **category-keeping** lanes — the classifier's parked flag disagreeing with the crawl-observed lane is exactly why they still carried a category — so filtering `domain_status = 'parked'` will not find them. **Read your own delivery's manifest for the count on each partition.** A second population, already on the parked / dead / inactive lanes, loses its `last_known_*` historical label for the same reason. **Some parked domains keep an adult or gambling verdict on purpose.** A row whose label projects to a block-critical policy group — `adult`, `drugs`, `gambling`, `gambling_games`, `harmful_content`, `hate`, `piracy`, `weapons` — is deliberately **retained** rather than suppressed, and ships exactly as it does today, until an in-group accuracy read clears the hold. What that looks like depends on the lane: on the category-keeping lanes (`active`, `blocked`, `uncategorizable`) the row keeps its category and `confidence_tier = 'PARKED'`; on the `parked` / `dead` / `inactive` lanes, where the lane rules already blank the current category, the hold instead keeps the row's `last_known_*` history intact. The hold does not override the older guards: a row one of the sibling suppression blocks already owns — parked-stub, no-evidence or boilerplate-template — is suppressed by that guard as before, block-critical group or not. Removing a brand-safety block is the change that can hurt you, so it is not made on an unmeasured population. The `demoted_label_suppression` manifest block gains `filter_model_names` (the four families above), `block_critical_held` (rows the hold retained — counted there, never in `row_count_overridden`), `block_critical_groups` (the groups actually held in that run), `block_critical_group_policy` (the eight-group rule), `unresolved_group_held` (rows held because their policy group could not be resolved) and `policy_group_delta` (per-group `before` / `after` counts). No 1.13.0 manifest key was removed. Same transition rule as the 1.8.0 / 1.10.0 / 1.11.0 guards: consume the first ≥ 1.13.1 set as a **full per-partition replace**. Rows this build newly masks do not cross the delta cutoffs, so a delta applied on a 1.13.0 base keeps categories 1.13.1 withdrew. --- ## 2026-09-04 — eight corrections to the data dictionary (wording only, no schema or version change) No column change, no manifest change, no `feed_schema_version` change and no published value moves. Eight statements in the data dictionary described the pipeline as it was configured at some earlier point rather than as it runs today. Each is now corrected against the running configuration, and each correction is pinned by a test so it cannot drift back: - **The sensitive-content filter fires at `0.50`, not `0.30`.** `0.30` is a code default that has never been the deployed value; the real-time and bulk lanes both run the filter at `0.50`. - **A page whose body text exceeds 1,500 characters is not scored by that filter at all.** It is skipped, not truncated — so the absence of an `*/st_filter` row is not evidence the filter cleared the page. - **The filter has a video-streaming bypass:** a fire scored in `[0.50, 0.65)` on a video-streaming domain is discarded and the page falls through to normal classification. Those rows are not `Sensitive Topics` and do not project to the `sensitive_other` policy group, so both counts understate what the filter fired on. - **Eight `model_name` families ship, on seven scales.** The counts differ because the vendor-rule flip republishes the trained classifier's own probability rather than scoring anything. - **`v1+teacher_bf1` rows carry `calibrated_*` values.** The 2026-08-13 refit (`calib-map-v2`) deliberately fitted that suffix in; the page still described every re-scoring suffix as out of scope. The map's allow-list today is: no suffix, `ml_en`, `ml_non_en`, `nocontent_bf1`, `teacher_bf1`. - **All five calibration ceilings are documented.** Only `en|rest` and `__global__` rows can reach `PROBABLE`; `en|bf` and `nonen|rest` are capped below it and `nonen|bf` well below. Previously only the `nonen|bf` ceiling was described, which read as if it were the only capped segment. - **`domain_status = 'active'` does not guarantee a category.** A live site can ship with every content column NULL — never labelled, or a suppression guard withheld the label. Test `tier1 IS NOT NULL`. - **`confidence_tier = 'PARKED'` is reachable and now observed.** The page said it had occurred zero times; a small number of rows carry it, all written by rule filters disagreeing with the crawl-observed lane. Parse for the token. Two further wording changes in the same pass. The `last_known_*` coverage figure was measured on one partition and scaled into a feed-wide count; the population underneath it has grown, so the scaled figure is withdrawn and that section now tells you to read `row_count_with_last_known_category` and `row_count_by_domain_status` off your own delivery, which is exact and needs no scaling. And a number of cross-references inside the dictionary pointed at section anchors that do not exist; all of them now resolve. One API-side note, recorded here because the names collide: `content_purpose` (the legacy API field; not in the feed) is reserved and null in every current deployment — no shipped model emits it. It is unrelated to the feed's `site_purpose` axis (columns 27–35) and to the `policy` projection; no feed column has ever carried it. --- ## 2026-09-04 — how `tier1..tier4` is defined, stated one way: purpose-first (wording only, no schema or version change) No column change, no manifest change, no `feed_schema_version` change, and no published value moves — this entry exists only because the definition of the stored label was written two different ways and is now written one way. The stored label follows the ratified **single-label convention**, and that convention is **purpose-first**: it names the site's primary purpose — what the site is _for_ — rather than its topic, industry or audience, with a small set of rules settling the recurring cases: a gambling home is `Attractions > Casinos & Gambling` and the harmful subset ships as `Sensitive Topics`; software, SaaS and hosting vendors are `Technology & Computing` whoever they sell to; a multi-brand store is `Shopping` while a brand's own store takes its vertical; consumer banking, BNPL, investing and pensions are `Personal Finance`; a content platform takes the content type it carries, not its delivery mechanism. The data dictionary previously described the same convention as "about-first, plus a small set of purpose conventions", which stated the base principle backwards; the convention itself is unchanged since v1.1 (2026-08-25). Nothing you ingest changes because of this entry: no row is relabelled by it, no threshold moves, and the reserved `site_purpose_*` columns (27–35) stay reserved — eight NULL, `site_purpose_status` never NULL. If a purpose pass is ever run, it is announced here before it ships. --- ## 2026-09-03 — 1.13.0: four optional manifest keys for a stricter coverage denominator (no column change) `feed_schema_version` 1.12.0 → **1.13.0**. No column changes and no change to any existing manifest key — the version covers manifest keys too, so keys alone bump it. Partition and run-level manifests may now also carry `row_count_dead_status_by_rank_bucket`, `inventory_dead_status_by_rank_bucket`, `inventory_never_shippable_by_rank_bucket` and `coverage_shippable_by_rank_bucket`. The existing `coverage_by_rank_bucket` counts every inventory key in its denominator, including keys whose crawl status is in our dead set (in practice `not_found`, 404 / NXDOMAIN) and that carry no category row — keys that cannot produce a feed row until a re-crawl succeeds. The new `coverage_shippable_by_rank_bucket` takes those out. Both are published; nothing you read today moves. The keys appear from the first artifact published by a 1.13.0 image and only when the inventory scan that already feeds `inventory_total_by_rank_bucket` ran (snapshots; deltas never carry the scan). A 1.13.0 delta on a 1.12.0 snapshot is safe to ingest: the rows are identical. --- ## 2026-09-03 — `policy_v1` mapping 2026.09.2: casino video games move to `gambling_games` Same columns, different published values for one IAB node. Pack `policy_v1` moves from mapping version **2026.09.1** to **2026.09.2**: `Video Gaming > Video Game Genres > Casino and Gambling Video Games` (simulated gambling, no real-money play) now projects to a new group, `gambling_games`, instead of `gambling`; the three real-money nodes (casinos, lotteries, poker) stay in `gambling`, and the group count is 47. Every vendor definition, regulator, rating body and app store we checked draws the line at real money, and two groups can be unioned by you (or merged by us later) while a merged group can never be un-merged. **Block `gambling` and `gambling_games` to keep the previous behaviour** — including real-money casinos the classifier may have stored under the video-game genre, which is a stored-label error the pack cannot correct. Takes effect in the feed with the **first full snapshot** published from an image that bundles 2026.09.2 — a pack change never rides a delta, so between that image's deploy and that snapshot no delta is published, and the first delta after it covers the gap. From that snapshot on, `policy_pack_version` on every row and `policy_pack.version` in the manifest read `2026.09.2`. Group definitions: [`website-categorization-policy-pack-v1.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v1.md). --- ## 2026-09-02 — the ensemble tokenizer repair and the re-categorization of English-routed rows (no schema change) No column changes. The values on a large set of rows change, and their `confidence` scale moves. **What was wrong.** Until 2026-08-29 the production ensemble fed three of its five English-language members tokens produced by another member's tokenizer, so those three ran near chance and the ensemble's tier-1 accuracy sat measurably below its design. Every English-routed label written before the fix carries that deficit; the non-English route was unaffected. **What was re-scored.** From 2026-09-01 the affected rows that have stored page text were re-scored with the repaired ensemble: English-routed rows plus rows with no recorded page language that nevertheless carry text. Rows with no stored text at all were not re-scored — most already ship without a category under the no-evidence guard (1.8.0). The pass covered the rows that had stored text when its corpus was assembled in late August 2026, and completed on 2026-09-03. **What you will see.** - Label changes on a minority of re-scored rows, concentrated in rows the old ensemble scored `LOW`. The most common move is between `Technology & Computing` and `Business and Finance`. - A confidence-scale shift on re-scored rows: the repaired ensemble is more decisive, so mean raw `confidence` rises and the `LOW` share falls. **A re-scored row's `confidence` is therefore not comparable with its own pre-repair value, and `confidence_tier` mixes are not comparable across the repair boundary.** - `calibrated_confidence` and `calibrated_confidence_tier` on re-scored rows still come from the map fitted on 2026-08-13 (`confidence_calibration_map_version` unchanged), i.e. on the pre-repair scale. Checked against a blind-judged cohort the map is not over-confident on the new scale — it understates `LOW` rows. A refit is planned and will be announced here with a new `confidence_calibration_map_version`. - `classified_at` moves to the re-score date; `model_name` (`ensemble-5en-3nonen`) and `model_version` (`v1+ml_en` / `v1+ml_non_en`) are unchanged, so the provenance columns alone do not distinguish a re-scored row — use `classified_at >= 2026-09-01`. - The vendor-rule flip (next entry) was not applied during this pass. **Which artifacts.** The off-cycle snapshot dated 2026-09-02 carries the rows re-scored before each of its partitions was exported; the 2026-09-06 snapshot carries the complete pass; daily deltas carry re-scored rows as they land, so the delta covering the pass is substantially larger than a normal night's. --- ## 2026-09-02 — two `model_name` families the dictionary did not yet list: `pipeline/adult_filter` and the new `pipeline/vendor_rule` flip (no schema change) No column changes. This entry documents one component that has shipped since artifact 1.10.0 and one that is new, and the data dictionary's `model_name` family count moves from six to eight. **`pipeline/adult_filter` (artifact ≥ 1.10.0).** A multilingual rule layer that overrides a benign-looking label when adult-platform markers, explicit title terms or adult-keyword density identify an adult site, always to `Sensitive Topics > Adult & Explicit Sexual Content`. Its `confidence` is not a probability — a fixed `0.95` or `0.90`, else a density ratio capped at `1.0`, range `[0.5, 1.0]` — and its rows carry NULL `calibrated_*`. Nothing about these rows changes with this entry; the dictionary now lists them. **`pipeline/vendor_rule` (rows from 2026-09-01).** The trained classifier over-generalizes software, SaaS, hosting, IT-services and telecom vendor sites into `Business and Finance`. A deterministic post-classifier now rewrites such a row to `tier1 = Technology & Computing` (`tier2`–`tier4` NULL, `iab_id = IAB-596`) when the classifier's label sits in one of three vendor-heavy tier-3 cells — `Business I.T.`, `Technology Industry`, `Telecommunications Industry` — and the page text carries high-precision vendor-identity signals. It is one-directional (it can remove a `Business and Finance` label from those cells, never add one), keeps the classifier's `confidence`, `confidence_tier` and `model_version` unchanged, and ships NULL `calibrated_*`. Filter on `model_name = 'pipeline/vendor_rule'` to see every flipped row; `policy_group` follows the new tier path. The rule was measured against a reserved holdout before it was enabled. Where it applies: the real-time API has run it since 2026-09-01 23:00 UTC. The bulk categorization lanes do not apply it yet — this page will say when they do — and the September re-categorization pass (previous entry) ran without it. Expect these rows to be absent or rare in early-September artifacts, and the vendor over-generalization to persist on bulk-labeled rows until then. --- ## 2026-09-01 — twelve columns appended: the reserved `site_purpose_*` block and the `policy_v1` projection (schema 1.12.0) Ships with the first ≥ 1.12.0 snapshot set. Twelve new columns, **appended** after column 26 in one bump; no existing column changes in meaning, position or value, so a consumer reading by name or by position is unaffected. Delta files carry all 38 columns. **Columns 27–35, `site_purpose_tier1` … `site_purpose_model_version`, are reserved for a second axis** — what a site is _for_, in the same IAB v3.1 vocabulary — and are **NULL on every row** at 1.12.0, with one exception: `site_purpose_status` (column 33) is never NULL. It reads `not_available` on every row that has not been assessed for a purpose, which at 1.12.0 is every live row, and `undefined` on rows that have no separate business purpose — the parked / inactive / dead lanes and rows the parked detectors flagged — derived from the same status evidence as `domain_status`, never from a model. `labeled` appears only once a purpose pass runs (a later release; the manifest's `site_purpose_model` is `null` until then). Reserving the columns now means one schema bump instead of two and lets you build your table once. Read `not_available` as "not assessed", not as "no purpose". **Columns 36–38 are the `policy_v1` policy-pack projection**: `policy_group`, a fixed, versioned rollup of the 704 IAB categories onto filtering-ready groups (`adult`, `gambling`, `piracy`, `drugs`, `weapons`, `hate`, `shopping`, `social_networking`, …) in the vocabulary DNS-filter, secure-web-gateway and endpoint-agent policies already use; `policy_basis`, the axis the group was projected from (`primary` at 1.12.0; `site_purpose` once that axis is labeled); and `policy_pack_version` (`policy_v1@2026.09.1`, constant within an artifact, carried per row so a CSV row is self-describing). The manifest gains `policy_pack: {id, version, sha256}`, `row_count_with_policy_group`, `site_purpose_model` and `row_count_by_site_purpose_status`. What the projection is: a pure function of the stored label. `Attractions > Casinos & Gambling`, `Shopping > Lotteries and Scratchcards`, `Sports > Poker and Professional Gambling` and the casino video-game genre all projected to `gambling` at this mapping version, so a blocking integration no longer has to know that IAB scatters gambling across four branches. What it is **not**: a correction of the stored label. A site whose stored label is wrong projects to the wrong group (an adult site stored as `Entertainment > Movies` reads `entertainment`); per-group accuracy has not been measured and no accuracy claim attaches to any group. The pack cannot express what IAB v3.1 has no node for — there is no malware, phishing, botnet or anonymizer group; threat intelligence is a separate product. The three policy columns are NULL exactly where `tier1` is NULL (every suppressed row) and where the tier path does not resolve; none of the new columns is carried into the `last_known_*` block. The policy columns are **derived, not tracked**: a pack version change is delivered by the next full snapshot, never by a delta, because it touches every row. The first ≥ 1.12.0 artifact you ingest must therefore be a snapshot. Pin `policy_pack.version` in `manifest.json` (or the per-row `policy_pack_version`). Group definitions and the open mapping decisions: [`website-categorization-policy-pack-v1.md`](https://website-categorization.whoisxmlapi.com/samples/website-categorization-db/website-categorization-policy-pack-v1.md). --- ## 2026-08-31 — NULL-category semantics: the demoted-label guard (schema 1.11.0) Ships with the first ≥ 1.11.0 snapshot set. Ensemble-family rows whose label the **write side branded parked-demoted** — since 2026-08-22 the pipeline stamps `is_parked` on the category row when its classify-time placeholder detector fires, and the crawl's own parked verdict on the record being classified carries the same flag — now ship **status-only**: category columns NULL, no `last_known_*` block, `domain_status` unchanged. The manifest gains a `demoted_label_suppression` block with the count. This closes the residual the 1.10.0 vetted-hash guard could not reach: the detector generalizes across template variants while the hash list is exact-match, so a demoted row whose template is not on the vetted list previously shipped its fabricated category — most visibly as placeholder `Technology & Computing > Computing` labels. Rows the hash guards already suppress are billed to those blocks; the suppression counts are disjoint. Consume the first ≥ 1.11.0 set as a **full per-partition replace** — a delta stream cannot retro-apply the mask (same rule as the 1.8.0/1.10.0 guards). The count trends toward 0 as re-crawls under the fixed detector replace the demoted labels' ground truth. --- ## 2026-08-24 — NULL-category semantics: the boilerplate-template guard (schema 1.10.0) Ships with the first ≥ 1.10.0 snapshot set. Ensemble-family rows whose category was produced from a **known boilerplate template** — the crawl captured a hosting provider's parking/coming-soon page, a stock web-server or panel default, or another vetted template rather than owner content, identified by `content_hash` membership on a vetted list — now ship **status-only**: category columns NULL, no `last_known_*` block, `domain_status` unchanged. The manifest's `boilerplate_template_suppression` block publishes the full hash list beside the count. At ship time the list is 60 vetted templates, concentrated in but not limited to the former `Web Hosting` tier-4 population: the crawl read the HOST's boilerplate, so the label described the hosting provider, not the domain. Consume the first ≥ 1.10.0 set as a **full per-partition replace** — a delta stream cannot retro-apply the mask (same rule as the 1.8.0 guard). The count trends toward 0 as re-crawls land: a re-crawl that captures real owner content changes the row's `content_hash` and releases it. Context for the `Web Hosting` category specifically: independent of this guard, a graded audit found the great majority of that tier-4 class to be placeholder inflation. This guard masks the template-provable share; the residual inflation drains through re-crawls under the fixed categorize-time detector. Until then, treat low-confidence `Web Hosting` leaf labels per the calibrated-band guidance — the class runs DOUBTFUL-heavy. --- ## 2026-08-15 — deltas carry status-only changes (branch 3, schema 1.9.0) A delta now includes status-only rows — inventory rows with no category row — whose inventory moved since the cutoff, keyed on the same `crawl_attempted_at` cursor as the `domain_status` branch. Previously these parked/placeholder rows changed silently between snapshots. Purely additive: no column changes, snapshots unchanged, and the rows look like any other status-only row (NULL category columns). The manifest's `status_only_lane` key is now part of a delta's identity — re-publishing the same cutoff with the lane toggled is refused. The status-only BASELINE still arrives only with a snapshot, and removals still require the full per-partition replace — the snapshot-ingest obligation stands. --- ## 2026-08-15 — NULL-category semantics: the no-evidence guard (schema 1.8.0) Ships with the first ≥ 1.8.0 snapshot set. Ensemble-family rows whose category was produced with **no captured page content** (`content_hash` NULL, with the `language` guard conjunct) now ship **status-only**: category columns NULL, no `last_known_*` block, `domain_status` unchanged. The manifest's `no_evidence_suppression` block counts the suppressed rows. Consume the first ≥ 1.8.0 set as a **full per-partition replace** — a delta stream cannot retro-apply the mask to rows that do not otherwise change. The count trends toward 0 as re-crawls land. --- ## 2026-08-13 — confidence calibration map `calib-map-v1` → `calib-map-v2` The map behind `calibrated_confidence`, `calibrated_confidence_tier` and `calibration_segment` was refitted. The manifest's `confidence_calibration_map_version` moves from `calib-map-v1` to `calib-map-v2`. Raw columns are untouched. Three things change: 1. **The metric changed, not just the numbers.** v1 estimated P(label correct) judged against **stored crawl-time page text**. v2 estimates P(label correct) judged against the page **as fetched at judge time** (blind LLM judge, exact tier-1 agreement). This is the stricter, buyer's-eye question: content drift since the crawl and domains that have since been parked now count **against** the label. Calibrated values drop substantially as a result — this is a semantics change, not a data regression. If you pin `confidence_calibration_map_version`, treat v1 and v2 values as incomparable. 2. **Teacher-rescored rows are now calibrated.** Rows whose `model_version` carries the `teacher_bf1` suffix (non-English Business-and-Finance re-judgments) previously shipped NULL `calibrated_*` by design, because v1's curves were fitted on the pre-rescore confidence scale. v2's fit sampled that population directly; those rows now publish calibrated values. `row_count_calibrated` in the manifest rises accordingly. 3. **`calibration_segment` loses one value.** `NULL|rest` no longer has its own curve — too few judged rows at fit time against the 50-row per-segment floor — so rows with no recorded language now publish `__global__`. No new segment values are introduced. Fit provenance is embedded in the artifact's own `fitted_on` field: blind-judged rows across three stratified samples of the 2026-08-09 snapshot, design-weighted to the calibrated population.