Website Categorization Bulk Feed — Changelog

Consumer-visible changes to the bulk feed's published values and semantics — including the ones that are invisible in the schema: same columns, different meaning or different numbers. Nothing else forces a consumer to notice those, so they are called out here.

Read this page beside the data dictionary Website Categorization bulk feed data dictionary, which defines every column and every value domain. Entries are newest first.

Download this document: website-categorization-feed-changelog.md Also available: the data dictionary website-categorization-feed-data-dictionary.md, the policy pack policy_v1 group reference website-categorization-policy-pack-v1.md and the policy pack policy_v2 group reference website-categorization-policy-pack-v2.md.

How to read a version number

feed_schema_version in manifest.json is a semantic version, and the component that moves tells you what to check. A MINOR bump is purely additive to the schema: new columns appended at the end of the contract order, new manifest keys, or a new manifest block — a reader that selects by name, or that stops reading after the column it integrated against, is unaffected by the append itself. A PATCH bump changes the published row population with no column change at all: no column is added, removed, renamed or retyped, and no manifest key is removed or given a new meaning, though a PATCH may add keys inside an existing manifest object — so validate the manifest permissively, as an extensible object rather than a closed set of properties. A MAJOR bump would reorder or rename a column; that is the licensable contract and it does not move without notice. Two bumps below are MINOR because they do both things at once — they append columns and move the row population — and each says so in its own entry.

One rule cuts across several entries and is worth stating once: a change that is derived at export rather than tracked per row reaches you only through a full snapshot. A row whose band, policy group or suppression status moves under a new rule is not a changed row and does not cross a delta cutoff. Every entry that carries that property says "consume the first ≥ x.y.z set as a full per-partition replace", and the exporter holds the same line on its side: a delta whose policy differs from the snapshot it extends is reported as a chain gap and is refused.

Delivery cadence

Export folders are named by the start of the window they cover, not by the time the export finished. A dated snapshot is complete if and only if exports/snapshot_<date>/run_manifest.json exists: read the sixteen partition manifests from that marker rather than listing the prefix, and do not ingest a partial set. The marker's absence on a current delivery means the set is still being written or recovered.

Deltas alone cannot keep you compliant. Removals — takedowns and suppressions — retract a row, and only a full per-partition replace expresses a retraction; the status-only lane's baseline is likewise snapshot-borne. The licence carries a term requiring each full snapshot to be ingested within an agreed window and applied as a per-partition replace rather than a merge. See the data dictionary Website Categorization bulk feed data dictionary for the exact wording.

2026-11-03 — three corrections to the data dictionary (wording only, no schema or version change)

Nothing published changes with this entry: no column, value or row moves.

Corrected in the data dictionary published beside this entry Website Categorization bulk feed data dictionary:

• §3: uncategorizable rows keep whatever label is already on file, if there is one. The dictionary no longer adds that usually there is none.
• §5: the note that introduces calibrated_confidence now says what it estimates, as the column's own section does: the probability that a strict blind judge, reading the page, assigns the same tier1, a strict-match estimate rather than the chance that a label is usable. It said only that the column is meant to be read as a probability.
• §10, the snapshot-ingest term: the next weekly set starts within seven days and is usually complete the following Monday, UTC, as §1.1 says. It said the set is complete by the following Monday morning.

2026-11-03 — the run marker's policy_group_transitions projects each move's full tier path, with an erratum for the sets from 2026-09-20 through 2026-11-01 (first artifact: the 2026-11-08 set's run marker; no schema or version change)

No column, manifest key or feed_schema_version changes. From the 2026-11-08 set, the run marker's policy_group_transitions projects each move through the full tier path it was recorded at, as data dictionary §10 Website Categorization bulk feed data dictionary now says. Deltas still do not carry the key.

Erratum. From the 2026-09-20 set through the 2026-11-01 set, the projection read each move's tier-1 label alone, so a move whose path falls under a deeper rule of the policy pack was counted in the group of its bare tier-1 label. The 2026-09-17 set, the first 1.14.0 set, had no earlier marker to measure from and published []. Deeper rules sit under several tier-1 labels: under Sensitive Topics, for example, deeper paths map to groups such as adult, hate and weapons, while the bare label maps to sensitive_other. In the 2026-10-18 set's marker, a move to or from Attractions > Casinos & Gambling (gambling) was counted as travel, the group of bare Attractions. Other policy_group_transitions entries can also have counted part of their moves in the wrong group pair. top_tier1_transitions and row_count_tier1_changed are unchanged.

2026-11-03 — for domains ranked in the top million (popularity buckets 0 and 1), a page that gave us no readable text ships as blocked, not parked (first artifacts: delta_2026-11-03_100000Z and the 2026-11-08 set; no schema or version change)

No column, manifest key or feed_schema_version changes. What changes is the lane of one group of rows: those whose crawl succeeded but whose capture extracted to no readable text — no title, no description and no body text.

• Before: every such row that held a category read from a page shipped as parked, with its category columns NULL, because the capture's content fingerprint was read as the parking-provider redirect stub that produced most of these captures in July 2026. A page rendered entirely by JavaScript, a meta or JavaScript redirect and a bot interstitial produce the same fingerprint, so live sites shipped as parked. That contradicted the data dictionary's meaning of parked, "an affirmative parking signal".
• Now: since 2026-07-31 our crawler recognizes the parking-provider redirect stub when it fetches it and records the domain as parked directly. On a domain in popularity bucket 0 or 1 — a best rank of 1,000,000 or better on the Tranco, Majestic Million or Cisco Umbrella list (data dictionary §10 Website Categorization bulk feed data dictionary) — a capture made from that date that extracted to no readable text is read as a live site we could not read, and its row ships as blocked. Most such pages are live; a minority are hosting placeholders or parking pages our crawler could not recognize. Everywhere else such a row still ships as parked: outside the Tranco top million, many of the pages we checked turned out to be parking, for-sale or registrar pages, or hosting placeholders, whose text sits in an image or a script, so an empty capture alone cannot tell them from a live site. A capture made before 2026-07-31 still ships as parked until a crawl fetches the page again: a 304 Not Modified answer keeps the earlier capture, and with it the parked lane. An export run without the popularity buckets (rank_ladder: absent) moves no row. Data dictionary §3 now says so for both lanes.
• What does not change: the category stays withheld on these rows wherever they land, with no last_known_* block, because no label can be checked against a capture with no text. A domain our crawler recorded as parked (a parking page, a for-sale page, the redirect stub itself) still ships as parked. A row whose only label is a name-only one keeps the lane its crawl decides, as since the 2026-10-02 entry below. parked_stub_suppression.row_count_overridden counts the same rows as before.
• site_purpose_status: on a blocked or uncategorizable row in popularity bucket 0 or 1 whose empty capture was made from 2026-07-31 it now reads not_available instead of undefined, including rows whose lane does not move, because an empty capture is not evidence that a domain has no business purpose. Rows on parked, inactive and dead still read undefined, as does a parked-demoted label (data dictionary §11), and row_count_by_site_purpose_status in manifest.json moves with it.
• What you will see: rows that qualify move from parked to blocked; no row outside popularity buckets 0 and 1 changes lane. Live checks of a sample of these pages found mostly live sites, along with pages with no site behind them, redirects to other domains, JavaScript apps that looked like gambling sites (judged without rendering them), very few parking or for-sale pages, and hosting placeholders or blank pages, which are wrong as parked too: their lane is inactive.
• When: from the first daily delta and the first Sunday set exported after the deploy that carries this change. The lane is decided when the feed is exported, so a row that moves is not a changed row: a daily delta carries it only when we crawl the domain again within that delta's window, and the first full set after the deploy is the first artifact that carries every such row. A domain's bucket is re-stamped monthly, so a domain that enters or leaves buckets 0 and 1 can change lane without being crawled again; that move, too, reaches you only through a full set.

2026-11-02 — the 1.17.0 notice: IAB Content Purpose = News in two appended columns (first artifacts: the Wednesday 2026-11-04 delta and the Sunday 2026-11-08 set)

Nothing published changes with this entry. It is the customer notice for feed 1.17.0, dated ahead of its first artifact. feed_schema_version 1.16.0 → 1.17.0, a MINOR bump on the definition in data dictionary §1 Website Categorization bulk feed data dictionary: two columns are appended after column 44, and columns 1–44 keep their names, positions and types. Reading by name, or by position up to column 44, is unaffected, and no row enters or leaves the feed.

What it says. IAB Content Taxonomy v3.1 has no News topic, so a news outlet's tier1–tier4 name what it covers — Politics, Sports and so on. The IAB Tech Lab expresses news as a Content Purpose: a Descriptive Vector carried next to the topic, never in place of it, and column 45 carries its id. No topic column changes: filter news on column 45 and keep your topic rules on columns 2–6. 1020 is not a Content Taxonomy id and never appears in iab_id or openrtb_cat.

Where it comes from — a curated list, and only that. The value is set on the domains of a curated list of known news publishers maintained by our team, and on no other row; the list is versioned, and every change to it is reviewed before release. Every domain on the list is also one of our hand-vetted curated/* rows, so the value appears only on those rows. The feed reads no page markup, so it cannot tag a news site that is not on the list: this is a list, not news-site coverage. Most news sites in the feed read NULL.

NULL means unknown, never "not news". Do not build an "exclude news" rule out of the NULL rows.

schema_org is a reserved value of column 46. The real-time API's opt-in include=content_purpose block has a second source, schema_org — the site's own schema.org NewsMediaOrganization markup. The feed does not emit it in 1.17.0, because the export carries no page markup. It is named now so a loader that validates column 46 against a strict enum can accept it in advance; a release that starts emitting it will say so here first. For a listed domain the feed and that API block agree (1020, curated); a domain the API tags from markup alone reads NULL in the feed.

When. The 1.17.0 exporter deploys no earlier than Tuesday 2026-11-03 at 13:00 UTC, after that day's 10:00 UTC delta; if this date moves, we will say so in a dated amendment to this entry. If it deploys then, the first 1.17.0 artifact is Wednesday 2026-11-04's delta, in the folder delta_2026-11-03_100000Z, and the first 1.17.0 full set is the Sunday 2026-11-08 set. Nothing published before the deploy carries the two columns.

The deltas are enough for this change. Curated rows ride every delta whole, so the Wednesday 2026-11-04 delta already carries every tagged row. If you consume deltas only, add the two columns to your table as NULL and apply that delta: the two columns then hold exactly what a 1.17.0 full set publishes in them. A chain check that compares each artifact with the one before it (data dictionary §10) reports the 1.16.0 → 1.17.0 version step on that first delta — it cannot know what a version change carries — so expect that one report; this entry is what it points to.

The 1.16.0 rule still binds. 1.17.0 carries 1.16.0's band policy unchanged: column 8 and confidence_band_policy (v4) mean exactly what the 1.16.0 notice below says. That notice asks for a full set at 1.16.0 or later before you rely on ≥ 1.16.0 deltas, and the 2026-10-11 set was the first such set.

Two new manifest keys, on every partition and delta manifest (data dictionary §10); manifest_keys_version moves 1.0.0 → 1.1.0, and no key is removed or given a new meaning:

• content_purpose_list — {path, sha256, row_count} of the list the values were read from. sha256 changes whenever the list does; row_count is how many domains it names.
• row_count_by_content_purpose_source — rows per column 46 value: curated, schema_org (always 0 in 1.17.0) and null; sums to row_count, and the run marker sums it across the sixteen partitions.

On a delta, and on a full set's run marker, the curated count equals the list's row_count unless a listed domain did not ship as a curated row in that artifact — a takedown withholds it, for example. A partition manifest counts only its own partition's tagged rows. The run marker also carries the set's list as content_purpose_list ({sha256, row_count}), and a set whose partitions were tagged from two different lists gets no marker — it is not complete until it is re-published from one.

2026-10-30 — deltas re-read crawls stamped in the 30 minutes before their window, and documentation corrections

No column or schema-version change. Which rows a delta carries changes slightly, and the data dictionary published beside this entry Website Categorization bulk feed data dictionary corrects three passages.

From delta_2026-10-30_100000Z, each delta also re-reads crawls stamped in the 30 minutes before its window. A crawl is stamped when its page is fetched and saved a little later, so a crawl stamped just before a 10:00 UTC cutoff could be saved after that delta's read and still fall outside the next delta's window, so it reached no delta and only the full set carried it. The look-back delivers any crawl saved within 30 minutes of its stamp. Its cost is repetition: a delta can repeat a row the previous delta carried (its status_observed_at falls in the 30 minutes before the window), which an upsert by domain absorbs.

Corrected in the data dictionary published beside this entry:

• §1.1: after loading a full set, resume with the delta whose window contains the whole of the set's export — for the weekly set, normally the one dated the day before it — and then every later delta; skipping it loses rows. "Nothing is lost" is withdrawn: a crawl saved after a delta's read could reach no delta, a row can reach you one delta after its crawl, and a full set reconciles anything a delta did not carry.
• §2: domain is normally, not always, the registrable domain — some rows are a host below one or a bare public suffix. Look a name up as written, then fall back to its registrable domain.
• §10: deltas are not partitioned. Replace all sixteen partitions of a set together, or route each row to its partition with the published recipe, and re-apply that partition's delta rows after reloading it.

2026-10-21 — documentation corrections (no data change)

Nothing published changes with this entry: no column, value or row moves. It corrects documentation that a review of the 2026-10-18 set found out of step with the data, and it carries one notice that is late.

Late notice: the bulk categorization lanes apply the vendor rule, and have since 2026-09-02. The data dictionary said they did not yet and that this changelog would say when they did; it did not say so until now. A row the rule rewrote carries model_name = pipeline/vendor_rule, tier1 = Technology & Computing with tier2–tier4 NULL, and NULL calibrated_*. The rule itself, and the trade it makes, are in data dictionary §5 Website Categorization bulk feed data dictionary and in the 2026-09-02 entry on the pipeline/adult_filter and pipeline/vendor_rule families, below.

Corrected in the data dictionary published beside this entry:

• §1.1: how long the weekly set lately takes, when it is complete, when a daily delta lands, and the delta rows with no timestamp inside the window: a small share of every delta, whose classified_at is NULL, not old.
• §3: a known issue — some live sites ship as parked or inactive, so neither lane is a cull key on its own; parked, not blocked, is the largest non-active lane; how to filter the labels blocked rows carry; curated infrastructure zones carry an asserted active.
• §4: no crawl cadence is promised.
• §5: what calibrated_confidence estimates — the probability that a strict blind judge assigns the same tier1, which on average understates how often a label is acceptable; the vendor rule in the bulk lanes; the gambling filter's band, which reads LOW on every row from 1.14.0; which rows the language-NULL guard spares; that the v4 API returns no confidence_tier; and how to read a name-only row's model_version.
• §8: the lane mix differs from one partition to the next.
• §10: rank buckets "0", "1" and "3" are emitted and "2" is reserved, and coverage_by_rank_bucket counts presence; parked_stub_suppression does not trend to zero; the run-level language map usually names 29 subtags.
• §13: the brand-safety example no longer reads as exact.

Corrected in this changelog: in the 2026-09-11 entry below, how the delta rows with no timestamp inside the window read, and, under the 2026-09-13 correction of its blocked-lane item, a dated note that 30-day eligibility for a re-attempt is not a schedule.

2026-10-07 — the 1.16.0 notice: teacher-rescored rows never publish HIGH (first artifacts: delta_2026-10-09_100000Z, published Saturday 2026-10-10, and the Sunday 2026-10-11 set)

Nothing published changes with this entry. It is the customer notice for feed 1.16.0, published before the deploy it describes so that a loader can adjust first. feed_schema_version 1.15.0 → 1.16.0, a MINOR bump: no column is added, removed or moved, and the 44 columns keep their order and their types. What moves is the published value of one column — confidence_tier (column 8) — on one population.

Unlike 1.15.0, this re-rates rows you already have. The 1.15.0 notice below describes a new lane arriving and says no other row re-bands; that was true of 1.15.0 and is not true of this release. Rows published in every set since 1.14.0 change band here. That is why this is its own release with its own notice rather than an amendment to that one.

When. The 1.16.0 exporter deploys Friday 2026-10-09 at 13:00 UTC. The first artifact that reports feed_schema_version 1.16.0 is the daily delta published Saturday 2026-10-10, in folder delta_2026-10-09_100000Z: a delta folder is named by the start of its window, so this one carries the deploy's date but is cut after the deploy. Every delta after it reports 1.16.0 or later. The re-rating itself does not arrive with them. The band is derived at export rather than tracked, so a row whose band moves is not a changed row and does not cross a delta cutoff: no delta carries the re-rating as a whole, and a capped row reaches a delta only when the row changes for some other reason, such as a re-crawl or a re-label. The re-rating reaches you with the Sunday 2026-10-11 full set, and only by ingesting that set. Nothing published before 2026-10-09 carries any of this.

Ingest the full 2026-10-11 snapshot set before relying on 1.16.0 deltas. This is the same transition rule the 1.14.0 notice states, for the same reason and with no exception: the first ≥ 1.16.0 artifact you ingest must be the full sixteen-partition set, or your table will carry two bands for one row set with nothing marking which is which. manifest.json carries confidence_band_policy so you can tell them apart — see below. In practice: skip delta_2026-10-09_100000Z, the one delta that reports 1.16.0 before that set, and do not apply it after the set either, because it is older than the set; once the set is loaded, resume with delta_2026-10-10_100000Z.

Which rows re-band

Rows whose model_version (column 11) carries one of three provenance suffixes — the second +-delimited field of that column, split_part(model_version, '+', 2) — and whose model_name (column 10) is an ensemble writer (ensemble-*, multitask-ensemble-*):

On such a row:

• a stored HIGH publishes MEDIUM;
• and publishes LOW where calibrated_confidence_tier (column 25) is DOUBTFUL;
• a stored MEDIUM also publishes LOW where that same column is DOUBTFUL — a second, smaller population that leaves MEDIUM as well as the one that leaves HIGH. If you key on confidence_tier IN ('HIGH','MEDIUM') rather than on 'HIGH' alone, these rows leave your selection too;
• a stored LOW is unchanged, and so is a stored MEDIUM at any other calibrated tier — the rule is a cap: it moves a band down or leaves it alone, and it never promotes anything;
• PARKED and a NULL band pass through untouched, as under every policy.

No other row and no other column moves. A row outside the three suffixes — including the ml_en / ml_non_en route stamps and the bare base version, which record which routed model produced the label rather than a pass over it — bands exactly as it does today. The value domain of column 8 is unchanged: HIGH / MEDIUM / LOW, plus PARKED. confidence, the calibrated_* block, tier1–tier4, iab_id, model_name, model_version and policy_group carry exactly what they would have carried under the old policy, on every row including these.

No Business and Finance row moves. The teacher pass writes an ensemble model_name, so a teacher row that kept tier1 = 'Business and Finance' has been capped by 1.14.0's ensemble rule since that release. What 1.16.0 moves is the capped-suffix population whose tier1 is not Business and Finance.

This is not a loss of rows, labels or groups. Every row that re-bands still ships, with the same tier1–tier4, the same iab_id, the same policy_group and the same model_name. If your block list or your routing is keyed on confidence_tier = 'HIGH', it will stop seeing them — and if it is keyed on confidence_tier IN ('HIGH','MEDIUM') it will still stop seeing the smaller MEDIUM-at-DOUBTFUL population, which lands on LOW. That is what this notice is for.

Why

A blind re-read on 2026-09-12 put a sample of these rows in front of a judge that reads the live page and compared its verdict with the stored label. This family's labels agreed with a fresh reading of the page far less often than the threshold we had pre-registered for keeping them at our top confidence word — so publishing them as that word was not defensible. They are scheduled for re-vetting; until that has happened they publish at MEDIUM at most, which is the honest reading of what a re-scoring pass over a label actually establishes.

The three suffixes travel together because they are one lane: a re-scoring pass, its "nothing to read" marker, and the purpose pass that follows it.

How to leave it out, and how to rebuild the band you have

To exclude the capped population entirely, on any version of the feed:

WHERE COALESCE(split_part(model_version, '+', 2), '') NOT IN ('teacher_bf1', 'nocontent_bf1', 'teacher_purpose_v2')

The COALESCE is load-bearing, not defensive. Rows that ship a crawl status but no category — the status-only lane — publish model_version as NULL, and in SQL NULL NOT IN (...) is NULL, not true. Without it those rows fail the predicate and drop out of a query whose job is to exclude the capped population and keep everything else.

To rebuild the band as it was before any of the policies — the raw confidence cut — confidence (column 7) is untouched on every row, and the policy-v1 reconstruction recipe in the 2026-09-10 1.14.0 notice below still applies unchanged, including its rounding rule. On every row the feed already carried before 1.15.0, that recipe returns the band the feed published through 1.13.x, whatever policy the artifact you are reading was produced under. A name-only row (model_name beginning llm-name/, the 1.15.0 lane) is the exception: it is stored at MEDIUM whatever its confidence, so there the recipe can return HIGH where policy v1 returns MEDIUM.

confidence_band_policy in manifest.json reads v4 from the first 1.16.0 artifact on (v1 or an absent key is the raw pass-through, v2 from 1.14.0, v3 from 1.15.0). Pin it beside confidence_calibration_map_version, as before: the DOUBTFUL cell above is decided by the calibration map, so a refit moves which of these rows land on LOW rather than MEDIUM.

The v4 API returns no confidence_tier, so no v4 answer changes with this release. On the legacy /categorize route, an answer served from the stored corpus the feed is exported from (routed_to: "hetzner-cache") carries the v4 band from the same deploy, so until you load the 2026-10-11 set it can differ from the band in your copy of the feed.

2026-10-02 — a name-only label no longer changes a row's domain_status (first artifacts: delta_2026-10-02_100000Z and the 2026-10-04 set; no schema or version change)

No column, manifest key or feed_schema_version changes. The 1.15.0 notice (the 2026-09-14 entry below) and data dictionary §3 Website Categorization bulk feed data dictionary say a name-only label leaves domain_status as the crawl decided it. The 2026-10-01 entry describes a small set of rows for which the deltas published before this change, starting with delta_2026-09-30_100000Z, did not keep that promise: the exporter counted the name-only label as a category when it chose the lane. From the artifacts named above it no longer does:

• Rows that would ship as active ship as uncategorizable, still with their name-only label. A name-only label no longer puts a row in active, so these rows also leave the manifest's content_accuracy_denominator_row_count.
• Rows that would ship as parked ship as uncategorizable. The last page captured for each was a parking provider's redirect page, so their category columns stay NULL, as before.

Only rows whose sole label is a name-only one and whose latest crawl succeeded change lane; no other row changes lane and no category changes. Deltas carry these rows again only when their classified_at or crawl_attempted_at crosses a delta's cutoff, as a new label or a new crawl attempt does; this change moves neither, so a consumer that applies only deltas keeps the earlier lane until then. To pick up every corrected lane, replace from the 2026-10-04 set, the first full set carrying name-only rows.

2026-10-01 — the name-only lane added domains the 2026-09-13 set did not carry, rather than labelling existing rows (wording only, no schema or version change)

No column, manifest key or feed_schema_version changes. This entry corrects the 1.15.0 notice (the 2026-09-14 entry below) on four points: how many rows the lane labels, which rows those are, the lane a labelled row ships in, and when a page-derived label replaces a name-only one. The rows were written on 2026-09-30 as announced:

• The load wrote fewer labels than the notice expected. A name-only label is written only where we hold no category row for a domain, labelled or not, and some of the domains on which the two model families agreed had one when the load ran: some were blocked or uncategorizable rows of the 2026-09-13 set without a category, the rows the notice described, which the lane cannot label (next point), and the others were not in that set but had a category row by the time of the load.
• None of the domains the lane wrote was in the 2026-09-13 set. The notice said the lane labels today's blocked and uncategorizable rows. It cannot: the lane writes only where we hold no category row for a domain, and the feed carries such a domain only when its crawl finds it parked or a placeholder, or it is on our curated list. The first artifacts carry them.
• Most of them ship with their name-only label, nearly all as blocked. The rest ship with NULL category columns, as data dictionary §3 Website Categorization bulk feed data dictionary describes. Nearly all of them are in the feed only because a name-only row now exists for them. Some are domains the crawl found dead, inactive or parked, where a name-only label is withheld like any other; for others the last page we captured was a parking provider's redirect page, which withholds the category on every lane (parked_stub_suppression in the manifest, dictionary §10). The filters under How to leave it out in the notice remove every name-only label but not these rows, which look like any other row without a category.
• When we checked on 2026-09-30, a small number of them would ship in the lane a label read from the page would give them, although the notice said the label leaves the lane the crawl decided. Their crawl reached the site but produced no label from the page, which on its own makes a domain uncategorizable: some would ship as active with their name-only label, and others as parked with NULL category columns, because the last page we captured from them was a parking provider's redirect page. To tell a name-only label from one read from the page, test policy_confidence_basis or model_name, not domain_status. Data dictionary §3 now notes this and points here.
• Some are already being replaced. When a later crawl reads a domain's page and our classifier labels it, that label replaces the name-only one. The first replacements came within hours of the load. The notice and data dictionary §5 said this happens on the domain's next successful crawl; both now say it this way.

The first artifact carrying these rows is the delta in folder delta_2026-09-30_100000Z, the 10:00 UTC run on 2026-10-01 (folders are named by the start of their window); its manifest counts the labelled ones under name_only in row_count_by_writer_family_x_tier. The 2026-09-14 entry now carries a dated amendment pointing here, and its description of the rows the lane labels and its sentence on replacement are corrected in place; nothing else in it changes.

2026-09-28 — domains whose name no longer resolves now ship as dead, not blocked (no schema or version change)

No column, manifest key or feed_schema_version changes. What changes is the lane a domain ships in once our crawler learns that its name no longer resolves, and with the lane, the category. dead rows carry every content column NULL by design (data dictionary §3 and §9 Website Categorization bulk feed data dictionary), so a row that would have stayed labelled in blocked now ships without a current label. Where we keep history for its earlier label, that label appears in the last_known_* columns (§8).

• Before: when a crawl got a definitive DNS answer that the name does not exist (NXDOMAIN) or has no address, we recorded a failed crawl, and the domain shipped in blocked. That did not match the data dictionary's meaning of dead, which then read "DNS failure / NXDOMAIN / 404".
• Now: the crawler records that answer as its own status, and the domain ships as dead. DNS answers that are not definitive, such as a timeout or a server failure, still ship as blocked. Data dictionary §3 now says so for both lanes.
• When: in crawls from 2026-09-23. It applies to a domain the next time we crawl it, so a blocked row whose status_observed_at is before 2026-09-23 may still be a name that no longer resolves. Daily deltas carry the change from delta_2026-09-22_100000Z, published 2026-09-23, and the 2026-09-27 set is the first weekly set to carry it.
• What you will see: the dead lane grew sharply in the 2026-09-27 set, and blocked grew more slowly than in earlier weeks. In that set, each partition our nightly crawl visited after the change added far more dead rows than the partitions it visited just before. Most rows that move from active or parked straight to dead are this change; a 404 could already do that.
• Which domains this reaches. A domain whose name no longer resolves ships as dead only if we hold a category row for it, labelled or not (data dictionary §10), and a curated/* row stays active, because our curated list asserts that lane rather than a crawl observing it (§3). A domain without a category row does not ship as dead, as before, so the manifest's inventory_dead_status_by_rank_bucket grows much faster than the dead row count. Those keys also count as never-shippable, which is the main reason coverage_shippable_by_rank_bucket rose in the 2026-09-27 set.

2026-09-27 — how deep a path goes, and what the 2026-08-22 audit adjudicated (wording only, no schema or version change)

No column change, no manifest change, no feed_schema_version change and no value in the feed moves. Data dictionary §5 Website Categorization bulk feed data dictionary ("What it estimates — and what it deliberately does not") made two statements that were wrong, both now corrected:

• The trained classifier does not cut a path where the page evidence runs out. The dictionary said labels are emitted "to the depth the evidence supports". Below tier1 the classifier keeps the most likely child at each tier whenever that child holds more than a small fixed share among its siblings, so nearly every path it writes runs on until the taxonomy has no deeper category; how deep a path goes is not a measure of how much the page supported. The guidance is unchanged: key policy on tier1 (or tier1 + tier2) and treat tier3 / tier4 as descriptive detail.
• No person adjudicated the 2026-08-22 audit. The dictionary said its two judge passes had "human adjudication of every disagreement". A language model re-ruled most of the rows the first pass judged wrong, settled the rows where the reviewed first pass and the second judge disagreed over correct versus wrong, and marked one more row correct that neither judge had called correct. No verdict was adjudicated by a person, and the judges' other disagreements were not adjudicated, among them disagreements on depth alone and on whether the page could be judged at all.

2026-09-15 — a second, convention-named copy of every set and delta under datafeeds/ (withdrawn the same day)

Withdrawn, 2026-09-15. A second copy of every set and delta's data files and manifests, under a datafeeds/ sub-folder and named the way the WhoisXML datafeeds server names files, was announced for the exporter deploy of Thursday 2026-09-17 and withdrawn the same day. The WhoisXML datafeeds team asked for a different shape, and that copy is now built and delivered outside this bucket, so no set and no delta under exports/ carries a datafeeds/ sub-folder, and there is nothing to exclude when you sync. Only the 2026-09-13 set has one, filled by hand before the change of plan; it stays (nothing under exports/ is ever deleted) and you can ignore it. Nothing about the documented objects changed: the sixteen partition folders, run_manifest.json, the delta folders and every name in the data dictionary stay exactly as they are.

2026-09-15 — 1.14.0 moves a week earlier: first artifact Thursday 2026-09-17, first Sunday set 2026-09-20

The date the entries below first announced — the Sunday 2026-09-27 snapshot set — is superseded. Nothing about the schema, the columns or the corrections changes; only when they reach you. The schema and the corrections it carries were ready ahead of the announced date, and moving the exporter change forward keeps it clear of a separate release the following week.

Amended 2026-09-17: no delta publishes on Friday; the first 1.14.0 delta is Saturday's. The 2026-09-17 set started right after Thursday's deploy and, like every full set, takes about a day to run, so it still holds the export lock when Friday's 10:00 UTC delta is due. That run is skipped, exactly as Sunday's is every week while the weekly set runs. Saturday's 10:00 UTC delta covers Thursday 10:00 UTC to Saturday 10:00 UTC, so nothing is lost and the chain has no gap.

• The exporter deploys Thursday 2026-09-17, after that day's 10:00 UTC daily delta.
• The first 1.14.0 artifact is an off-cadence full set, exports/snapshot_2026-09-17/. The run starts immediately after the deploy and the set is complete when its run_manifest.json appears — expect that on Friday 2026-09-18 UTC, about a day after the run starts. It is a full per-partition replace exactly like a weekly set: the same sixteen partitions, the same partition and run manifests, nothing special in its shape. Treat it as a weekly set that lands on a Thursday.
• The first 1.14.0 delta is Saturday 2026-09-19's 10:00 UTC run, in the folder delta_2026-09-17_100000Z — named, as always, by the start of its window — covering Thursday 10:00 UTC to Saturday 10:00 UTC. Thursday's delta ran before the deploy, on 1.13.2, so the chain has no gap.
• Sunday 2026-09-20 is the first regular weekly 1.14.0 set. The Sunday 2026-09-27 set is now an ordinary weekly set with nothing special about it.
• Nothing published before 2026-09-17 carries 1.14.0. The 2026-09-13 set and every delta through the one that ran at 10:00 UTC on 2026-09-17 (folder delta_2026-09-16_100000Z) report feed_schema_version 1.13.2.

Ingest a full 1.14.0 snapshot before relying on 1.14.0 deltas. The rule the 2026-09-10 entry states is unchanged and now points at the 2026-09-17 set: three of the 1.14.0 changes are re-derivations that touch every row, and a delta cannot carry a row that did not move. The dates in the 2026-09-10 entries below have been corrected in place to match.

2026-09-14 — the 1.15.0 notice: a name-only label lane for the unreadable head (first artifacts: the daily deltas from 2026-09-30 and the Sunday 2026-10-04 set)

Nothing published changes with this entry. It is the customer notice for feed 1.15.0, dated two weeks ahead so a loader has the whole window to adjust. feed_schema_version 1.14.0 → 1.15.0, a MINOR bump: no column is added, removed or moved; column 42 gains one value, and a small population of domains is added to the feed, most of them with a category.

Amended 2026-09-15, before any 1.15.0 artifact exists: the lane ships the HEAD ONLY. The torso validation this entry reserved came back below the precision bar we set for shipping it, so rows ranked 100,000–1,000,000 are not labelled this way for now. Nothing else in this entry changes: the mechanism, the column values, the dates and the filter are exactly as first published. If the torso ships later it will be its own changelog entry, not a quiet addition to this one.

Amended 2026-10-01: the lane added domains that the 2026-09-13 set did not carry, not labels on existing rows. Most of them ship with a name-only label, and when we checked on 2026-09-30 a small number would ship in the lane a label read from the page would give them, although the domain_status row of the table below says the label leaves the lane the crawl decided. The first delta carrying them is folder delta_2026-09-30_100000Z; see the 2026-10-01 entry. The filters under How to leave it out remove every name-only label, but not the rows the lane added that ship without one. Under What the lane is below, the description of the rows it labels and the sentence on replacement are corrected in place; nothing else in this entry changes.

Amended 2026-10-02: the domain_status row of the table below holds again from the first artifacts exported after the 2026-10-02 deploy; a consumer that applies only deltas picks up the corrected lanes by replacing from the 2026-10-04 set. See the 2026-10-02 entry.

When. The 1.15.0 exporter deploys Tuesday 2026-09-29 at 13:00 UTC. The rows described here are written on Wednesday 2026-09-30, so the first artifacts carrying them are the daily deltas from 2026-09-30 and the Sunday 2026-10-04 full set. Nothing published before 2026-09-29 carries any of this; a delta between the deploy and the load reports 1.15.0 with no name-only rows in it yet. No full-snapshot ingest is required before relying on 1.15.0 deltas: no existing row is re-derived, so every new row moves and every delta carries it.

What the lane is

For domains in the Tranco top 1,000,000 for which we held no category row, mostly sites the crawler cannot read, two independent language-model families label the site from the domain name alone, separately. A row ships only where both agree on the tier-1 category and each puts at least 0.7 on it; otherwise nothing is written and the lane leaves the domain as it was, which keeps it out of the feed unless its crawl finds it parked or a placeholder, or it is on our curated list. Every row that ships with a name-only label is ranked in the top 100,000 (see the amendments above and the 2026-10-01 entry). The lane is gap-fill only: it never replaces a label that exists, and a page-derived label replaces it once a later crawl reads the page and our classifier labels it. Only the top 100,000 is labelled this way today, and nothing ranked past 1,000,000 ever will be.

How a name-only row reads

The torso

The separate validation this entry reserved for the torso (rank 100,000–1,000,000) ran on 2026-09-15, judged by the same blind judge that scores the rest of the feed, and did not clear the bar we set as the condition for shipping it. Those labels are therefore not shipped, and the first artifacts carry head rows only. We would rather withhold a label than ship one below the bar we set.

How to leave it out

To exclude the name-only lane entirely, on any version of the feed:

WHERE COALESCE(policy_confidence_basis, '') <> 'name_only'

or, equivalently, on the writer family:

WHERE COALESCE(model_name, '') NOT LIKE 'llm-name/%'

The COALESCE is load-bearing, not defensive. Rows that ship a crawl status but no category — the status-only lane — publish policy_confidence_basis and model_name as NULL, and in SQL NULL <> 'name_only' and NULL NOT LIKE 'llm-name/%' are NULL, not true. Without it those rows fail the predicate and drop out of a query whose job is to exclude the name-only lane and keep everything else.

A block list built on confidence_tier = 'HIGH' never sees these rows. The real-time API does not serve name-only labels: it keeps answering those domains live, as today.

2026-09-13 — a 304 Not Modified answer is not a blocked site (first artifact: the 2026-09-17 set)

No column change and no schema-version change: the same rows carry a different domain_status. The correction ships with the 1.14.0 exporter (deploy Thursday 2026-09-17), so the 2026-09-17 snapshot set is the first artifact that carries it.

• What was wrong. The nightly refresh re-reads a site with a conditional request. When the server answers 304 Not Modified, the crawler records the status not_modified: the content we hold is confirmed current and nothing was fetched. The exporter listed that status among the blocked ones, so a healthy, unchanged site was published as blocked — against §3 of the data dictionary Website Categorization bulk feed data dictionary, which defines blocked as a site we could not read.
• What changes. From the 2026-09-17 set a not_modified status lanes like a successful crawl: active where the row carries a label, parked where the content we hold is the parking-provider stub (the stub override in §10 applies as before), uncategorizable where no label exists. Nothing else about the row moves: the label, its calibrated value and status_observed_at (which dates the 304 answer) are as they were. The real-time API is not affected: its domain_status answer comes from the category store, not from this rule, and the category it returns for these rows does not change.
• What to do. The rows that move did not change in the inventory, so a delta cannot carry them; the first 1.14.0 set is a full per-partition replace anyway (see the 2026-09-10 entry), and ingesting it brings every such row to its corrected lane. Expect row_count_by_domain_status to show blocked lower and active higher, before the week's own movement.
• The 2026-09-11 entry below first attributed the blocked lane's growth over the 2026-08-30 → 2026-09-06 sets to paused re-crawls (corrected there). The mapping corrected here also contributed to that growth, alongside the re-crawl cadence and real anti-bot answers.

2026-09-11 — one new manifest key and three clarifications from the data audit of the 2026-09-06 set

No column change. The key ships with 1.14.0 (first artifact the 2026-09-17 set); the three clarifications describe artifacts you already have.

• Unicode-form domains are withheld and counted. The 2026-09-06 set carried rows whose domain was an internationalised name in Unicode form (helaförsäkring.se), most of them beside the same name in punycode (xn--helafrskring-mcb3w.se) — two rows for one registrable domain, and the two disagreed on lane or label in many of those pairs. From 1.14.0 the exporter withholds a non-ASCII domain and publishes the count as row_count_excluded_non_ascii_domain (partition and run manifests; 0 is a measured zero). The withheld rows include names whose punycode form is not in the feed yet; they return once the inventory ingest normalises them, after which the counter reads 0. A delta cannot retract a row you already hold, so the correction reaches you with the first 1.14.0 set — a full replace. Data dictionary §2 and §7 Website Categorization bulk feed data dictionary now state the form: lowercase ASCII, punycode for internationalised names.
• status_observed_at can be null on a handful of rows, and the manifest says where. A handful of rows of the 2026-09-06 set carry a status with no attempt timestamp. The dictionary's §4 promised every row is dated; it now says every crawler-observed row is dated and points at the null bucket of row_count_by_status_age (1.14.0), which counts the exceptions per lane. A curated row is dated only where the inventory has a row for that domain.
• Deltas are upserts; do not filter them by timestamp. A small share of every delta has no timestamp inside the delta's window: classified_at is NULL and status_observed_at is older (corrected 2026-10-21: first stated with both timestamps older). Such a row qualified through a category-side event the feed does not show — an abstain, or a label withdrawn by suppression — and it is a real update: applying only the rows whose timestamps fall inside the window keeps withdrawn labels. Stated in dictionary §1.1.
• The blocked lane is growing. It grew set-wide across the 2026-08-30, 2026-09-02 and 2026-09-06 sets. A blocked row keeps its last label and dates it (§3, §9), and row_count_by_domain_status in each manifest is the number to watch. Corrected 2026-09-13: this item first attributed the growth to paused re-crawls of blocked sites and said the share would keep rising until they resumed. That was wrong. Re-crawls of blocked sites are not paused — a blocked domain becomes eligible for a re-attempt after 30 days. The growth comes from the nightly refresh reaching sites it last observed in May 2026, most of which no longer answer; between the 2026-09-06 and 2026-09-13 sets, nearly every p06 row that moved from active carried a status_observed_at from May 2026 before the move (your own copies of those two sets show this). Sites answering 304 Not Modified were also published as blocked (see the entry above). The rest is sites that did not answer the re-fetch, and real anti-bot answers. Corrected 2026-10-21: "eligible for a re-attempt after 30 days" is eligibility, not a schedule. No re-try schedule is promised, so a blocked row can keep its verdict for months. row_count_by_status_age in each manifest shows how old the verdicts are (dictionary §4).

2026-09-10 — 1.14.0: six new columns, a new policy pack, a derived band, and a smaller labelled population

feed_schema_version 1.13.2 → 1.14.0, a MINOR bump. Ships with the first ≥ 1.14.0 export — the Thursday 2026-09-17 snapshot set (exporter deploy Thursday 2026-09-17, after that day's 10:00 UTC delta); nothing published before that date carries any of this. Ingest a full snapshot at 1.14.0 before relying on 1.14.0 deltas: three of the changes below are re-derivations that touch every row, and a delta cannot carry a row that did not move.

Six appended columns (39–44)

Columns 1–38 are unchanged in name, position and type — the append-only contract, same precedent as 1.6.0 and 1.12.0. A reader that selects by name or by position is unaffected by the append itself.

The brand-safety axis is documented in data dictionary §13, columns 42–43 in §11 Website Categorization bulk feed data dictionary.

policy_basis gains the value safety — a breaking value addition

The feed's policy pack moves from policy_v1 to policy_v2. The new pack reads the safety axis first, then site_purpose, then the primary tier path, and policy_basis (column 37) says which answered. Its documented values are therefore primary, site_purpose and now safety. role is reserved for a later release and is not emitted at 1.14.0.

If you validate column 37 against a strict enum, add safety before ingesting. It is confined to an opt-in block, and it is why the pack change ships via a full snapshot rather than a delta: policy_pack.version in manifest.json moves with it, and a delta whose pack differs from its predecessor is a chain gap the exporter refuses to write.

Because the safety axis is read first, some flagged rows now carry the theme's group where they previously carried their label's — a gambling-flagged row labelled Sensitive Topics reads gambling rather than sensitive_other. Nothing about the label, the confidence or the calibrated block moves with it.

policy_group: news is retired, current_events replaces it

policy_v2 also renames one group, so this is a breaking value change on column 36, not only on column 37. The three current-affairs roots — Crime, Disasters and War and Conflicts — published as news under policy_v1; from 1.14.0 they publish as current_events. No other path changes group, and the group count is unchanged at 47.

If you filter or validate policy_group against a fixed list or an enum, replace news with current_events before ingesting the first 1.14.0 snapshot — from that artifact on, a policy keyed on news matches nothing. policy_group_set (column 43) carries the new id too, and so does row_count_by_policy_group in manifest.json.

The rename is why policy_v2 is a new pack id rather than a mapping bump: a group id never changes meaning inside a pack, so the old id was retired with its pack. The old id was misleading — IAB v3.1 has no news-publisher node, so news never contained newspapers or news portals, and a policy that allowed or blocked it allowed or blocked no news site. current_events says what the group actually is; there is still no group that contains news publishers. Full definition, the per-group table and the machine-readable diff: website-categorization-policy-pack-v2.md.

confidence_tier is derived (band policy v2)

Column 8 is derived at export from the writer (model_name), the label (tier1) and the calibrated tier (calibrated_confidence_tier), instead of being the stored 0.9 / 0.7 cut passed through. The value domain is unchanged — HIGH / MEDIUM / LOW, plus PARKED. manifest.json carries confidence_band_policy: "v2", and a manifest that does not carry the key at all is v1. The notice below carries the per-family detail and the recipe for rebuilding the old band from columns you already receive; read it before the first 1.14.0 artifact lands.

Fewer rows carry a category: the language-NULL guard is now armed

This is the change that makes 1.14.0 a MINOR rather than a PATCH. Rows written by the ensemble family that carry a category with no page language recorded, behind a captured content hash, now ship status-only: the category block is suppressed and no last_known_* projection is made. Those labels were produced from content we cannot show was in a language the model was fitted for, and a judged read put their agreement low enough that suppression is the correct answer rather than a re-score.

manifest.json carries a language_null_suppression block from 1.14.0 on, counting the rows this export changed and the three exempt populations (block-critical groups, the popularity head, and rows whose group could not be resolved) — those are not suppressed. Rule lanes are untouched. Expect the labelled population of a partition to fall; the manifest block of your own delivery is the authority on how much.

New manifest keys

Every export now publishes the distributions a consumer previously had to scan for, at partition, delta AND run level: row_count_by_confidence_tier, row_count_by_writer_family_x_tier, row_count_by_policy_group (every pack group present, 0 where empty), row_count_policy_group_uncalibrated, row_count_by_policy_group_by_writer_family, row_count_by_safety_status, row_count_by_safety_flag and row_count_by_language_primary. The four coverage counters that existed per partition (row_count_with_policy_group, row_count_calibrated, row_count_by_site_purpose_status, row_count_with_last_known_category) are now summed in the run-level run_manifest.json too, so a snapshot's totals no longer require adding up sixteen files.

Deltas additionally carry row_count_tier1_changed, top_tier1_transitions and tier1_transitions_window — a planned re-score announces itself there. A delta's row_count says how many rows moved; those keys say whether they moved between categories, and which moves were the largest.

manifest_keys_version is new and names the KEY SET's own version, independent of feed_schema_version. Pin it if what you parse is the manifest rather than the columns. Every key is described in data dictionary §10 Website Categorization bulk feed data dictionary.

2026-09-10 — the 1.14.0 notice: what confidence_tier will say, and how to rebuild the old band

Nothing published changes with this entry. It is the customer notice for the release above: which band each writer family can reach, which rows move, and the recipe for rebuilding the band you have today. Read it now, not on the day the snapshot lands.

Why this entry is dated where it is. The first artifact carrying any of this is the Thursday 2026-09-17 snapshot set. The exporter that produces it deploys on Thursday 2026-09-17, after that day's 10:00 UTC delta, and everything described here is frozen a week before that. This notice is therefore dated before the first 1.14.0 artifact exists — deliberately, because the band is derived rather than tracked and there is no changed-row marker that would tell a loader to look for it. If you are reading this after 2026-09-17, the change has already shipped.

Amended 2026-09-11, before any 1.14.0 artifact exists. The three brand-safety filter families are written under five model_name values: pipeline/st_filter, pipeline/gambling_filter, pipeline/adult_filter and the two cascade twins cascade/st_filter and cascade/gambling_filter, whose confidence is the same non-probability as their pipeline/ spelling. The filter rule below covers all five. A human-feedback row — a reviewer's accepted verdict, written at a fixed confidence of 1.0 — reads policy_confidence_basis = curated, not model_raw, and keeps the ordinary 0.9 / 0.7 cut, because no rule of the policy is keyed on that writer.

What changes in column 8

confidence_tier stops being the stored 0.9 / 0.7 cut passed through verbatim and becomes a band derived at export from the writer (model_name), the label (tier1) and the calibrated tier (calibrated_confidence_tier). The per-family table — what band each model_name family can reach — is in data dictionary §5 Website Categorization bulk feed data dictionary. In one sentence: rows written by the three brand-safety filters band LOW; ensemble rows labelled Business and Finance never band HIGH; and an ensemble row the cut put in LOW bands MEDIUM when its calibrated tier is TOSSUP or PROBABLE.

The value domain of column 8 is unchanged — HIGH / MEDIUM / LOW, plus PARKED. No word is added, none is removed, and a PARKED or NULL band passes through untouched. confidence, the calibrated_* block, tier1–tier4, iab_id, model_name, policy_group and every other column carry exactly what they would have carried under the old cut.

Which rows move, and in which direction

All three rules move rows, and the one with the clearest consequence for a block list — a brand-safety filter's confidence is a rule score, so those rows can no longer band HIGH — is the smallest of the three by row count, not the largest.

This is not a loss of rows, labels or groups. Every row still ships, under all three rules, with the same tier1–tier4, the same iab_id, the same policy_group and the same model_name. What changes is the one column that never carried the information a blocker needs: on the first rule's rows the old HIGH meant "a keyword rule fired hard", never "this label is 90% likely to be right". If your block list is keyed on confidence_tier, it will lose those rows — which is why the data dictionary says, in as many words, to key on policy_group first and to read its ranking of the other keys before relying on them. Read your own delivery's manifest counts; the per-partition figures behind this entry are not published here, and one partition's composition is not the feed's.

Rebuilding the policy-v1 band client-side

confidence (column 7) is untouched, so the band you have today is reconstructible on every row. Evaluate it exactly as the export does, or you will disagree with us on the boundary:

-- The policy-v1 band, on any row whose confidence_tier is not PARKED and not NULL.
CASE
  WHEN round(confidence::numeric, 4) >= 0.9 THEN 'HIGH'
  WHEN round(confidence::numeric, 4) >= 0.7 THEN 'MEDIUM'
  ELSE 'LOW'
END

Round to four decimals first — both cut points need it. The pipeline produces confidence rounded to four decimals and the Parquet column is a 32-bit float, so the decoded value is the nearest float32 rather than the decimal you were shown. Both boundary values fall below their cut: 0.9 decodes as 0.899999976… and 0.7 as 0.699999988…, so an unrounded >= 0.9 silently bands every exact-0.9 row MEDIUM and an unrounded >= 0.7 bands every exact-0.7 row LOW. The csv.gz cell is already written at four decimals and needs no rounding; the Parquet column does.

Two rows this recipe deliberately does not touch: a PARKED band and a NULL band are not cut points and are passed through under both policies.

The transition rule: the first 1.14.0 set is a full replace

The band is derived, not tracked: a row whose band moves under the new policy is not a changed row and does not cross a delta cutoff. The first ≥ 1.14.0 artifact you ingest must be a snapshot — the full sixteen-partition set — or your table will carry two bands for one row set with nothing marking which is which. manifest.json carries confidence_band_policy so you can tell them apart: v1 (or the key's absence) is the pass-through, v2 is the derived band. The same release also makes the language-NULL ensemble class ship status-only, which moves the published row population. One full per-partition replace satisfies both.

The brand-safety columns ride the same release

Columns 39–41 — safety_status, safety_flags, safety_detail — ship on the first 1.14.0 artifact. One property of them is worth knowing, because it decides what they are evidence of. A domain's per-filter verdicts are stored from the 1.14.0 release onwards. Rows whose last write predates it are derived from the row's existing model_name and tier2 and carry a legacy version marker in safety_detail. That derivation is deliberate — without it every pre-existing filter row would report not_assessed and a blocker keyed on the new columns would under-block on day one — but on those rows it is a re-statement of what column 10 already told you, not new evidence. Values written from a real verdict are distinguishable by their version marker.

2026-09-09 — ahead of 1.14.0: the transition rule goes on record

Nothing published changes with this entry. It puts the transition rule for the next MINOR release, 1.14.0, on record before the artifact that needs it exists: column 8 becomes a derived band, the first ≥ 1.14.0 artifact you ingest must be a full sixteen-partition snapshot, and manifest.json gains confidence_band_policy so the two policies are distinguishable ("v1", or the key's absence, is the pass-through every artifact to date has carried). The key can appear on a 1.13.x manifest ahead of the bump, always reading "v1"; under the extensible-manifest rule that carries no change to the artifact. Blockers should key on policy_group and model_name, never on the band. The full notice is the entry above.

2026-09-06 — 1.13.2: curated rows on every delta (no column change)

feed_schema_version 1.13.1 → 1.13.2, a PATCH bump. Ships with the first ≥ 1.13.2 export — and, unlike 1.13.1, that includes the first daily delta after the exporter deploy, not only the first Sunday export after it.

Every delta now carries the whole hand-vetted curated list — the curated/* rows the real-time API answers from — and not only the exclusion that already kept a competing model-produced row out. Until now a curated row shipped in snapshots alone, so a consumer building its table from deltas kept the old model guess for a curated domain until the next weekly snapshot. Those are the most-visible domains in the feed, so the divergence sat exactly where it was least acceptable. Snapshot artifacts are unchanged.

A daily delta exported between the deploy and the Sunday snapshot carries 1.13.2 too — both kinds are stamped from the same constant — and here that delta is already complete for this change: the curated rows are not keyed on a cutoff, so unlike 1.13.1's masked rows they do cross the delta boundary and arrive with the first ≥ 1.13.2 delta. No snapshot is needed to receive them. The snapshot-ingest obligation is unchanged for every other reason: removals and the status-only baseline still travel only in a full per-partition replace.

What else changes for you:

• The whole list rides every delta, not a changed subset. The curated list is a file, not a table, so there is no cutoff to narrow it by. The same rows arrive in every delta; under the upsert-by-domain a delta already requires, re-applying them is a no-op.
• Apply curated rows by domain, never by classified_at ordering. A curated row is dated when the entry was curated, so it is routinely older than the model-produced row it replaces. A loader that keeps the newest classified_at per domain will discard the curated answer — match on domain and take the row from the most recent artifact. This was already true of snapshots; deltas make it matter daily.
• curated_lane in a delta's manifest.json now reads true (it read false before), and row_count_curated reports how many curated rows that delta carried rather than always 0. Both keys already existed; neither is given a new meaning.
• A delta's row_count is no longer 0 on a quiet night — it is at least the curated count. To ask "did anything change in the source?", read row_count - row_count_curated.

2026-09-05 — 1.13.1: parked-flagged brand-safety filter rows ship status-only (no column change)

feed_schema_version 1.13.0 → 1.13.1, the feed's first PATCH bump: the published row population changes, no column is added, removed, renamed or retyped, and no manifest key is removed. Ships with the first ≥ 1.13.1 snapshot set — the first Sunday export after the exporter deploy.

Parked-flagged brand-safety filter labels — pipeline/st_filter, pipeline/gambling_filter, pipeline/adult_filter and pipeline/vendor_rule — now ship status-only, matching the treatment parked-branded ensemble-family labels have had since 1.11.0. Parked-flagged rows outside both those families (cascade/st_filter, for one) are not touched by this guard at all. Affected rows publish site_purpose_status = undefined with NULL category columns and an empty last_known_* block; the domain's status lane is unchanged. Until now such a row asserted a current category and "no current content" at the same time.

Those rows sit on the category-keeping lanes — the classifier's parked flag disagreeing with the crawl-observed lane is exactly why they still carried a category — so filtering domain_status = 'parked' will not find them. Read your own delivery's manifest for the count on each partition. A second population, already on the parked / dead / inactive lanes, loses its last_known_* historical label for the same reason.

Some parked domains keep an adult or gambling verdict on purpose. A row whose label projects to a block-critical policy group — adult, drugs, gambling, gambling_games, harmful_content, hate, piracy, weapons — is deliberately retained rather than suppressed, and ships exactly as it does today, until an in-group accuracy read clears the hold. What that looks like depends on the lane: on the category-keeping lanes (active, blocked, uncategorizable) the row keeps its category and confidence_tier = 'PARKED'; on the parked / dead / inactive lanes, where the lane rules already blank the current category, the hold instead keeps the row's last_known_* history intact. The hold does not override the older guards: a row one of the sibling suppression blocks already owns — parked-stub, no-evidence or boilerplate-template — is suppressed by that guard as before, block-critical group or not. Removing a brand-safety block is the change that can hurt you, so it is not made on an unmeasured population.

The demoted_label_suppression manifest block gains filter_model_names (the four families above), block_critical_held (rows the hold retained — counted there, never in row_count_overridden), block_critical_groups (the groups actually held in that run), block_critical_group_policy (the eight-group rule), unresolved_group_held (rows held because their policy group could not be resolved) and policy_group_delta (per-group before / after counts). No 1.13.0 manifest key was removed.

Same transition rule as the 1.8.0 / 1.10.0 / 1.11.0 guards: consume the first ≥ 1.13.1 set as a full per-partition replace. Rows this build newly masks do not cross the delta cutoffs, so a delta applied on a 1.13.0 base keeps categories 1.13.1 withdrew.

2026-09-04 — eight corrections to the data dictionary (wording only, no schema or version change)

No column change, no manifest change, no feed_schema_version change and no published value moves. Eight statements in the data dictionary described the pipeline as it was configured at some earlier point rather than as it runs today. Each is now corrected against the running configuration, and each correction is pinned by a test so it cannot drift back:

• The sensitive-content filter fires at 0.50, not 0.30. 0.30 is a code default that has never been the deployed value; the real-time and bulk lanes both run the filter at 0.50.
• A page whose body text exceeds 1,500 characters is not scored by that filter at all. It is skipped, not truncated — so the absence of an */st_filter row is not evidence the filter cleared the page.
• The filter has a video-streaming bypass: a fire scored in [0.50, 0.65) on a video-streaming domain is discarded and the page falls through to normal classification. Those rows are not Sensitive Topics and do not project to the sensitive_other policy group, so both counts understate what the filter fired on.
• Eight model_name families ship, on seven scales. The counts differ because the vendor-rule flip republishes the trained classifier's own probability rather than scoring anything.
• v1+teacher_bf1 rows carry calibrated_* values. The 2026-08-13 refit (calib-map-v2) deliberately fitted that suffix in; the page still described every re-scoring suffix as out of scope. The map's allow-list today is: no suffix, ml_en, ml_non_en, nocontent_bf1, teacher_bf1.
• All five calibration ceilings are documented. Only en|rest and __global__ rows can reach PROBABLE; en|bf and nonen|rest are capped below it and nonen|bf well below. Previously only the nonen|bf ceiling was described, which read as if it were the only capped segment.
• domain_status = 'active' does not guarantee a category. A live site can ship with every content column NULL — never labelled, or a suppression guard withheld the label. Test tier1 IS NOT NULL.
• confidence_tier = 'PARKED' is reachable and now observed. The page said it had occurred zero times; a small number of rows carry it, all written by rule filters disagreeing with the crawl-observed lane. Parse for the token.

Two further wording changes in the same pass. The last_known_* coverage figure was measured on one partition and scaled into a feed-wide count; the population underneath it has grown, so the scaled figure is withdrawn and that section now tells you to read row_count_with_last_known_category and row_count_by_domain_status off your own delivery, which is exact and needs no scaling. And a number of cross-references inside the dictionary pointed at section anchors that do not exist; all of them now resolve.

One API-side note, recorded here because the names collide: content_purpose (the legacy API field; not in the feed) is reserved and null in every current deployment — no shipped model emits it. It is unrelated to the feed's site_purpose axis (columns 27–35) and to the policy projection; no feed column has ever carried it.

2026-09-04 — how tier1..tier4 is defined, stated one way: purpose-first (wording only, no schema or version change)

No column change, no manifest change, no feed_schema_version change, and no published value moves — this entry exists only because the definition of the stored label was written two different ways and is now written one way.

The stored label follows the ratified single-label convention, and that convention is purpose-first: it names the site's primary purpose — what the site is for — rather than its topic, industry or audience, with a small set of rules settling the recurring cases: a gambling home is Attractions > Casinos & Gambling and the harmful subset ships as Sensitive Topics; software, SaaS and hosting vendors are Technology & Computing whoever they sell to; a multi-brand store is Shopping while a brand's own store takes its vertical; consumer banking, BNPL, investing and pensions are Personal Finance; a content platform takes the content type it carries, not its delivery mechanism. The data dictionary previously described the same convention as "about-first, plus a small set of purpose conventions", which stated the base principle backwards; the convention itself is unchanged since v1.1 (2026-08-25).

Nothing you ingest changes because of this entry: no row is relabelled by it, no threshold moves, and the reserved site_purpose_* columns (27–35) stay reserved — eight NULL, site_purpose_status never NULL. If a purpose pass is ever run, it is announced here before it ships.

2026-09-03 — 1.13.0: four optional manifest keys for a stricter coverage denominator (no column change)

feed_schema_version 1.12.0 → 1.13.0. No column changes and no change to any existing manifest key — the version covers manifest keys too, so keys alone bump it. Partition and run-level manifests may now also carry row_count_dead_status_by_rank_bucket, inventory_dead_status_by_rank_bucket, inventory_never_shippable_by_rank_bucket and coverage_shippable_by_rank_bucket. The existing coverage_by_rank_bucket counts every inventory key in its denominator, including keys whose crawl status is in our dead set (in practice not_found, 404 / NXDOMAIN) and that carry no category row — keys that cannot produce a feed row until a re-crawl succeeds. The new coverage_shippable_by_rank_bucket takes those out. Both are published; nothing you read today moves. The keys appear from the first artifact published by a 1.13.0 image and only when the inventory scan that already feeds inventory_total_by_rank_bucket ran (snapshots; deltas never carry the scan). A 1.13.0 delta on a 1.12.0 snapshot is safe to ingest: the rows are identical.

2026-09-03 — policy_v1 mapping 2026.09.2: casino video games move to gambling_games

Same columns, different published values for one IAB node. Pack policy_v1 moves from mapping version 2026.09.1 to 2026.09.2: Video Gaming > Video Game Genres > Casino and Gambling Video Games (simulated gambling, no real-money play) now projects to a new group, gambling_games, instead of gambling; the three real-money nodes (casinos, lotteries, poker) stay in gambling, and the group count is 47. Every vendor definition, regulator, rating body and app store we checked draws the line at real money, and two groups can be unioned by you (or merged by us later) while a merged group can never be un-merged. Block gambling and gambling_games to keep the previous behaviour — including real-money casinos the classifier may have stored under the video-game genre, which is a stored-label error the pack cannot correct.

Takes effect in the feed with the first full snapshot published from an image that bundles 2026.09.2 — a pack change never rides a delta, so between that image's deploy and that snapshot no delta is published, and the first delta after it covers the gap. From that snapshot on, policy_pack_version on every row and policy_pack.version in the manifest read 2026.09.2. Group definitions: website-categorization-policy-pack-v1.md.

2026-09-02 — the ensemble tokenizer repair and the re-categorization of English-routed rows (no schema change)

No column changes. The values on a large set of rows change, and their confidence scale moves.

What was wrong. Until 2026-08-29 the production ensemble fed three of its five English-language members tokens produced by another member's tokenizer, so those three ran near chance and the ensemble's tier-1 accuracy sat measurably below its design. Every English-routed label written before the fix carries that deficit; the non-English route was unaffected.

What was re-scored. From 2026-09-01 the affected rows that have stored page text were re-scored with the repaired ensemble: English-routed rows plus rows with no recorded page language that nevertheless carry text. Rows with no stored text at all were not re-scored — most already ship without a category under the no-evidence guard (1.8.0). The pass covered the rows that had stored text when its corpus was assembled in late August 2026, and completed on 2026-09-03.

What you will see.

• Label changes on a minority of re-scored rows, concentrated in rows the old ensemble scored LOW. The most common move is between Technology & Computing and Business and Finance.
• A confidence-scale shift on re-scored rows: the repaired ensemble is more decisive, so mean raw confidence rises and the LOW share falls. A re-scored row's confidence is therefore not comparable with its own pre-repair value, and confidence_tier mixes are not comparable across the repair boundary.
• calibrated_confidence and calibrated_confidence_tier on re-scored rows still come from the map fitted on 2026-08-13 (confidence_calibration_map_version unchanged), i.e. on the pre-repair scale. Checked against a blind-judged cohort the map is not over-confident on the new scale — it understates LOW rows. A refit is planned and will be announced here with a new confidence_calibration_map_version.
• classified_at moves to the re-score date; model_name (ensemble-5en-3nonen) and model_version (v1+ml_en / v1+ml_non_en) are unchanged, so the provenance columns alone do not distinguish a re-scored row — use classified_at >= 2026-09-01.
• The vendor-rule flip (next entry) was not applied during this pass.

Which artifacts. The off-cycle snapshot dated 2026-09-02 carries the rows re-scored before each of its partitions was exported; the 2026-09-06 snapshot carries the complete pass; daily deltas carry re-scored rows as they land, so the delta covering the pass is substantially larger than a normal night's.

2026-09-02 — two model_name families the dictionary did not yet list: pipeline/adult_filter and the new pipeline/vendor_rule flip (no schema change)

No column changes. This entry documents one component that has shipped since artifact 1.10.0 and one that is new, and the data dictionary's model_name family count moves from six to eight.

pipeline/adult_filter (artifact ≥ 1.10.0). A multilingual rule layer that overrides a benign-looking label when adult-platform markers, explicit title terms or adult-keyword density identify an adult site, always to Sensitive Topics > Adult & Explicit Sexual Content. Its confidence is not a probability — a fixed 0.95 or 0.90, else a density ratio capped at 1.0, range [0.5, 1.0] — and its rows carry NULL calibrated_*. Nothing about these rows changes with this entry; the dictionary now lists them.

pipeline/vendor_rule (rows from 2026-09-01). The trained classifier over-generalizes software, SaaS, hosting, IT-services and telecom vendor sites into Business and Finance. A deterministic post-classifier now rewrites such a row to tier1 = Technology & Computing (tier2–tier4 NULL, iab_id = IAB-596) when the classifier's label sits in one of three vendor-heavy tier-3 cells — Business I.T., Technology Industry, Telecommunications Industry — and the page text carries high-precision vendor-identity signals. It is one-directional (it can remove a Business and Finance label from those cells, never add one), keeps the classifier's confidence, confidence_tier and model_version unchanged, and ships NULL calibrated_*. Filter on model_name = 'pipeline/vendor_rule' to see every flipped row; policy_group follows the new tier path. The rule was measured against a reserved holdout before it was enabled.

Where it applies: the real-time API has run it since 2026-09-01 23:00 UTC. The bulk categorization lanes do not apply it yet — this page will say when they do — and the September re-categorization pass (previous entry) ran without it. Expect these rows to be absent or rare in early-September artifacts, and the vendor over-generalization to persist on bulk-labeled rows until then.

2026-09-01 — twelve columns appended: the reserved site_purpose_* block and the policy_v1 projection (schema 1.12.0)

Ships with the first ≥ 1.12.0 snapshot set. Twelve new columns, appended after column 26 in one bump; no existing column changes in meaning, position or value, so a consumer reading by name or by position is unaffected. Delta files carry all 38 columns.

Columns 27–35, site_purpose_tier1 … site_purpose_model_version, are reserved for a second axis — what a site is for, in the same IAB v3.1 vocabulary — and are NULL on every row at 1.12.0, with one exception: site_purpose_status (column 33) is never NULL. It reads not_available on every row that has not been assessed for a purpose, which at 1.12.0 is every live row, and undefined on rows that have no separate business purpose — the parked / inactive / dead lanes and rows the parked detectors flagged — derived from the same status evidence as domain_status, never from a model. labeled appears only once a purpose pass runs (a later release; the manifest's site_purpose_model is null until then). Reserving the columns now means one schema bump instead of two and lets you build your table once. Read not_available as "not assessed", not as "no purpose".

Columns 36–38 are the policy_v1 policy-pack projection: policy_group, a fixed, versioned rollup of the 704 IAB categories onto filtering-ready groups (adult, gambling, piracy, drugs, weapons, hate, shopping, social_networking, …) in the vocabulary DNS-filter, secure-web-gateway and endpoint-agent policies already use; policy_basis, the axis the group was projected from (primary at 1.12.0; site_purpose once that axis is labeled); and policy_pack_version (policy_v1@2026.09.1, constant within an artifact, carried per row so a CSV row is self-describing). The manifest gains policy_pack: {id, version, sha256}, row_count_with_policy_group, site_purpose_model and row_count_by_site_purpose_status.

What the projection is: a pure function of the stored label. Attractions > Casinos & Gambling, Shopping > Lotteries and Scratchcards, Sports > Poker and Professional Gambling and the casino video-game genre all projected to gambling at this mapping version, so a blocking integration no longer has to know that IAB scatters gambling across four branches. What it is not: a correction of the stored label. A site whose stored label is wrong projects to the wrong group (an adult site stored as Entertainment > Movies reads entertainment); per-group accuracy has not been measured and no accuracy claim attaches to any group. The pack cannot express what IAB v3.1 has no node for — there is no malware, phishing, botnet or anonymizer group; threat intelligence is a separate product.

The three policy columns are NULL exactly where tier1 is NULL (every suppressed row) and where the tier path does not resolve; none of the new columns is carried into the last_known_* block. The policy columns are derived, not tracked: a pack version change is delivered by the next full snapshot, never by a delta, because it touches every row. The first ≥ 1.12.0 artifact you ingest must therefore be a snapshot. Pin policy_pack.version in manifest.json (or the per-row policy_pack_version). Group definitions and the open mapping decisions: website-categorization-policy-pack-v1.md.

2026-08-31 — NULL-category semantics: the demoted-label guard (schema 1.11.0)

Ships with the first ≥ 1.11.0 snapshot set. Ensemble-family rows whose label the write side branded parked-demoted — since 2026-08-22 the pipeline stamps is_parked on the category row when its classify-time placeholder detector fires, and the crawl's own parked verdict on the record being classified carries the same flag — now ship status-only: category columns NULL, no last_known_* block, domain_status unchanged. The manifest gains a demoted_label_suppression block with the count.

This closes the residual the 1.10.0 vetted-hash guard could not reach: the detector generalizes across template variants while the hash list is exact-match, so a demoted row whose template is not on the vetted list previously shipped its fabricated category — most visibly as placeholder Technology & Computing > Computing labels. Rows the hash guards already suppress are billed to those blocks; the suppression counts are disjoint. Consume the first ≥ 1.11.0 set as a full per-partition replace — a delta stream cannot retro-apply the mask (same rule as the 1.8.0/1.10.0 guards). The count trends toward 0 as re-crawls under the fixed detector replace the demoted labels' ground truth.

2026-08-24 — NULL-category semantics: the boilerplate-template guard (schema 1.10.0)

Ships with the first ≥ 1.10.0 snapshot set. Ensemble-family rows whose category was produced from a known boilerplate template — the crawl captured a hosting provider's parking/coming-soon page, a stock web-server or panel default, or another vetted template rather than owner content, identified by content_hash membership on a vetted list — now ship status-only: category columns NULL, no last_known_* block, domain_status unchanged. The manifest's boilerplate_template_suppression block publishes the full hash list beside the count. At ship time the list is 60 vetted templates, concentrated in but not limited to the former Web Hosting tier-4 population: the crawl read the HOST's boilerplate, so the label described the hosting provider, not the domain.

Consume the first ≥ 1.10.0 set as a full per-partition replace — a delta stream cannot retro-apply the mask (same rule as the 1.8.0 guard). The count trends toward 0 as re-crawls land: a re-crawl that captures real owner content changes the row's content_hash and releases it.

Context for the Web Hosting category specifically: independent of this guard, a graded audit found the great majority of that tier-4 class to be placeholder inflation. This guard masks the template-provable share; the residual inflation drains through re-crawls under the fixed categorize-time detector. Until then, treat low-confidence Web Hosting leaf labels per the calibrated-band guidance — the class runs DOUBTFUL-heavy.

2026-08-15 — deltas carry status-only changes (branch 3, schema 1.9.0)

A delta now includes status-only rows — inventory rows with no category row — whose inventory moved since the cutoff, keyed on the same crawl_attempted_at cursor as the domain_status branch. Previously these parked/placeholder rows changed silently between snapshots. Purely additive: no column changes, snapshots unchanged, and the rows look like any other status-only row (NULL category columns). The manifest's status_only_lane key is now part of a delta's identity — re-publishing the same cutoff with the lane toggled is refused. The status-only BASELINE still arrives only with a snapshot, and removals still require the full per-partition replace — the snapshot-ingest obligation stands.

2026-08-15 — NULL-category semantics: the no-evidence guard (schema 1.8.0)

Ships with the first ≥ 1.8.0 snapshot set. Ensemble-family rows whose category was produced with no captured page content (content_hash NULL, with the language guard conjunct) now ship status-only: category columns NULL, no last_known_* block, domain_status unchanged. The manifest's no_evidence_suppression block counts the suppressed rows. Consume the first ≥ 1.8.0 set as a full per-partition replace — a delta stream cannot retro-apply the mask to rows that do not otherwise change. The count trends toward 0 as re-crawls land.

2026-08-13 — confidence calibration map calib-map-v1 → calib-map-v2

The map behind calibrated_confidence, calibrated_confidence_tier and calibration_segment was refitted. The manifest's confidence_calibration_map_version moves from calib-map-v1 to calib-map-v2. Raw columns are untouched. Three things change:

1. The metric changed, not just the numbers. v1 estimated P(label correct) judged against stored crawl-time page text. v2 estimates P(label correct) judged against the page as fetched at judge time (blind LLM judge, exact tier-1 agreement). This is the stricter, buyer's-eye question: content drift since the crawl and domains that have since been parked now count against the label. Calibrated values drop substantially as a result — this is a semantics change, not a data regression. If you pin confidence_calibration_map_version, treat v1 and v2 values as incomparable.
2. Teacher-rescored rows are now calibrated. Rows whose model_version carries the teacher_bf1 suffix (non-English Business-and-Finance re-judgments) previously shipped NULL calibrated_* by design, because v1's curves were fitted on the pre-rescore confidence scale. v2's fit sampled that population directly; those rows now publish calibrated values. row_count_calibrated in the manifest rises accordingly.
3. calibration_segment loses one value. NULL|rest no longer has its own curve — too few judged rows at fit time against the 50-row per-segment floor — so rows with no recorded language now publish __global__. No new segment values are introduced.

Fit provenance is embedded in the artifact's own fitted_on field: blind-judged rows across three stratified samples of the 2026-08-09 snapshot, design-weighted to the calibrated population.