Website Categorization Bulk Feed — Data Dictionary

Part 2 of 2. Sections 6 to 14 are on this page. Sections 1 to 5 are on Data dictionary, part 1.

6. openrtb_cat — an array column

openrtb_cat holds the OpenRTB / AdCOM raw category ids for the row's tier path, ordered root → leaf. The taxonomy those ids belong to is published once per export as cattax in manifest.json (value 9 = IAB Content Taxonomy 3.1).

This column is array-valued, and its physical representation differs by file format:

Integration warning — this will surprise your loader. No comparable categorization feed on the market ships an array-valued column, so most CSV pipelines are not built to expect one. In the CSV the cell contains a JSON document that must be parsed, not split on a delimiter, and the standard CSV quoting rules apply to the embedded quotes. Budget for this explicitly.

openrtb_cat is NULL — not an empty array — whenever the tier path does not resolve to ids. An empty array would read as "resolved to zero categories", which is a different statement.

7. Nulls and encoding

Absence is always null, never a sentinel. No column uses a placeholder to stand in for a missing value: you will not find "N/A", "Unknown", -1, or an empty string meaning "we don't know". If a fact is absent, the field is null.

language (column 14) is the one column whose value comes from the page itself: the crawled page's lang attribute, passed through unchanged. A page that declares lang="null" therefore ships the four-character string null. That is the page's word, not our placeholder — treat any such token as "no usable language". The 1.14.0 language-NULL guard (§10) keys on a genuinely absent value and leaves such rows alone.

One value looks like a sentinel and is not. confidence_tier can carry the non-ordinal token PARKED (see §5). That is a positive statement from the classifier, not a stand-in for a missing confidence — where confidence is genuinely absent, confidence_tier is null like everything else.

Columns 45–46 are the plainest case of null meaning absent. content_purpose_id is null on every row outside a curated list of known news publishers — which is nearly every row, news sites included — so a null there is unknown, never "not news" (§14).

Other encoding notes:

• Files are UTF-8. The CSV is RFC 4180 quoted.
• From 1.14.0 (first artifact the 2026-09-17 set), domain is lowercase ASCII, with internationalised names in punycode (xn--): a source row whose name arrives in Unicode form is withheld by the exporter rather than shipped beside its punycode twin, and the withheld count is published in the manifest as row_count_excluded_non_ascii_domain. Artifacts before 1.14.0 still carry those rows, most of them beside the same name in punycode, so one registrable domain appears twice. To apply the 1.14.0 rule to an earlier artifact, drop every row whose domain is not ASCII.
• All timestamps are ISO-8601 with a UTC offset, with or without six fractional digits: classified_at reads 2026-09-09T18:46:49.941343+00:00 on model-written rows and 2026-05-08T00:00:00+00:00 on curated rows, while status_observed_at carries the crawl's own precision on every lane. Parse every timestamp column with a format that accepts an optional fraction. They are transported as strings in both formats — including in Parquet — for portability across loaders.
• confidence and calibrated_confidence are written with 4 decimal places in CSV so they match the Parquet float32 value exactly.

8. The last_known_* columns

Columns 18–23 carry a historical content label: one this domain held at an earlier point, which is no longer presented as current.

They are populated only on parked, inactive and dead rows — never on active (where the label is current, in columns 2–6) and never on blocked.

Coverage: sparse, and read it off your own delivery

This is a sparse column block, and how sparse it is moves between exports: on the large majority of non-live rows it is empty, and the non-live population per partition grows over time, so a figure measured on one export is out of date on the next.

Read coverage off the delivery you actually received. Every export publishes its own row_count_with_last_known_category and row_count_by_domain_status in manifest.json. That pair is exact for the artifact in your hands and needs no scaling; a figure quoted in documentation never will be, which is why none is quoted here.

The reason coverage is low at all is structural, not a gap we intend to close: the great majority of non-live domains were already parked, inactive or dead the first time we crawled them, so there was never a live label to retain. A historical label exists only where we observed the domain live, labelled it, and later observed it non-live.

Do not treat this as a parked-wide feature. If your integration depends on knowing what a parked domain used to be, it will be unpopulated on the overwhelming majority of rows — verify against your delivery's own count before building on it.

Scope of a partition measurement, stated precisely. Both feed tables are HASH(domain) on the same 16-way modulus, so domains are distributed uniformly across partitions, but the lane mix is not: in the 2026-10-18 set the blocked and dead shares and the last_known_* count differed noticeably from one partition to the next. Scaling one partition by sixteen gives an estimate, and only for the export it was taken from. The count grows over time as more domains are categorized while live and later observed non-live, so a scaled figure ages the moment the next snapshot is built. Sum the sixteen partition manifests of the delivery in front of you (or read the run-level run_manifest.json, §10) when the exact feed-wide number matters.

Where a last_known_* value is present it is dated by last_known_classified_at and is a statement about the past. It is never a claim about the domain's current content or its current owner.

9. Why a non-active row has no category

On parked, inactive and dead rows the content columns are NULL deliberately.

A parked page, a coming-soon placeholder and a dead domain are not "sites about something" — running a content classifier over a parking lander would produce a label describing the parking service, not the domain. Our position is that the honest classification of a parked domain is domain_status = parked, dated by status_observed_at. Categorization vendors generally take the same approach.

Two consequences to encode in your integration:

1. A NULL category is not a claim that no category exists. It is a statement that we are not asserting one for this domain in this state.
2. Do not blend lanes when measuring accuracy. A content-accuracy figure computed over this feed is computed over active rows only. Every export states this explicitly — see below.

10. manifest.json

Every export directory contains a manifest. The keys most relevant to a consumer:

On accuracy figures: row_count is blended, so it is the wrong denominator. content_accuracy_denominator_row_count is published precisely so you do not have to know which lane is answerable — it is the active count, and non-active lanes are excluded from it by construction.

The run-level run_manifest.json — is the snapshot COMPLETE?

A full snapshot is delivered as sixteen partition directories (snapshot_<date>_p00/ … _p15/), each with its own manifest.json — which proves that one partition, not the set. Listing the prefixes for a date cannot tell you 14/16 from complete, and a fan-out interrupted near UTC midnight can even leave a recovered partition under the next day's date.

So each completed snapshot run also publishes one run-level marker:

exports/snapshot_<date>/run_manifest.json

It is written only after all sixteen partition manifests for that date are present and coherent (each parseable, matching its partition, with a positive row_count, and all on one feed_schema_version). It carries type (snapshot_run), date, feed_schema_version, partitions_expected (16), partitions (each partition's manifest_key and row_count), and row_count_total. From artifact 1.17.0 the sixteen must also have been tagged from one news-publisher list, which the marker names as content_purpose_list — {"sha256": …, "row_count": …}, without the partitions' path — so a set whose partitions were tagged from two lists gets no marker until it is re-published (§14).

From artifact 1.7.0 it also rolls the per-partition keys up to run level, when and only when every one of the sixteen partitions carries them: row_count_curated_total, rank_ladder, row_count_by_rank_bucket, inventory_total_by_rank_bucket, coverage_by_rank_bucket, row_count_dead_status_by_rank_bucket, inventory_dead_status_by_rank_bucket, inventory_never_shippable_by_rank_bucket and coverage_shippable_by_rank_bucket, and (artifact ≥ 1.14.0) the status-age keys — status_age_reference_date, row_count_by_status_age, stale_fraction_active_gt_90d (re-derived from the summed active map, never averaged) and, with a present ladder, row_count_by_status_age_by_rank_bucket — which roll up only when every partition aged its rows against the same reference date. A key missing from even one partition is omitted from the roll-up rather than summed over a subset — a distribution over 14 partitions is not a distribution over the corpus. These keys never affect whether the marker is written; the completeness contract above is unchanged.

From artifact 1.14.0 the marker also carries the churn block — tier1_transitions_window, tier1_transitions_since, tier1_transitions_reason, row_count_tier1_changed, top_tier1_transitions and policy_group_transitions — read with ONE query over our category-transition ledger for the interval between the previous snapshot's marker and this one, never assembled from the deltas published in between. It is telemetry on the same terms as the roll-up keys: it can never decide whether the marker is written, and a run that could not measure it publishes tier1_transitions_reason instead of a number.

tier1_transitions_window is the half-open interval the three counts cover — since is the PREVIOUS snapshot's marker created_at, until is this marker's — so two consecutive snapshots tile the timeline and no ledger event is counted by both. since is established once, on the run that first publishes the date, and a re-run reuses it (see below). That previous marker has to be the one this chain published for its own date: it must sit at that date's key, declare that date, and enumerate that date's partition manifest keys under the same prefix. An object that fails any of those (including a marker for a different snapshot restored to the key) bounds nothing — the counts are withheld and tier1_transitions_reason says predecessor marker unreadable, never a number over a window we could not establish. tier1_transitions_reason is null exactly when the three counts are measured; otherwise it names the case, and two of them (predecessor marker missing, no 1.14.0 predecessor) publish an honest 0 / [] / [] because the chain simply starts there.

Tiling needs the chain to be published in date order, and a recovery can break that: a snapshot date marked for the first time after a later date was already published closes its window at the moment it is marked, which is past the later snapshot's own created_at, so its interval would contain that snapshot's entirely. Such a run publishes a later snapshot's window is already published and withholds the counts — with tier1_transitions_since: null, so a re-run of it decides the same way — rather than announce churn you have already been given. The later snapshot's marker, its bounds and its counts are not touched, and the recovered date still gets its completeness marker. If a later date's marker cannot be read at all, the run says later snapshot marker unreadable and withholds for the same reason. That check walks forward one date at a time and is bounded at five weeks, so a recovery more than five weeks late runs out of candidates before it reaches the day its own window would close on; it then says successor scan exhausted before this run's upper bound and withholds too, because the days it could not ask about may hold a window you have already been given. All three withhold with tier1_transitions_since: null, so a re-run of the date decides afresh rather than inheriting a bound that would skip the check.

Use it as the completeness gate: treat a dated snapshot as complete iff its run_manifest.json exists, and read the sixteen manifest keys from it rather than globbing the prefix. Its absence on a current delivery means the set is still incomplete or being recovered — do not ingest a partial set. Snapshot dates published before this marker existed (up to and including 2026-08-02) have none; absence is only meaningful on deliveries after that.

The marker describes the set as of its created_at, which is the date's FIRST published stamp: a re-run retains it rather than re-stamping the clock, because it is also the churn window's upper bound and the next snapshot's lower bound. The window's own since is retained from that same marker for the mirror-image reason — it is the interval you were told these counts cover — so a re-run re-derives neither bound. Without that, a marker for a date inside the published interval appearing between two runs of the same snapshot (a re-published older date, an archive copy-back, a run recovered out of order) would move since forward on the re-run and shrink the counts beside it, with until unchanged and no reason given. Both values are only retained from an object that passes the same identity test the previous marker above has to pass — this date, and this date's partition manifest keys under the same prefix — so a marker for another set restored onto the key supplies neither bound; the run stamps its own clock, measures its own window, and the marker is still written. A re-run that cannot read that object back fails instead — the marker already on the key is left exactly as it is — so a published boundary never moves under a successor that has already been delivered. The retained since is read from tier1_transitions_since, which a marker carries whether or not it managed to measure over it — and whatever stopped it, a failed read, a timed-out one or no ledger configured on the run at all — so a run that could not measure does not release the bound and hand the run after it back to deriving one: a re-run publishes the bound it was handed, and a first run of a date publishes the one it had validated before its read failed or timed out (a first run with no ledger to read never locates a predecessor, so it publishes null); either way the interval survives the failure. A date whose marker carries tier1_transitions_since: null — no previous marker was located at all — has no bound to retain, so a later re-run of it measures normally. In the rare case that a date is re-published after its marker was written, the marker is rewritten when the re-publish completes; per-file integrity is always the sha256 values inside each partition's own manifest.json — verify your downloads against those regardless of the marker.

The snapshot-ingest obligation (contract term)

Deltas alone cannot keep you compliant. Removals — GDPR takedowns and suppressions — retract a row, and only a full per-partition replace expresses a retraction; the status-only lane's baseline is likewise snapshot-borne. (Curated rows are not: from artifact ≥ 1.13.2 they ride every delta as well — §5 — which changes nothing about this obligation, since neither removals nor the status-only baseline travel in a delta.) The licence therefore carries this term:

Licensee shall ingest each full snapshot within 14 days of publication, and shall apply it as a per-partition replace rather than a merge. Deltas alone do not convey removals; a Licensee consuming only deltas will retain records WhoisXML API has withdrawn.

The 14 days are set against the platform's 30-day end-to-end erasure target: a withdrawn record leaves the next weekly snapshot (that set starts within seven days and is usually complete the following Monday, UTC), and your 14-day ingest window puts the worst case at about 22 days — inside the target, with margin for our own publishing latency.

Replacing a set, and which partition a domain is in

Deltas are not partitioned: each one mixes rows from all sixteen partitions and carries no partition column. A full set withdraws a row by no longer carrying it, so replacing one partition at a time removes a withdrawn row only if you know which partition it belongs to — rows a delta added included. Two ways to apply a set:

• Replace all sixteen partitions of a set together. Load the complete set (its run_manifest.json exists) in place of everything you hold, delta-applied rows included, then resume with the deltas as §1.1 says.
• Or replace one partition at a time, first deleting every row you hold that routes to that partition, delta-applied rows included; once it is reloaded, re-apply that partition's rows from the delta §1.1 names and from every later one. A domain's partition is PostgreSQL's hash-partition routing of the domain value exactly as published:
  1. h = hashtextextended(domain, 0x7A5B22367996DCFD): PostgreSQL's 64-bit hash of the name's UTF-8 bytes under the seed every hash-partitioned table uses, read as an unsigned 64-bit integer;
  2. hash_combine64(0, h), which for a single key is (h + 0x49A0F4DD15E5A8E3) mod 2^64;
  3. that value mod 16, which is the NN of snapshot_<date>_pNN.

In PostgreSQL itself the partition is ((hashtextextended(domain, 8816678312871386365) & 15) + 3) & 15 for a text column under a deterministic collation (the default). Anywhere else, this standard-library Python reproduces it exactly:

def feed_partition(domain: str) -> int:
    """The partition (0-15, the NN of snapshot_<date>_pNN) that holds `domain`."""
    m32 = 0xFFFFFFFF

    def rot(x, k):
        return ((x << k) | (x >> (32 - k))) & m32

    def mix(a, b, c):
        a = ((a - c) & m32) ^ rot(c, 4)
        c = (c + b) & m32
        b = ((b - a) & m32) ^ rot(a, 6)
        a = (a + c) & m32
        c = ((c - b) & m32) ^ rot(b, 8)
        b = (b + a) & m32
        a = ((a - c) & m32) ^ rot(c, 16)
        c = (c + b) & m32
        b = ((b - a) & m32) ^ rot(a, 19)
        a = (a + c) & m32
        c = ((c - b) & m32) ^ rot(b, 4)
        b = (b + a) & m32
        return a, b, c

    def final(a, b, c):
        c = ((c ^ b) - rot(b, 14)) & m32
        a = ((a ^ c) - rot(c, 11)) & m32
        b = ((b ^ a) - rot(a, 25)) & m32
        c = ((c ^ b) - rot(b, 16)) & m32
        a = ((a ^ c) - rot(c, 4)) & m32
        b = ((b ^ a) - rot(a, 14)) & m32
        c = ((c ^ b) - rot(b, 24)) & m32
        return a, b, c

    # 1. hashtextextended(domain, 0x7A5B22367996DCFD): PostgreSQL's 64-bit lookup3.
    key = domain.encode("utf-8")
    seed = 0x7A5B22367996DCFD
    a = b = c = (0x9E3779B9 + 3923095 + len(key)) & m32
    a = (a + (seed >> 32)) & m32  # the seed is hashed first, as a 12-byte block
    b = (b + (seed & m32)) & m32
    a, b, c = mix(a, b, c)
    while len(key) >= 12:
        a = (a + int.from_bytes(key[0:4], "little")) & m32
        b = (b + int.from_bytes(key[4:8], "little")) & m32
        c = (c + int.from_bytes(key[8:12], "little")) & m32
        a, b, c = mix(a, b, c)
        key = key[12:]
    tail = key.ljust(11, b"\0")  # the last 0-11 bytes; the low byte of c is never written
    a = (a + int.from_bytes(tail[0:4], "little")) & m32
    b = (b + int.from_bytes(tail[4:8], "little")) & m32
    c = (c + (int.from_bytes(tail[8:11], "little") << 8)) & m32
    a, b, c = final(a, b, c)
    h = (b << 32) | c
    # 2. hash_combine64(0, h), then 3. mod 16.
    return ((h + 0x49A0F4DD15E5A8E3) & 0xFFFFFFFFFFFFFFFF) % 16

Check it on your own delivery before relying on it: every domain in snapshot_<date>_p07.parquet returns 7.

11. The reserved site_purpose_* block and the policy_group projection

Columns 27–38 (artifact ≥ 1.12.0) add two things: a reserved second axis and a projection over the label. Neither changes any existing column.

The reserved site_purpose_* block (columns 27–35)

The stored label (tier1..tier4) is the primary label under the purpose-first single-label convention (§2). The site_purpose_* columns are reserved for a separately modelled business-purpose answer, produced by a different model against the same taxonomy — nothing populates them today — in the same IAB v3.1 vocabulary (Shopping for a store, Attractions > Casinos & Gambling for a sportsbook), with its own confidence, its own provenance (site_purpose_model_version) and a site_purpose_relation to the primary label (same_path / same_tier1 / different_tier1).

At 1.12.0 no purpose pass has run: every one of these columns is NULL on every row except site_purpose_status, which is never NULL and tells you why the rest is empty:

The columns are reserved now so that one schema bump, not two, carries both features and you build your table once. row_count_by_site_purpose_status in the manifest shows the split without a scan. When a purpose pass lands, the purpose columns are tracked like the category columns (a purpose-label change alone produces a delta row).

These nine columns keep their names and positions: they will not be renamed or repurposed, under the §1 column-order contract. Eight stay NULL and site_purpose_status keeps its documented values until a purpose pass is announced in the Website Categorization feed changelog. A second axis, if one ever ships, arrives as new columns appended at the end of the then-current schema, never by changing what these hold.

policy_group, policy_basis, policy_pack_version (columns 36–38)

policy_group is the row's label rolled up into one of the pack's 47 policy groups — the filtering vocabulary (adult, gambling, piracy, drugs, weapons, hate, dating, social_networking, shopping, …) that DNS-filter, secure-web-gateway and endpoint policies already use. policy_basis says which axis it was projected from (primary = the tier path; site_purpose once that axis is labeled, which takes precedence; safety from artifact 1.14.0, which takes precedence over both). policy_pack_version is the pack and mapping version the row was projected with. The full group list, each group's definition and the IAB nodes it covers are in website-categorization-policy-pack-v1.md and, from artifact 1.14.0, website-categorization-policy-pack-v2.md.

Which pack, by artifact version. policy_v1@2026.09.2 at 1.12.0–1.13.x; policy_v2@2026.09.1 from 1.14.0. policy_v2 changes the projection in two ways, both of them visible in this column: the brand-safety axis is read before the label (§13), and the group news is retired in favour of current_events — the same three current-affairs roots (Crime, Disasters, War and Conflicts), an id that says what they are, and the reason the pack ID moved rather than the mapping version. A policy keyed on news matches nothing from 1.14.0 on; see the Website Categorization feed changelog. No other path changes group and the group count is unchanged.

Why it exists: IAB v3.1 scatters what a blocking policy treats as one thing. Gambling alone lives at Attractions > Casinos & Gambling, Shopping > Lotteries and Scratchcards, Sports > Poker and Professional Gambling and a Video Gaming genre — an integration that blocks the Sensitive Topics subtree blocks none of them. The three real-money nodes project to gambling and the video-game genre (simulated gambling, no real-money play) to gambling_games (mapping 2026.09.2); a policy is one equality test per group on this column.

How it is computed — and what that means for you:

• Through 1.13.x (policy_v1) it was a pure function of the stored label: the deepest rule matching the tier path won (a label stored as Attractions > Casinos & Gambling → gambling; one that stopped at Attractions → travel), it read nothing else about the domain, and policy_basis was primary on every projected row.
• From 1.14.0 (policy_v2) the label is no longer the only input. The projection reads the brand-safety axis first — the flagged themes in safety_flags (§13) — then site_purpose (still unlabeled, so unreachable today), then the tier path, and policy_basis names the axis that answered. It still never changes any other column. Three consequences a policy has to be written against:
  • A flagged row whose label sits in a non-sensitive group takes the theme's group: gambling flagged on a row labelled Automotive reads policy_group = gambling, policy_basis = safety. So policy_group need not correspond to tier1 at all, and an integration that re-derives the group from the tier path will disagree with the column on exactly these rows.
  • A row already in a block-critical group never leaves it. When the label resolves to a different sensitive group and no flagged theme names that group, the label answers and policy_basis reads primary; the verdict stays visible in safety_flags. The one exception is the theme-unknown verdict (sensitive, theme not identified), which is never more specific than a theme-specific one — an adult flag on a bare Sensitive Topics label reads adult / safety.
  • A flagged theme this pack maps to no group, and one it does not know, are ignored — the projection runs over every row and must be total.
• It cannot correct a wrong stored label. A domain whose category is wrong projects to the wrong group; an adult site stored as Entertainment > Movies reads entertainment. Per-group precision and recall have not been measured, and no accuracy figure attaches to any group. Evaluate on your own domains before blocking.
• It expresses only what IAB v3.1 has nodes for. There is no malware, phishing, botnet or proxy/anonymizer group, because the taxonomy has no such category; threat intelligence is a separate WhoisXML product.

• When the three columns are NULL. Through 1.13.x that was exactly where tier1 was NULL or was not an IAB v3.1 tier-1 name. From 1.14.0 that is no longer the rule: a row carrying a flagged theme the pack maps to a group resolves from the safety axis alone, so policy_group, policy_basis and policy_pack_version are populated (basis safety) even where tier1 is NULL or unresolvable. From 1.14.0 the three columns are NULL exactly where:

  • the row is suppressed — parked, inactive, dead, and the stub/no-evidence/template/demoted guards on any lane. The whole category block is blanked on those rows, so the policy columns go NULL there whatever the safety axis says. safety_flags and safety_detail are blanked with them; safety_status is preserved, and is the only reading that survives on a suppressed row; or
  • neither axis resolves on a live row: no flagged theme names a group and tier1 is NULL or is not an IAB v3.1 tier-1 name.

  A deeper tier that does not resolve (an unknown tier2/tier3/tier4 name) falls back to the deepest resolvable prefix — the same rule as openrtb_cat, which keeps the ids of the resolvable prefix — so a group read from the label can only ever be the parent's, never a more specific claim than the label supports. None of the three is carried into the last_known_* block.

• The policy columns are derived, not tracked. The mapping is versioned independently of the schema: manifest.json carries policy_pack: {id, version, sha256}, and a pack edit (say, moving casino-themed video games out of gambling) bumps the version and is listed in the changelog with no column change. Because it touches every row, a pack version change ships via the next full snapshot, never as a delta: every delta carries the policy_pack of the snapshot it extends, a delta whose pack differs from its predecessor is a chain gap, and the exporter refuses to write one. So the first ≥ 1.12.0 artifact you ingest must be a snapshot, and after any policy_pack change you will receive a snapshot before the next delta. The pack id is half of that identity, not decoration: policy_v1@2026.09.1 and policy_v2@2026.09.1 carry the same version and different mappings, so compare policy_pack as the id + version pair — the <pack>@<version> form the per-row policy_pack_version column already uses. Pin that pair if your policy depends on exact membership.

policy_confidence_basis and policy_group_set (columns 42–43)

Two columns that make the policy block usable without a lookup table.

policy_confidence_basis says what column 7 / column 24 IS on this row. §5 documents that confidence is not one scale — a trained classifier's probability, a filter's internal score, a keyword density and a hand-chosen trust value all live in the same column, and until now the only way to tell them apart was to know the model_name families by heart. The value is a closed enum:

The writer family is decided first: a filter, rule, curated, carry-over or name-only row keeps its family whatever the calibrated column holds, so model_calibrated appears only where the row would otherwise read model_raw. Two identities hold on any artifact: count(policy_confidence_basis = 'model_calibrated') equals row_count_calibrated, and every *_filter writer reads filter.

policy_basis (column 37) is a different question — which AXIS the group came from — and from 1.14.0 its documented values are primary, site_purpose and safety. role is reserved for a later release and is not emitted at 1.14.0. If you validate column 37 against a strict enum, add safety before ingesting a 1.14.0 artifact; this is the one breaking value addition in the release, and it is confined to an opt-in block.

policy_group_set is the group plus its declared union. Blocking "gambling-shaped" content means matching gambling, gambling_games and harmful_content; blocking drugs means drugs, alcohol and tobacco. The pack declares those unions, and this column materializes them per row, own group first, then its partners in pack order:

policy_group = "gambling"  → policy_group_set = ["gambling", "gambling_games", "harmful_content"]
policy_group = "shopping"  → policy_group_set = ["shopping"]

One array_contains replaces a join against the taxonomy endpoint. It is NULL exactly where policy_group is NULL, so the two columns can never disagree about whether the projection ran.

12. Quick reference

• 46 columns, fixed order, additive-only. Read by name where you can. Columns 27–35 are the reserved site_purpose_* axis (NULL through 1.13.x except site_purpose_status); columns 39–44 were appended at 1.14.0 and columns 45–46 at 1.17.0.
• content_purpose_id (column 45) is 1020, IAB Content Purpose = News, on the domains of a curated list of known news publishers and NULL everywhere else — a list, not news-site coverage, and NULL is unknown, never "not news". The topic columns are unchanged (§14).
• domain_status has six values. parked / inactive / dead carry no category at all; active, blocked and uncategorizable may. active is not a promise of a label — test tier1 IS NOT NULL instead (§3).
• Want a reliability signal? Use calibrated_confidence (a probability), not confidence (an uncalibrated model score). See §5.
• calibrated_confidence is populated on a subset of categorized rows. tier1 IS NULL implies it is NULL; the converse does not hold — see §5.
• calibration_segment = '__global__' marks the weakest calibrated rows.
• blocked = live but unreadable. Not parked, not dead — and it keeps its category and its calibrated value from the previous crawl. A 304 Not Modified answer is not blocked (from the 2026-09-17 set).
• classified_at = when we processed it. status_observed_at = when the crawl saw the status. Neither = when the content changed.
• confidence is not one scale: on filter rows (*/st_filter, */gambling_filter, */adult_filter) and rule/lookup rows (curated/*, tld_rule/*, schema_rule/*, v3_historical_fallback) it is a component's own score, not a category probability — check model_name before comparing or thresholding (§5).
• pipeline/vendor_rule rows carry the classifier's confidence under a label the rule chose (Business and Finance → Technology & Computing on vendor sites); calibrated_* is NULL there (§5).
• curated/* rows are hand-vetted, tier = 3, always domain_status = active by assertion, always NULL calibrated_*, and ride every snapshot and every delta (deltas from artifact ≥ 1.13.2) — apply them by domain, never by classified_at order, because a curated row is usually dated earlier than the row it replaces (§5).
• model_version may carry a +variant suffix (v1+ml_en) — compare on the base version, never with = (§5).
• openrtb_cat is an array — JSON-encoded in CSV, native list in Parquet.
• policy_group is the policy-pack rollup (gambling, adult, …), versioned by policy_pack_version / policy_pack.version. Below 1.14.0 it was the policy_v1 rollup of the tier path and nothing else, so it was NULL exactly where that path did not resolve; from 1.14.0 the pack is policy_v2, which reads safety_flags first and renames news to current_events. So from 1.14.0 a flagged row carries a group even where tier1 is NULL, and the column is NULL only on a suppressed row (parked / inactive / dead and the stub guards) or where neither axis resolves — never read tier1 IS NULL as "no group". site_purpose_status is never NULL (§11).
• last_known_* is historical and sparse. Read your delivery's own row_count_with_last_known_category; figures quoted in documentation age (§8).
• NULL is an empty CSV field / a true Parquet null. No sentinels.
• A dated snapshot is complete iff exports/snapshot_<date>/run_manifest.json exists — read the sixteen partition manifests from it instead of globbing (§10).
• safety_status (column 39) is never NULL and is kept even on a suppressed row; safety_flags / safety_detail are NULLed with the category block. safety_status and safety_flags read brand-safety verdicts only (§13).
• policy_confidence_basis (column 42) tells you what the number in column 7 / 24 is without knowing the model_name families; policy_basis (column 37) gains the value safety at 1.14.0 — a breaking value addition if you validate it against a strict enum (§11).
• policy_group_set (column 43) is the group plus its declared union — match one list instead of rebuilding gambling ∪ gambling_games ∪ harmful_content yourself.
• language_primary (column 44) is the BCP-47 primary subtag of column 14. eng stays eng — it is a valid three-letter subtag and is deliberately not folded to en, because the two spellings come from different detectors and folding them would claim an equivalence we have not measured. Column 14 stays raw.

13. The brand-safety axis (columns 39–41)

Three columns, appended at artifact 1.14.0, carrying what our brand-safety filters concluded about a domain — separately from what its category says.

Why it is a separate axis: a category answers "what is this site for?", and a blocking policy usually also needs "did anything flag it?". Those are different questions, and they disagree in both directions — a gambling operator labelled Attractions > Casinos & Gambling is normally flagged, a news outlet reporting on gambling normally is not, and a parked ex-gambling domain can carry a flag with no category at all. Neither filter is exact: a model-adjudicated read of the 2026-10-18 set found the sensitive-content filter flagging some clearly legitimate sites at the head. Reading the second question off tier1 or off model_name is what customers were doing, and it under-blocks by construction.

safety_status is preserved through suppression, like site_purpose_status. A parked or dead row ships no category, but "a brand-safety filter looked at this domain" stays true whatever the feed does with the label — so column 39 keeps its value while columns 40 and 41 go NULL with the rest of the category block. A flagged status on a parked row is a fact about the assessment, not a category claim; the flags that would substantiate it are not published on that row, by design.

What counts as a brand-safety verdict — and what does not. safety_status and safety_flags are computed over brand-safety verdicts only. The vendor rule is a ROLE classification, not a brand-safety one: it decides whether a Business and Finance label belongs on a company that sells to that industry rather than operating in it. Its verdict appears in safety_detail under the family vendor and nowhere else — never in safety_flags, never enough to make a row flagged, and never enough to move a domain out of not_assessed. A role basis is reserved for a later release. Shadow themes (measurements whose precision has not been read yet, such as adult_nonadjacent) are recorded in safety_detail on the same terms: they are not block signals.

Rows written before the verdict store. A domain's verdicts are stored from the 1.14.0 release onwards. Rows whose last write predates it are derived from the row's existing model_name and tier2 and carry a legacy version marker in safety_detail. That derivation exists so the axis is complete on the first artifact rather than reading not_assessed for every pre-existing filter row — a domain last crawled 80 days ago is 90-day-shielded and would otherwise never gain a verdict. For those rows the flag is a re-statement of what column 10 already told you, not new evidence; safety_detail's version marker is how you tell the two apart.

Reading it. safety_status = 'flagged' is the coarse gate. safety_flags is the one to key a block list on if you want every flagged row regardless of where its label landed — policy_group under policy_v2 reads the safety axis first, so a flagged row usually carries the theme's group, but a row whose label already resolves to a different sensitive group keeps that group instead. safety_detail is diagnostic: read it to understand a verdict, not to build a filter, because its shape is per-family and will gain families.

14. IAB Content Purpose (columns 45–46)

Two columns, appended at artifact 1.17.0, that say a domain is a news publisher in the IAB's own vocabulary for it. The first artifact carrying them is the Wednesday 2026-11-04 delta (delta_2026-11-03_100000Z); the first full set carrying them is the Sunday 2026-11-08 set.

Why a separate pair of columns. IAB Content Taxonomy v3.1 has no News topic, so a news outlet's tier1–tier4 name what it covers — Politics under the single-label convention (§2), or Sports, Business and Finance and so on. The IAB Tech Lab Content Taxonomy 3.0 Implementation Guide (section 8, Descriptive Vectors) expresses news as a Content Purpose, carried next to the topic and never in place of it: Content Purpose > Informational > News, unique id 1020. These two columns carry that vector. The topic columns are unchanged on every row: filter news on column 45, and keep your topic rules on columns 2–6.

Where the value comes from: a list, and only a list. In the feed the only source is a curated list of known news publishers maintained by our team; the list is versioned, and every change to it is reviewed before release. Every domain on it is also one of our hand-vetted curated/* rows (§5), so the value appears only on curated/* rows. No model sets these columns and no page is read to set them. That makes this a list, not news-site coverage: most news sites in the feed are not on it and read NULL, and will until a later release adds another source. content_purpose_list.row_count in any manifest says how many domains the list names (§10).

NULL means unknown, never "not news". A NULL in column 45 says we hold no news-publisher statement about the domain. It is not a statement that the site is not a news outlet, and it must not be used as one — for example, to build an "exclude news" list out of the NULL rows.

schema_org is reserved. The real-time API's opt-in content_purpose block (include=content_purpose) has a second source, schema_org: the site's own schema.org NewsMediaOrganization markup, read from the crawled page. The feed does not emit it in 1.17.0, because the export carries no page markup. It is named now so a loader that validates column 46 against a strict enum can accept it ahead of time; a release that starts emitting it will be announced in the Website Categorization feed changelog first.

The same answer as the API, where both give one. For a domain on the list, column 45 and the API block's id are both 1020, and column 46 and its source are both curated. A domain the API tags from its markup alone (schema_org) reads NULL here. The legacy v1 response field that is also called content_purpose is a different thing: reserved, null in every deployment, and not carried by these columns.

How it reaches you. Curated rows ride every snapshot and every delta whole, so the deltas are enough: if you consume deltas only, add the two columns as NULL to your table and apply the first 1.17.0 delta — the two columns then hold exactly what a 1.17.0 full set publishes in them. When the list changes, the next delta re-sends every curated row with its current value, so a domain taken off the list reads NULL on your side after that delta. A domain taken off our curated list altogether stops riding the deltas, so your copy keeps its last curated row, tag included, until the domain ships again from our other lanes — in a later delta or the next full set, with columns 45–46 NULL — or a full set no longer carries it (§10).

Suppression. Both columns belong to the content-category block, so they are NULL on any row whose category is withheld (§9). A curated row is always active (§3), so this removes no value today.

Counting it. row_count_by_content_purpose_source counts rows per column 46 value, and content_purpose_list names the list the values were read from — its sha256 and its row_count (§10). On a delta, and on a full set's run marker (which sums the sixteen partitions), the curated count equals the list's row_count unless a listed domain did not ship as a curated row in that artifact — a takedown withholds it, for example. A partition manifest counts only its own partition's tagged rows. The run marker also names the one list the whole set was tagged from; a set whose partitions were tagged from two different lists is not marked complete.