Website Categorization Bulk Feed — Data Dictionary

Part 1 of 3. Sections 1 to 4 are on this page. Sections 5 to 9 are on Data dictionary, part 2, and sections 10 to 15 on Data dictionary, part 3.

Audience: licensees integrating the Website Categorization bulk domain-categorization feed. This page defines every column, every value domain, and the two timestamps that are most often confused. It assumes no knowledge of our internals.

Download this document: website-categorization-feed-data-dictionary.md — the same reference as a single file, for your integration team. Version history: Website Categorization feed changelog — every dated change to the feed's published values and semantics, with the two policy-pack references beside it.

If you are integrating the real-time API instead, see the API documentation API quickstart & error reference — the feed and the API share a taxonomy but not a schema.


1. What you receive

Each export is a directory containing:

File Contents
*.csv.gz gzipped CSV, one header row then one row per domain
*.parquet the same rows, same column order, native types
manifest.json row counts, per-file SHA-256, schema version, and lane semantics

Both data files carry identical rows in identical column order. Choose whichever suits your loader; you do not need both.

1.1 Where and when it arrives

Delivery is an S3 bucket you read with your own read-only key. The bucket, the key and the commands come with your license.

Item Value
Full snapshot weekly, starting on Sundays and lately taking about 30 hours to run: sixteen partition folders exports/snapshot_<date>_p00/ … _p15/, each with snapshot_pNN.csv.gz, snapshot_pNN.parquet and manifest.json. The set is complete only when exports/snapshot_<date>/run_manifest.json exists (usually on Monday, UTC) — ingest nothing before it appears (§10).
Daily delta 10:00 UTC every day except Sundays, when the snapshot holds the export lock: exports/delta_<window-start>_100000Z/ with delta.csv.gz, delta.parquet and manifest.json, landing within about an hour on weekdays; the Monday delta waits for the weekly set's run marker, so it lands after the set does. The folder is named by the start of the window it covers, so the delta published on the 10th is delta_2026-09-09_100000Z. A window opens at the prior run's cutoff, so after a skipped run the next spans two days; to resume after a set, see below. Apply every delta row as an upsert by domain and never filter a delta by its own timestamps: a row can qualify through a category-side event the feed does not show (an abstain, or a label withdrawn by suppression), so a small share of every delta has no timestamp inside the window: its classified_at is NULL and its status_observed_at is older than the window.
Documentation this dictionary, the changelog and both policy-pack references sit beside each set's run_manifest.json, each stamped with the revision it was published from.
Older folders snapshot folders dated on or before 2026-08-03 predate the completeness marker and are not whole sets; ignore them.

Resuming after a full set. After loading a set, apply the delta whose window contains the whole of the set's export — it opened before the set began and was read after its run_manifest.json appeared — then every later delta in order. For the weekly set that is normally the delta dated the day before it — delta_2026-10-17_100000Z after the 2026-10-18 set — which runs only after the set is complete, so it never carries an older version of a row than the set holds. Skipping it loses what changed while the set was being written. A row can still reach you one delta after its crawl (a new domain crawled before a delta's read but first labelled after it, which ships in the next delta), and a full set reconciles anything a delta did not carry. From delta_2026-10-30_100000Z, a delta also re-reads crawls stamped in the 30 minutes before its window opens, so a crawl stamped just before a cutoff but saved just after it is not missed; one saved more than 30 minutes after its stamp still can be, until the next full set. The cost is repetition: a row can repeat one the previous delta carried, with its status_observed_at up to 30 minutes older than the window and its classified_at older too, or NULL. An upsert by domain absorbs the repeat.

Verify every downloaded file against the SHA-256 in its manifest.json (§10).

Which schema version you are reading. The 2026-09-06 set reports feed_schema_version 1.13.0; every delta from 2026-09-08 and the 2026-09-13 set report 1.13.2 with the same 38 columns. The first 1.14.0 artifact is the Thursday 2026-09-17 snapshot set — an off-cadence full set published the day the exporter deploys — and it is a full replace (Website Categorization feed changelog). Every artifact from the Tuesday 2026-09-29 exporter deploy reports 1.15.0: the same 44 columns in the same order, with one published value moving — confidence_tier is capped at MEDIUM for the name-only LLM lane (§5). Those rows are written on 2026-09-30, so the first artifacts carrying any of them are the daily deltas from 2026-09-30 and the Sunday 2026-10-04 set; a delta between the deploy and the load reports 1.15.0 with no such row in it yet. Every artifact from the Friday 2026-10-09 exporter deploy reports 1.16.0: the same 44 columns in the same order, with one published value moving again — confidence_tier is capped at MEDIUM, and at LOW where the calibrated tier is DOUBTFUL, for the teacher-rescored provenance suffixes (§5). Those rows already exist, so unlike 1.15.0 this re-rates rows you already have: the deltas from 2026-10-10 report 1.16.0 (the first is in folder delta_2026-10-09_100000Z), but the re-rating reaches you only with the Sunday 2026-10-11 full set (Website Categorization feed changelog). Every artifact from the Tuesday 2026-11-03 exporter deploy reports 1.17.0: columns 1–44 unchanged, and two columns appended — IAB Content Purpose = News for the domains on a curated list of known news publishers (§14). The first 1.17.0 artifact is the Wednesday 2026-11-04 delta (folder delta_2026-11-03_100000Z). Every artifact from the Friday 2026-11-06 exporter deploy reports 1.17.1, the same 46 columns with two of them cleaned up (see below), so no full set ships at 1.17.0: the first set carrying columns 45–46 is the Sunday 2026-11-08 set, at 1.17.1. Every artifact from the Friday 2026-11-13 exporter deploy reports 1.18.0: columns 1–46 unchanged, and five columns appended — the role axis (§15). The first 1.18.0 artifact is the delta in folder delta_2026-11-13_100000Z. Until the role lane is armed, every row reads role_status = 'not_assessed' with the other four role columns NULL; role-lane rows arrive from the first full set after that, no earlier than Sunday 2026-11-15.

The column order is the contract

The 51 columns below ship in exactly the order listed (38 at feed_schema_version 1.13.x; columns 39–44 arrive with artifact 1.14.0, columns 45–46 with artifact 1.17.0 and columns 47–51 with artifact 1.18.0), in both the CSV header and the Parquet schema. This order is the licensable contract. A reorder or a rename is a breaking change and will not happen without a major version change and notice. New columns are only ever appended to the end, so:

  • reading by column name is always safe;
  • reading by position is safe for every column that existed when you integrated;
  • a consumer that stops reading after column N is unaffected by later additions.

Check feed_schema_version in manifest.json to know which columns and manifest keys to expect. It is a semantic version. The MINOR component increments on purely additive changes — new columns, new manifest keys, or a new manifest block — with no change to the published row population. The PATCH component increments when the published row population changes with no column change: no column is added, removed, renamed or retyped, and no manifest key is removed or given a new meaning. A PATCH release may add keys inside an existing manifest.json object, so validate the manifest permissively — treat its objects as extensible rather than as a closed set of properties — but it never changes the column schema your reader is built against. Every such change is listed in the Website Categorization feed changelog.


2. Column reference

Types are the Parquet physical types. In CSV every value is text — see §7 Nulls and encoding.

# Column Type Meaning
1 domain string The name the row describes: normally the registrable domain, but not always — see the note below the table. Lowercase ASCII with internationalised names in punycode (xn--…) — from 1.14.0; artifacts before it also carry Unicode-form names, most beside their punycode twin (see §7). Unique — no domain value repeats within an export.
2 tier1 string IAB Content Taxonomy v3.1 tier-1 name, e.g. Technology & Computing. The broadest category. Columns 2–5 carry the single-label convention — see the note below the table.
3 tier2 string IAB v3.1 tier-2 name. Null when the label stops at tier 1.
4 tier3 string IAB v3.1 tier-3 name. Null when the label stops higher.
5 tier4 string IAB v3.1 tier-4 name. Null when the label stops higher.
6 iab_id string Canonical IAB leaf id for the tier path, e.g. IAB-596 or IAB-Rm3SiT. An opaque identifier — numeric on most rows, alphanumeric on a small minority. Match it whole; never parse structure out of it.
7 confidence float32 Score for the label in [0, 1], 4 decimal places. Its meaning depends on model_name — see §5.
8 confidence_tier string Discrete band: HIGH / MEDIUM / LOW, or PARKED. Through 1.13.x it is the stored confidence cut; from 1.14.0 it is derived at export under the policy named in confidence_band_policy. See §5.
9 tier int16 Provenance/priority of the label, 0–3. Not a taxonomy depth. See §5.
10 model_name string Component that produced the label, e.g. ensemble-5en-3nonen. Determines what confidence means — see §5.
11 model_version string Version of that component. May carry a +variant suffix — do not match with =, see §5.
12 taxonomy_version string Taxonomy the tier names belong to, e.g. iab_v3.1.
13 classified_at string ISO-8601 UTC. When we processed the domain — see §4.
14 language string Language detected on the crawled page, e.g. en. Null where not detected or not recorded. From 1.17.1 its whitespace is normalised — see §7.
15 domain_status string The status lane. One of six values — see §3.
16 openrtb_cat list<string> OpenRTB/AdCOM raw category ids for the tier path. Array-valued — see §6.
17 status_observed_at string ISO-8601 UTC. When the crawl observed domain_status — see §4.
18 last_known_tier1 string Historical tier-1 label. Populated only on some non-active rows — see §8.
19 last_known_tier2 string Historical tier-2 label.
20 last_known_tier3 string Historical tier-3 label.
21 last_known_tier4 string Historical tier-4 label.
22 last_known_iab_id string Historical IAB leaf id.
23 last_known_classified_at string ISO-8601 UTC. When that historical label was produced — its own as-of date.
24 calibrated_confidence float32 Calibrated estimate of the probability that a strict blind judge, reading the page, assigns the same tier1, [0, 1], 4 dp. On average it understates how often a label is acceptable, though not in its top band. See §5.
25 calibrated_confidence_tier string Band for calibrated_confidence: PROBABLE / TOSSUP / DOUBTFUL. See §5.
26 calibration_segment string Which calibration curve was applied, e.g. en|bf. See §5.
27 site_purpose_tier1 string Reserved (the site-purpose axis). NULL on every row through 1.13.x — see §11.
28 site_purpose_tier2 string Reserved. NULL through 1.13.x.
29 site_purpose_tier3 string Reserved. NULL through 1.13.x.
30 site_purpose_tier4 string Reserved. NULL through 1.13.x.
31 site_purpose_iab_id string Reserved. Canonical leaf id of the purpose label, same format as iab_id. NULL through 1.13.x.
32 site_purpose_confidence float32 Reserved. The purpose model's own probability — not comparable to confidence or calibrated_confidence. NULL through 1.13.x.
33 site_purpose_status string Never NULL. labeled / undefined / not_available. Through 1.13.x: undefined on parked / inactive / dead rows and parked-detector rows, not_available elsewhere. See §11.
34 site_purpose_relation string Reserved. same_path / same_tier1 / different_tier1 between the two axes. NULL through 1.13.x.
35 site_purpose_model_version string Reserved. Provenance of the purpose label. NULL through 1.13.x.
36 policy_group string The policy-pack group, e.g. gambling. At 1.12.0–1.13.x the policy_v1 rollup of the tier path; from artifact 1.14.0 the pack is policy_v2, which reads safety_flags FIRST and the tier path after, and retires the group news in favour of current_events. NULL only on a suppressed row or where neither axis resolves — from 1.14.0 a flagged row carries a group even where tier1 is NULL. See §11.
37 policy_basis string Which axis policy_group was projected from: primary (the tier path), site_purpose, or — from artifact 1.14.0 — safety (the brand-safety axis, read first). Always primary at 1.12.0–1.13.x; NULL where policy_group is NULL.
38 policy_pack_version string policy_v1@2026.09.2 at 1.12.0–1.13.x, policy_v2@2026.09.1 from artifact 1.14.0 — the pack and mapping version the row was projected with; constant within an artifact, NULL where policy_group is NULL.
39 safety_status string Never NULL (artifact ≥ 1.14.0). Whether a brand-safety filter reached a decision about this domain: flagged / assessed_clean / not_assessed. See §13.
40 safety_flags list<string> (artifact ≥ 1.14.0) The flagged block-signal themes, e.g. ["gambling"]. Empty list, never NULL, on a row that carries a category; NULL on a suppressed row. See §13.
41 safety_detail string (artifact ≥ 1.14.0) A compact JSON object, one entry per filter family that ran, carrying that filter's status, score, scale, theme and version — a family reporting not_assessed is retained, so a populated detail is not evidence that anything assessed the domain. NULL on a suppressed row, and on a row where no verdict was materialized at all. See §13.
42 policy_confidence_basis string (artifact ≥ 1.14.0) What the row's number IS: model_calibrated / model_raw / filter / rule / curated / carry_over / name_only (artifact ≥ 1.15.0) / none. Closed enum; if you validate it against a strict enum, add name_only before ingesting a 1.15.0 artifact. NULL only where the category block is suppressed. See §11.
43 policy_group_set list<string> (artifact ≥ 1.14.0) policy_group plus every group the pack declares it is blocked WITH, in pack order — one list to match instead of reconstructing the union. NULL exactly where policy_group is.
44 language_primary string (artifact ≥ 1.14.0) The BCP-47 primary subtag of column 14, e.g. zh for zh-CN / ZH_TW / zh-Hans-CN. Derived; column 14 is never rewritten beyond its whitespace. From 1.17.1 always a two- or three-letter code or NULL: NULL where no language was observed or where column 14 does not begin with a language code (§7).
45 content_purpose_id string (artifact ≥ 1.17.0) 1020 — the IAB Descriptive Vector id of Content Purpose > Informational > News — on the domains of a curated list of known news publishers; NULL on every other row. NULL means unknown, never "not news". Not a Content Taxonomy id: it never appears in iab_id or openrtb_cat. See §14.
46 content_purpose_source string (artifact ≥ 1.17.0) Which rule set column 45: curated (that curated list) — the only value the feed emits; schema_org is reserved. NULL exactly where column 45 is NULL. See §14.
47 role string (artifact ≥ 1.18.0) The role axis: what kind of thing the domain is, separately from what its content is about. infrastructure on a role-lane row — a live DNS zone with no website at its apex or www, shipped in the uncategorizable lane with no category — and NULL on every other row. Only infrastructure ships at 1.18.0; later releases may add values. Never a content category. See §15.
48 role_confidence float32 (artifact ≥ 1.18.0) NULL on every row at 1.18.0, role-lane rows included: a rule, not a model, makes the role-lane claim, so there is no probability to publish.
49 role_status string Never NULL (artifact ≥ 1.18.0). labeled on a role-lane row — its role columns are populated — and not_assessed on every other row. labeled does not make the row a labelled one: a role-lane row carries no category. See §15.
50 role_model_version string (artifact ≥ 1.18.0) The rule, and the version of its code, that made the role claim; NULL exactly where role is NULL. It changes whenever that code does, so compare it whole, never parse it, and do not pin a filter to it — filter on role and role_basis.
51 role_basis string (artifact ≥ 1.18.0) What the role claim rests on: dns_no_website, a DNS probe (§15) — the only value the feed emits; content_model and vendor_rule are reserved for later writers. NULL exactly where role is NULL.

What domain holds. Normally the registrable domain, but not on every row: some rows are a host below a registrable domain (blog.example.com) or a bare public suffix. Look a name up as written first, and fall back to its registrable domain's row only when there is no exact match: a host is labelled from its own pages, and it often carries a different tier1 from its parent.

Columns 24–26 were appended 2026-07-30 (feed_schema_version 1.6.0). Column 7 confidence is unchanged in both meaning and position. Columns 27–38 were appended 2026-09-01 (feed_schema_version 1.12.0); nothing before them changed. Four optional manifest keys for the shippable-key coverage denominator were added 2026-09-03 (feed_schema_version 1.13.0, §10); no column changed. The first PATCH bump, feed_schema_version 1.13.1, was specified to ship with the first Sunday export after its deploy: parked-flagged brand-safety filter rows ship status-only and the demoted_label_suppression manifest block gains counters (§10, Website Categorization feed changelog); no column changed and no manifest key was removed. No artifact reports 1.13.1: 1.13.2 superseded it before any export carried it — the 2026-09-06 set reports 1.13.0 and every export from 2026-09-08 reports 1.13.2, which carries the 1.13.1 change as well. The rule behind that sequence applies to every future bump: a daily delta exported between a deploy and the next Sunday snapshot is stamped with the new version too — both kinds are stamped from the same constant — so consume a full snapshot at the new version before relying on those deltas, because rows the new build masks do not cross the delta cutoffs. The second PATCH bump, feed_schema_version 1.13.2, ships with the first ≥ 1.13.2 export of either kind: every delta now carries the hand-vetted curated/* rows as well as the exclusion that already kept a competing row out (§5, Website Categorization feed changelog); no column changed and no manifest key was removed. For the curated rows the delta between deploy and snapshot needs no such caveat — they are not keyed on a cutoff, so they cross it and arrive with the first ≥ 1.13.2 delta. The 1.13.1 suppression change that 1.13.2 inherits is cutoff-keyed like any masking change, so a 1.13.0 snapshot plus 1.13.2 deltas still retains the categories it withdraws: consume a full ≥ 1.13.2 snapshot (the 2026-09-13 set) before treating your copy as reflecting it.

Columns 39–44 are appended at feed_schema_version 1.14.0, whose first artifact is the Thursday 2026-09-17 snapshot set (specified 2026-09-10; no export before that date carries them); columns 1–38 are unchanged in name, position and type. 1.14.0 is a MINOR bump rather than a PATCH because it does both things at once: it adds columns AND it moves the published row population — the language-NULL guard (§10) starts suppressing categories on rows that carried one at 1.13.2, column 8 becomes derived rather than the stored cut, and the feed's policy pack moves to policy_v2. Read the Website Categorization feed changelog entry for 1.14.0 before ingesting the first artifact at this version, and ingest a full snapshot at 1.14.0 before relying on 1.14.0 deltas.

Columns 45–46 are appended at feed_schema_version 1.17.0, whose first artifact is the Wednesday 2026-11-04 delta (announced 2026-11-02; nothing published before the Tuesday 2026-11-03 exporter deploy carries them); columns 1–44 are unchanged in name, position and type, and no row moves in or out of the feed. Unlike 1.14.0, the deltas are enough to receive them: the only rows that carry a value are the curated rows, and those ride every delta whole (§14). The full-snapshot obligation 1.16.0 set for its band re-rating still stands, and the 2026-10-11 set was the first to meet it.

The PATCH bump feed_schema_version 1.17.1 changes the values of two columns and nothing else: language (column 14) has its whitespace normalised, and language_primary (column 44) carries only two- or three-letter codes (§7, Website Categorization feed changelog). No column, manifest key or row moves, and no label, confidence, band, calibrated value or suppression changes. A row whose data did not change does not ride a delta, so the corrected values reach a delta-only consumer with the first ≥ 1.17.1 snapshot.

Columns 47–51 are appended at feed_schema_version 1.18.0, whose first artifact is the delta in folder delta_2026-11-13_100000Z (nothing published before the Friday 2026-11-13 exporter deploy carries them); columns 1–46 are unchanged in name, position and type. 1.18.0 is a MINOR bump rather than a PATCH for the reason 1.14.0 was: it adds columns AND, once the role lane is armed, it moves the published row population — domains in the top popularity bucket that shipped no row and qualify arrive as role-lane rows, and a few rows move out of dead or blocked into the same lane (§15). Until the lane is armed, every row reads role_status = 'not_assessed' with the other four role columns NULL, so you can add the five columns to the rows you already hold with exactly those values. To receive the role-lane rows, ingest a full ≥ 1.18.0 set at or after the lane is armed; from that set on, a later role write reaches you on an ordinary delta. A role-lane row leaves on a delta only when the delta carries another row for its domain — its category row, its curated row, or a status-only row for a parked page or a placeholder; otherwise it leaves with the next full set (§15).

What tier1..tier4 mean. The stored label follows the ratified single-label convention, which is purpose-first: it names the site's primary purpose — what the site is for — rather than its topic, industry or audience, and a small set of rules (R2, R3, R5, R8 and R9) settles the recurring cases: a gambling home is Attractions > Casinos & Gambling (the harmful subset ships as Sensitive Topics — see §5); a software, SaaS or hosting vendor is Technology & Computing whoever it sells to; a multi-brand store is Shopping while a brand's own store takes its vertical; consumer banking, BNPL, investing and pensions are Personal Finance; a content platform takes the content type it carries, not its delivery mechanism. The convention is set by our labeling rubric; subject-matter and taxonomy-gap rules sit on top of it — news outlets take Politics, since v3.1 has no News tier-1 — so the result is neither a pure topic label nor a pure purpose label. The reserved site_purpose_* columns (27–35) were added so a separately modelled business-purpose answer could be carried without changing this one; eight of them stay NULL and site_purpose_status keeps its documented values — see §11.


3. domain_status — the six lanes

domain_status is a crawl-observed status, and it is orthogonal to the content category. It answers "what kind of thing is at this domain right now", not "what this site is for".

There are exactly six values. They are closed — no other value ships.

Value Meaning
active A live site with real content — including one that answered our conditional request with 304 Not Modified, confirming the content we hold is current (from the 2026-09-17 set). This is the only lane whose content category, where one is present, is a current one — but the lane does not guarantee one; see below.
parked An affirmative parking signal — parking nameserver, marketplace landing page, for-sale page, or a capture that extracted to no readable text — the content fingerprint of a parking-provider redirect stub — except, from 2026-07-31, on a domain in popularity bucket 0 or 1 (§10; see blocked).
inactive The domain resolves but serves no real content and shows no parking signal: placeholder, coming-soon, under-construction. DNS-dead domains are dead, not this.
blocked A live site we could not read on this crawl — anti-bot challenge, 403/429/451/5xx, timeout (including a DNS timeout or server failure), connection or TLS error, or, on a domain in popularity bucket 0 or 1 (best rank 1,000,000 or better on Tranco, Majestic or Umbrella; §10; not applied when rank_ladder is absent), a page that gave us no readable text in a capture made from 2026-07-31 — usually one rendered entirely by JavaScript, a redirect or a bot interstitial. This is not parking and not death, with one exception: a minority of those empty captures are hosting placeholders or parking pages our crawler could not recognize (more often in bucket 1 than in bucket 0; see the changelog). The lane's share of the feed moves with the crawler's re-crawl cadence for blocked sites; the changelog records material moves. A 304 Not Modified answer to our conditional request is not blocked: the site confirmed the content we hold is current, so from the 2026-09-17 set the row is laned as a successful crawl is — active where a label is on file, uncategorizable where no label exists, and parked or blocked by the rule above where the content it confirms extracted to no readable text. From 1.18.0 a few rows leave this lane for the role lane: where the crawl attempt failed and a newer DNS probe confirms a live, delegated zone with no website at its apex or www, the row ships uncategorizable with role = infrastructure (§15).
dead The name does not resolve (a definitive NXDOMAIN or no-address answer, in crawls from 2026-09-23), or 404. Not a website. From 1.18.0 a row on either of those answers — never a 404 — can ship uncategorizable with role = infrastructure instead, where our DNS probe has confirmed a live, delegated zone with no website at its apex or www (§15).
uncategorizable Crawled successfully but no label could be assigned; or the underlying crawl status was not recognized; or — from 1.18.0 — a live DNS zone with no website at its apex or www, which carries role = infrastructure (§15). Read role to tell them apart. The first two are a deliberate fail-loud bucket, never a silent default.

Known issue (October 2026): some live sites ship as parked or inactive. A site whose page our crawler captured empty — often one that builds its page with JavaScript — can carry the parking-stub fingerprint and ship parked, with its label withheld (see parked_stub_suppression in §10). From the 2026-11-03 deploy, on a domain ranked in the top 1M (popularity bucket 0 or 1) such a capture made from 2026-07-31 ships blocked instead, and the 2026-11-08 set is the first to carry every such row. Such a capture made before 2026-07-31 stays parked until a crawl fetches the page again (a 304 Not Modified answer keeps the earlier capture), and below that rank such rows stay parked. Separately, some live sites were laned inactive by an earlier placeholder rule that misread them, and they stay inactive, with their label in last_known_*, until the crawler reads them again. Do not cull a domain on parked alone, nor on the real-time API's is_parked, which is true for inactive answers too.

Three points that matter for integration:

  1. blocked is not parked. A blocked domain is a real, live website that defended itself against our crawler. Treating blocked as parked or dead will discard live sites. This separation is a deliberate design invariant.
  2. uncategorizable is a fail-loud bucket, not a category. It means "we will not guess". It covers a clean crawl that produced no label and an inventory status our mapping does not recognize. It is never used to quietly absorb an unknown into active. From 1.18.0 it also holds a classified subset — live DNS zones with no website at their apex or www, each carrying role = infrastructure — which is still not a category: read role to tell it apart, and filter role IS NULL to leave it out (§15).
  3. Only active rows carry a current content category — but not every active row carries one. On parked, inactive and dead rows every content column (columns 2–8 and 10–14, plus openrtb_cat, the reserved site_purpose_* label columns and the three policy_* columns) is NULL by design. That is not missing data — see §9. The implication runs one way only: active says the crawl found a live site, not that a label was produced or kept for it. A live site can reach the export with every content column NULL — because it was never labelled, or because one of the suppression guards withheld the label it had (the stub, no-evidence, template and demoted-label guards described below and in §10). Test the columns you need (tier1 IS NOT NULL); do not infer them from domain_status = 'active'.

blocked rows are the exception worth knowing: they keep the category from a previous successful crawl, because the site is presumed live and unchanged. That label is real but was not re-verified on this crawl. Do not drop blocked rows as category-free — after parked, this is the largest non-active lane. Most of those carried labels are DOUBTFUL, as most labels on active rows are, and a model-adjudicated read of the 2026-10-18 set found them less often acceptable than labels on active rows, on a small sample: filter them on calibrated_confidence_tier rather than keeping or dropping the lane wholesale.

uncategorizable rows likewise keep whatever label is already on file, if there is one: where the lane was reached because the crawl status was not recognized, an existing label is retained rather than blanked. A role-lane row (from 1.18.0) is the exception: it never carries a label. Where one is on file for its domain, the role lane keeps it out of the artifact, and the manifest counts those rows in row_count_role_lane_category_superseded (§10).

From artifact 1.15.0 a blocked or uncategorizable row may carry a tier-1 category that was never read from the page: the name-only LLM lane (model_name = llm-name/consensus_v1, policy_confidence_basis = name_only; see §5 and §11) labels head domains (Tranco rank ≤ 100,000) the crawler cannot read from the domain name, where two model families agree. The torso tranche (rank 100,000–1,000,000) this lane first reserved was measured and withheld for now — see the 1.15.0 notice in the changelog — and nothing ranked past 1,000,000 is ever labelled this way. The lane the crawl decided is unchanged by it: the label is additional information about the domain, not evidence that the site was read. (When we checked on 2026-09-30, a small number of these rows, whose crawl produced no label from the page, would ship in the lane a label read from the page would give them: some as active, with the label, and others as parked, without it. From delta_2026-10-02_100000Z and the 2026-10-04 set they keep the lane the crawl decided; the changelog's 2026-10-01 and 2026-10-02 entries have the detail. To find a name-only label, test policy_confidence_basis or model_name, not domain_status.) On parked, inactive and dead rows such a label is withheld like any other, with no last_known_* block.

One exception applies on every lane, including blocked and uncategorizable: rows whose latest capture extracted to no readable text (see parked_stub_suppression in §10) ship with NULL category columns wherever they land. No label can be checked against such a capture — see the no-content note in §5 — so it is withheld rather than retained, with no last_known_* block either. Where the crawl itself succeeded and the label is not a name-only one (which never decides the lane; see above), such a capture ships parked (it is read as a parking-provider redirect stub), except that one made from 2026-07-31 on a domain in popularity bucket 0 or 1 (§10) ships blocked (read as a live site we could not read; a minority are hosting placeholders or parking pages). An export without the buckets (rank_ladder absent) makes no such exception. So a blocked row with a NULL category is either a site that was never labeled or one of these rows; in both cases the NULL is deliberate.

curated/* rows: active is asserted, not crawl-observed

Rows whose model_name begins with curated/ (see §5) are the one place where domain_status is not derived from a crawl. They always carry active, because a human placed the domain on the curated list and that list is the same first-tier lookup the real-time API answers from — so the feed and the API give you the same answer for those domains. The list includes CDN, DNS and API zones that serve no homepage (akamaiedge.net, cloudfront.net and googleapis.com, for example): their active is the curated assertion, not a sign that a website answered.

Two consequences, stated plainly rather than hidden:

  • The lane is a statement about the curated entry, not about the last crawl. These are disproportionately the sites that defend hardest against crawlers, which is precisely why they were curated; deriving their lane from a crawl would bury a human-verified answer under a blocked verdict.
  • status_observed_at on these rows still dates the last crawl attempt, and is NULL where the domain has no crawl record at all. It is the one column on a curated/* row that reports observation rather than assertion, so do not read it as dating the active lane.

Curated rows are a small share of the feed. If you need to exclude or specially handle them, filter on model_name LIKE 'curated/%'; manifest.json publishes the count as row_count_curated.

www. hosts: a curated domain ships once, under its bare (no-www.) name. The www. spelling of a curated domain is not published at all — not as a curated row and not as a model-produced one — so it cannot contradict the bare host. If your lookup key can carry a www. prefix, strip it before matching, which is what our real-time API does with the requested domain.


4. The two timestamps

The feed carries two dates and they mean different things. This is the single most common misreading, so it is stated plainly:

Column What it dates
classified_at When we processed the domain and produced or served the label.
status_observed_at When the crawler observed the domain_status on that row.

Neither is "when the website's content changed." Website Categorization does not publish a content-change date, and neither column should be used as a proxy for one.

  • classified_at records our processing, not the site's. In our own code comments: "classified_at only records when webcat served or cached the verdict." A recent classified_at does not mean the page was recently re-fetched.
  • status_observed_at is the crawl-attempt timestamp behind domain_status. It is populated on every row the crawler has observed, in every lane, and the exceptions are visible rather than silent: a row whose inventory carries a status but no attempt timestamp ships with it null and is counted, per lane, in the manifest's row_count_by_status_age null bucket (artifact ≥ 1.14.0), and a curated row carries it only where the inventory has a row for that domain. Use it to judge how old a parked / blocked / dead verdict is.
  • On a role-lane row (from 1.18.0) status_observed_at dates our DNS probe, not a crawl attempt: it is the latest probe that confirmed the role claim, because the probe, not a crawl, decided that row's lane (§15). classified_at is NULL there, since no label was produced. Contrast a curated row, where status_observed_at keeps dating the last crawl attempt (§3).
  • On rows where a historical label is present, last_known_classified_at dates that label, and is generally older than both of the above.
  • How old the status observations are is published in the manifest rather than left to a scan (artifact ≥ 1.14.0): row_count_by_status_age buckets every row, per lane, by status_age_reference_date minus the UTC calendar day of status_observed_at, and stale_fraction_active_gt_90d is the active lane's share observed more than 90 days before that reference — see §10. The reference is the artifact's key date (a snapshot's date, a delta's cutoff day), never the time the export happened to run, so a re-published artifact reports the same ages.
  • classified_at is usually the later of the two. Outside the top 1M, a re-crawl re-labels a row only when its content hash changed.