Provide current and historical ownership information on domains / IPs. Identify all connections between domains, registrants, registrars, and DNS servers.
Look into all current and historical DNS / IP connections between domains and A, MX, NS, and other records. Monitor suspicious changes to DNS records.
Get detailed context on an IP address, including its user’s geolocation, time zone, connected domains, connection type, IP range, ASN, and other network ownership details.
Access our web-based solution to dig into and monitor all domain events of interest.
Get access to a web-based enterprise-grade solution to search and monitor domain registrations and ownership details for branded terms, fuzzy matches, registrants of interest, and more.
Independent, evidence-based DNS and abuse intelligence for applicants, advisors, governments, and counsel participating in the ICANN 2026 New gTLD Program.
Predictive threat intelligence is your best first line of defense. Subscribe to the feeds to strengthen your cybersecurity posture. Contact us today for more information.
Unlock integrated intelligence on Internet properties and their ownership, infrastructure, and other attributes.
Our complete set of domain, IP, and DNS intelligence available via API calls as an annual subscription with predictable pricing.
Offers complete access to WHOIS, IP, DNS, and subdomain data for product enrichment, threat hunting and more.
Uncover entire attack surfaces with this API to embed asset discovery, vulnerability scanning, and technology metadata into your platform. Now in early access.
Talk to our APIs using LLMs. Connect your preferred LLM to WhoisXML API and simply chat about WHOIS, DNS, threat intelligence, and more.
I’m your Domain Intelligence Assistant. I make it easy to explore WHOIS, DNS, and threat data from WhoisXML API — I’m cloud-based, fast, and always ready to help.
A custom GPT for WHOIS, DNS, IP, and threat intelligence research. Connects ChatGPT directly to WhoisXML API to enable fast, conversational investigations and domain insights.
One Universal API Credit Pool
Simplify API credit purchasing and usage with one balance that works across our product portfolio.
Part 1 of 3. Sections 1 to 4 are on this page. Sections 5 to 9 are on Data dictionary, part 2, and sections 10 to 15 on Data dictionary, part 3.
Audience: licensees integrating the Website Categorization bulk domain-categorization feed. This page defines every column, every value domain, and the two timestamps that are most often confused. It assumes no knowledge of our internals.
Download this document: website-categorization-feed-data-dictionary.md —
the same reference as a single file, for your integration team.
Version history: Website Categorization feed changelog — every dated change
to the feed's published values and semantics, with the two policy-pack references beside it.
If you are integrating the real-time API instead, see the API documentation API quickstart & error reference — the feed and the API share a taxonomy but not a schema.
Each export is a directory containing:
| File | Contents |
|---|---|
*.csv.gz |
gzipped CSV, one header row then one row per domain |
*.parquet |
the same rows, same column order, native types |
manifest.json |
row counts, per-file SHA-256, schema version, and lane semantics |
Both data files carry identical rows in identical column order. Choose whichever suits your loader; you do not need both.
Delivery is an S3 bucket you read with your own read-only key. The bucket, the key and the commands come with your license.
| Item | Value |
|---|---|
| Full snapshot | weekly, starting on Sundays and lately taking about 30 hours to run: sixteen partition folders exports/snapshot_<date>_p00/ … _p15/, each with snapshot_pNN.csv.gz, snapshot_pNN.parquet and manifest.json. The set is complete only when exports/snapshot_<date>/run_manifest.json exists (usually on Monday, UTC) — ingest nothing before it appears (§10). |
| Daily delta | 10:00 UTC every day except Sundays, when the snapshot holds the export lock: exports/delta_<window-start>_100000Z/ with delta.csv.gz, delta.parquet and manifest.json, landing within about an hour on weekdays; the Monday delta waits for the weekly set's run marker, so it lands after the set does. The folder is named by the start of the window it covers, so the delta published on the 10th is delta_2026-09-09_100000Z. A window opens at the prior run's cutoff, so after a skipped run the next spans two days; to resume after a set, see below. Apply every delta row as an upsert by domain and never filter a delta by its own timestamps: a row can qualify through a category-side event the feed does not show (an abstain, or a label withdrawn by suppression), so a small share of every delta has no timestamp inside the window: its classified_at is NULL and its status_observed_at is older than the window. |
| Documentation | this dictionary, the changelog and both policy-pack references sit beside each set's run_manifest.json, each stamped with the revision it was published from. |
| Older folders | snapshot folders dated on or before 2026-08-03 predate the completeness marker and are not whole sets; ignore them. |
Resuming after a full set. After loading a set, apply the delta whose window contains the
whole of the set's export — it opened before the set began and was read after its
run_manifest.json appeared — then every later delta in order. For the weekly set that is
normally the delta dated the day before it — delta_2026-10-17_100000Z after the 2026-10-18
set — which runs only after the set is complete, so it never carries an older version of a row
than the set holds. Skipping it loses what changed while the set was being written. A row can still reach you one delta after its crawl (a new domain
crawled before a delta's read but first labelled after it, which ships in the next delta), and
a full set reconciles anything a delta did not carry. From delta_2026-10-30_100000Z, a delta
also re-reads crawls stamped in the 30 minutes before its window opens, so a crawl stamped just
before a cutoff but saved just after it is not missed; one saved more than 30 minutes after its
stamp still can be, until the next full set. The cost is repetition: a row can repeat one the previous delta carried, with its status_observed_at up to 30 minutes older than the window and
its classified_at older too, or NULL. An upsert by domain absorbs the repeat.
Verify every downloaded file against the SHA-256 in its manifest.json (§10).
Which schema version you are reading. The 2026-09-06 set reports feed_schema_version
1.13.0; every delta from 2026-09-08 and the 2026-09-13 set report 1.13.2 with the same 38
columns. The first 1.14.0 artifact is the Thursday 2026-09-17 snapshot set — an off-cadence
full set published the day the exporter deploys — and it is a full replace
(Website Categorization feed changelog). Every artifact from the Tuesday 2026-09-29 exporter
deploy reports 1.15.0: the same 44 columns in the same order, with one published value
moving — confidence_tier is capped at MEDIUM for the name-only LLM lane (§5).
Those rows are written on 2026-09-30, so the first artifacts carrying any of them are the daily
deltas from 2026-09-30 and the Sunday 2026-10-04 set; a delta between the deploy and the
load reports 1.15.0 with no such row in it yet. Every artifact from the Friday 2026-10-09
exporter deploy reports 1.16.0: the same 44 columns in the same order, with one published
value moving again — confidence_tier is capped at MEDIUM, and at LOW where the calibrated
tier is DOUBTFUL, for the teacher-rescored provenance suffixes (§5).
Those rows already exist, so unlike 1.15.0 this re-rates rows you already have: the deltas from
2026-10-10 report 1.16.0 (the first is in folder delta_2026-10-09_100000Z), but the
re-rating reaches you only with the Sunday 2026-10-11 full set
(Website Categorization feed changelog). Every artifact from the Tuesday 2026-11-03 exporter
deploy reports 1.17.0: columns 1–44 unchanged, and two columns appended — IAB Content
Purpose = News for the domains on a curated list of known news publishers
(§14). The first 1.17.0 artifact is the Wednesday
2026-11-04 delta (folder delta_2026-11-03_100000Z). Every artifact from the Friday
2026-11-06 exporter deploy reports 1.17.1, the same 46 columns with two of them cleaned
up (see below), so no full set ships at 1.17.0: the first set carrying columns 45–46 is the
Sunday 2026-11-08 set, at 1.17.1. Every artifact from the
Friday 2026-11-13 exporter deploy reports 1.18.0: columns 1–46 unchanged, and five
columns appended — the role axis (§15). The first 1.18.0
artifact is the delta in folder delta_2026-11-13_100000Z. Until the role lane is armed,
every row reads role_status = 'not_assessed' with the other four role columns NULL;
role-lane rows arrive from the first full set after that, no earlier than Sunday
2026-11-15.
The 51 columns below ship in exactly the order listed (38 at feed_schema_version 1.13.x;
columns 39–44 arrive with artifact 1.14.0, columns 45–46 with artifact 1.17.0 and columns
47–51 with artifact 1.18.0), in both the CSV header and the Parquet schema. This order is
the licensable contract. A reorder or a rename is a breaking change and will not happen
without a major version change and notice. New columns are only ever appended to the end, so:
Check feed_schema_version in manifest.json to know which columns and manifest keys
to expect. It is a semantic version. The MINOR component increments on purely additive
changes — new columns, new manifest keys, or a new manifest block — with no change to
the published row population. The PATCH component increments when the published row
population changes with no column change: no column is added, removed, renamed or
retyped, and no manifest key is removed or given a new meaning. A PATCH
release may add keys inside an existing manifest.json object, so validate the
manifest permissively — treat its objects as extensible rather than as a closed set of
properties — but it never changes the column schema your reader is built against. Every
such change is listed in the Website Categorization feed changelog.
Types are the Parquet physical types. In CSV every value is text — see §7 Nulls and encoding.
| # | Column | Type | Meaning |
|---|---|---|---|
| 1 | domain |
string |
The name the row describes: normally the registrable domain, but not always — see the note below the table. Lowercase ASCII with internationalised names in punycode (xn--…) — from 1.14.0; artifacts before it also carry Unicode-form names, most beside their punycode twin (see §7). Unique — no domain value repeats within an export. |
| 2 | tier1 |
string |
IAB Content Taxonomy v3.1 tier-1 name, e.g. Technology & Computing. The broadest category. Columns 2–5 carry the single-label convention — see the note below the table. |
| 3 | tier2 |
string |
IAB v3.1 tier-2 name. Null when the label stops at tier 1. |
| 4 | tier3 |
string |
IAB v3.1 tier-3 name. Null when the label stops higher. |
| 5 | tier4 |
string |
IAB v3.1 tier-4 name. Null when the label stops higher. |
| 6 | iab_id |
string |
Canonical IAB leaf id for the tier path, e.g. IAB-596 or IAB-Rm3SiT. An opaque identifier — numeric on most rows, alphanumeric on a small minority. Match it whole; never parse structure out of it. |
| 7 | confidence |
float32 |
Score for the label in [0, 1], 4 decimal places. Its meaning depends on model_name — see §5. |
| 8 | confidence_tier |
string |
Discrete band: HIGH / MEDIUM / LOW, or PARKED. Through 1.13.x it is the stored confidence cut; from 1.14.0 it is derived at export under the policy named in confidence_band_policy. See §5. |
| 9 | tier |
int16 |
Provenance/priority of the label, 0–3. Not a taxonomy depth. See §5. |
| 10 | model_name |
string |
Component that produced the label, e.g. ensemble-5en-3nonen. Determines what confidence means — see §5. |
| 11 | model_version |
string |
Version of that component. May carry a +variant suffix — do not match with =, see §5. |
| 12 | taxonomy_version |
string |
Taxonomy the tier names belong to, e.g. iab_v3.1. |
| 13 | classified_at |
string |
ISO-8601 UTC. When we processed the domain — see §4. |
| 14 | language |
string |
Language detected on the crawled page, e.g. en. Null where not detected or not recorded. From 1.17.1 its whitespace is normalised — see §7. |
| 15 | domain_status |
string |
The status lane. One of six values — see §3. |
| 16 | openrtb_cat |
list<string> |
OpenRTB/AdCOM raw category ids for the tier path. Array-valued — see §6. |
| 17 | status_observed_at |
string |
ISO-8601 UTC. When the crawl observed domain_status — see §4. |
| 18 | last_known_tier1 |
string |
Historical tier-1 label. Populated only on some non-active rows — see §8. |
| 19 | last_known_tier2 |
string |
Historical tier-2 label. |
| 20 | last_known_tier3 |
string |
Historical tier-3 label. |
| 21 | last_known_tier4 |
string |
Historical tier-4 label. |
| 22 | last_known_iab_id |
string |
Historical IAB leaf id. |
| 23 | last_known_classified_at |
string |
ISO-8601 UTC. When that historical label was produced — its own as-of date. |
| 24 | calibrated_confidence |
float32 |
Calibrated estimate of the probability that a strict blind judge, reading the page, assigns the same tier1, [0, 1], 4 dp. On average it understates how often a label is acceptable, though not in its top band. See §5. |
| 25 | calibrated_confidence_tier |
string |
Band for calibrated_confidence: PROBABLE / TOSSUP / DOUBTFUL. See §5. |
| 26 | calibration_segment |
string |
Which calibration curve was applied, e.g. en|bf. See §5. |
| 27 | site_purpose_tier1 |
string |
Reserved (the site-purpose axis). NULL on every row through 1.13.x — see §11. |
| 28 | site_purpose_tier2 |
string |
Reserved. NULL through 1.13.x. |
| 29 | site_purpose_tier3 |
string |
Reserved. NULL through 1.13.x. |
| 30 | site_purpose_tier4 |
string |
Reserved. NULL through 1.13.x. |
| 31 | site_purpose_iab_id |
string |
Reserved. Canonical leaf id of the purpose label, same format as iab_id. NULL through 1.13.x. |
| 32 | site_purpose_confidence |
float32 |
Reserved. The purpose model's own probability — not comparable to confidence or calibrated_confidence. NULL through 1.13.x. |
| 33 | site_purpose_status |
string |
Never NULL. labeled / undefined / not_available. Through 1.13.x: undefined on parked / inactive / dead rows and parked-detector rows, not_available elsewhere. See §11. |
| 34 | site_purpose_relation |
string |
Reserved. same_path / same_tier1 / different_tier1 between the two axes. NULL through 1.13.x. |
| 35 | site_purpose_model_version |
string |
Reserved. Provenance of the purpose label. NULL through 1.13.x. |
| 36 | policy_group |
string |
The policy-pack group, e.g. gambling. At 1.12.0–1.13.x the policy_v1 rollup of the tier path; from artifact 1.14.0 the pack is policy_v2, which reads safety_flags FIRST and the tier path after, and retires the group news in favour of current_events. NULL only on a suppressed row or where neither axis resolves — from 1.14.0 a flagged row carries a group even where tier1 is NULL. See §11. |
| 37 | policy_basis |
string |
Which axis policy_group was projected from: primary (the tier path), site_purpose, or — from artifact 1.14.0 — safety (the brand-safety axis, read first). Always primary at 1.12.0–1.13.x; NULL where policy_group is NULL. |
| 38 | policy_pack_version |
string |
policy_v1@2026.09.2 at 1.12.0–1.13.x, policy_v2@2026.09.1 from artifact 1.14.0 — the pack and mapping version the row was projected with; constant within an artifact, NULL where policy_group is NULL. |
| 39 | safety_status |
string |
Never NULL (artifact ≥ 1.14.0). Whether a brand-safety filter reached a decision about this domain: flagged / assessed_clean / not_assessed. See §13. |
| 40 | safety_flags |
list<string> |
(artifact ≥ 1.14.0) The flagged block-signal themes, e.g. ["gambling"]. Empty list, never NULL, on a row that carries a category; NULL on a suppressed row. See §13. |
| 41 | safety_detail |
string |
(artifact ≥ 1.14.0) A compact JSON object, one entry per filter family that ran, carrying that filter's status, score, scale, theme and version — a family reporting not_assessed is retained, so a populated detail is not evidence that anything assessed the domain. NULL on a suppressed row, and on a row where no verdict was materialized at all. See §13. |
| 42 | policy_confidence_basis |
string |
(artifact ≥ 1.14.0) What the row's number IS: model_calibrated / model_raw / filter / rule / curated / carry_over / name_only (artifact ≥ 1.15.0) / none. Closed enum; if you validate it against a strict enum, add name_only before ingesting a 1.15.0 artifact. NULL only where the category block is suppressed. See §11. |
| 43 | policy_group_set |
list<string> |
(artifact ≥ 1.14.0) policy_group plus every group the pack declares it is blocked WITH, in pack order — one list to match instead of reconstructing the union. NULL exactly where policy_group is. |
| 44 | language_primary |
string |
(artifact ≥ 1.14.0) The BCP-47 primary subtag of column 14, e.g. zh for zh-CN / ZH_TW / zh-Hans-CN. Derived; column 14 is never rewritten beyond its whitespace. From 1.17.1 always a two- or three-letter code or NULL: NULL where no language was observed or where column 14 does not begin with a language code (§7). |
| 45 | content_purpose_id |
string |
(artifact ≥ 1.17.0) 1020 — the IAB Descriptive Vector id of Content Purpose > Informational > News — on the domains of a curated list of known news publishers; NULL on every other row. NULL means unknown, never "not news". Not a Content Taxonomy id: it never appears in iab_id or openrtb_cat. See §14. |
| 46 | content_purpose_source |
string |
(artifact ≥ 1.17.0) Which rule set column 45: curated (that curated list) — the only value the feed emits; schema_org is reserved. NULL exactly where column 45 is NULL. See §14. |
| 47 | role |
string |
(artifact ≥ 1.18.0) The role axis: what kind of thing the domain is, separately from what its content is about. infrastructure on a role-lane row — a live DNS zone with no website at its apex or www, shipped in the uncategorizable lane with no category — and NULL on every other row. Only infrastructure ships at 1.18.0; later releases may add values. Never a content category. See §15. |
| 48 | role_confidence |
float32 |
(artifact ≥ 1.18.0) NULL on every row at 1.18.0, role-lane rows included: a rule, not a model, makes the role-lane claim, so there is no probability to publish. |
| 49 | role_status |
string |
Never NULL (artifact ≥ 1.18.0). labeled on a role-lane row — its role columns are populated — and not_assessed on every other row. labeled does not make the row a labelled one: a role-lane row carries no category. See §15. |
| 50 | role_model_version |
string |
(artifact ≥ 1.18.0) The rule, and the version of its code, that made the role claim; NULL exactly where role is NULL. It changes whenever that code does, so compare it whole, never parse it, and do not pin a filter to it — filter on role and role_basis. |
| 51 | role_basis |
string |
(artifact ≥ 1.18.0) What the role claim rests on: dns_no_website, a DNS probe (§15) — the only value the feed emits; content_model and vendor_rule are reserved for later writers. NULL exactly where role is NULL. |
What domain holds. Normally the registrable domain, but not on every row: some rows are a host below a registrable domain
(blog.example.com) or a bare public suffix. Look a name up as written first, and fall back
to its registrable domain's row only when there is no exact match: a host is labelled from
its own pages, and it often carries a different tier1 from its parent.
Columns 24–26 were appended 2026-07-30 (feed_schema_version 1.6.0). Column 7
confidence is unchanged in both meaning and position. Columns 27–38 were appended
2026-09-01 (feed_schema_version 1.12.0); nothing before them changed. Four optional
manifest keys for the shippable-key coverage denominator were added 2026-09-03
(feed_schema_version 1.13.0, §10); no column changed. The first
PATCH bump, feed_schema_version 1.13.1, was specified to ship with the first
Sunday export after its deploy: parked-flagged brand-safety filter rows ship status-only and
the demoted_label_suppression manifest block gains counters (§10,
Website Categorization feed changelog); no column changed and no manifest key was removed.
No artifact reports 1.13.1: 1.13.2 superseded it before any export carried it — the
2026-09-06 set reports 1.13.0 and every export from 2026-09-08 reports 1.13.2, which carries
the 1.13.1 change as well. The rule behind that sequence applies to every future bump: a
daily delta exported between a deploy and the next Sunday snapshot is stamped with the new
version too — both kinds are stamped from the same constant — so consume a full snapshot at
the new version before relying on those deltas, because rows the new build masks do not
cross the delta cutoffs. The second PATCH bump, feed_schema_version 1.13.2, ships with the first
≥ 1.13.2 export of either kind: every delta now carries the hand-vetted curated/* rows
as well as the exclusion that already kept a competing row out (§5,
Website Categorization feed changelog); no column changed and no manifest key was removed.
For the curated rows the delta between deploy and snapshot needs no such caveat — they are
not keyed on a cutoff, so they cross it and arrive with the first ≥ 1.13.2 delta. The 1.13.1
suppression change that 1.13.2 inherits is cutoff-keyed like any masking change, so a
1.13.0 snapshot plus 1.13.2 deltas still retains the categories it withdraws: consume a
full ≥ 1.13.2 snapshot (the 2026-09-13 set) before treating your copy as reflecting it.
Columns 39–44 are appended at feed_schema_version 1.14.0, whose first artifact is the
Thursday 2026-09-17 snapshot set (specified 2026-09-10; no export before that date carries
them); columns
1–38 are unchanged in name, position and type. 1.14.0 is a MINOR bump rather than a PATCH
because it does both things at once: it adds columns AND it moves the published row
population — the language-NULL guard (§10) starts suppressing
categories on rows that carried one at 1.13.2, column 8 becomes derived rather than the
stored cut, and the feed's policy pack moves to policy_v2. Read the
Website Categorization feed changelog entry for 1.14.0 before ingesting the first artifact at
this version, and ingest a full snapshot at 1.14.0 before relying on 1.14.0 deltas.
Columns 45–46 are appended at feed_schema_version 1.17.0, whose first artifact is the
Wednesday 2026-11-04 delta (announced 2026-11-02; nothing published before the Tuesday
2026-11-03 exporter deploy carries them); columns 1–44 are unchanged in name, position and type,
and no row moves in or out of the feed. Unlike 1.14.0, the deltas are enough to receive
them: the only rows that carry a value are the curated rows, and those ride every delta whole
(§14). The full-snapshot obligation 1.16.0 set for its
band re-rating still stands, and the 2026-10-11 set was the first to meet it.
The PATCH bump feed_schema_version 1.17.1 changes the values of two columns and nothing
else: language (column 14) has its whitespace normalised, and language_primary (column 44)
carries only two- or three-letter codes (§7,
Website Categorization feed changelog). No column, manifest key or row moves, and no label,
confidence, band, calibrated value or suppression changes. A row whose data did not change does
not ride a delta, so the corrected values reach a delta-only consumer with the first ≥ 1.17.1
snapshot.
Columns 47–51 are appended at feed_schema_version 1.18.0, whose first artifact is the
delta in folder delta_2026-11-13_100000Z (nothing published before the Friday 2026-11-13
exporter deploy carries them); columns 1–46 are unchanged in name, position and type. 1.18.0 is
a MINOR bump rather than a PATCH for the reason 1.14.0 was: it adds columns AND, once the role
lane is armed, it moves the published row population — domains in the top popularity bucket
that shipped no row and qualify arrive as role-lane rows, and a few rows move out of dead or
blocked into the same lane (§15). Until the lane is armed,
every row reads role_status = 'not_assessed' with the other four role columns NULL, so you
can add the five columns to the rows you already hold with exactly those values. To receive
the role-lane rows, ingest a full ≥ 1.18.0 set at or
after the lane is armed; from that set on, a later role write reaches you on an ordinary
delta. A role-lane row leaves on a delta only when the delta carries another row for its domain
— its category row, its curated row, or a status-only row for a parked page or a placeholder;
otherwise it leaves with the next full set (§15).
What
tier1..tier4mean. The stored label follows the ratified single-label convention, which is purpose-first: it names the site's primary purpose — what the site is for — rather than its topic, industry or audience, and a small set of rules (R2, R3, R5, R8 and R9) settles the recurring cases: a gambling home isAttractions > Casinos & Gambling(the harmful subset ships asSensitive Topics— see §5); a software, SaaS or hosting vendor isTechnology & Computingwhoever it sells to; a multi-brand store isShoppingwhile a brand's own store takes its vertical; consumer banking, BNPL, investing and pensions arePersonal Finance; a content platform takes the content type it carries, not its delivery mechanism. The convention is set by our labeling rubric; subject-matter and taxonomy-gap rules sit on top of it — news outlets takePolitics, since v3.1 has no News tier-1 — so the result is neither a pure topic label nor a pure purpose label. The reservedsite_purpose_*columns (27–35) were added so a separately modelled business-purpose answer could be carried without changing this one; eight of them stay NULL andsite_purpose_statuskeeps its documented values — see §11.
domain_status — the six lanesdomain_status is a crawl-observed status, and it is orthogonal to the content
category. It answers "what kind of thing is at this domain right now", not "what this
site is for".
There are exactly six values. They are closed — no other value ships.
| Value | Meaning |
|---|---|
active |
A live site with real content — including one that answered our conditional request with 304 Not Modified, confirming the content we hold is current (from the 2026-09-17 set). This is the only lane whose content category, where one is present, is a current one — but the lane does not guarantee one; see below. |
parked |
An affirmative parking signal — parking nameserver, marketplace landing page, for-sale page, or a capture that extracted to no readable text — the content fingerprint of a parking-provider redirect stub — except, from 2026-07-31, on a domain in popularity bucket 0 or 1 (§10; see blocked). |
inactive |
The domain resolves but serves no real content and shows no parking signal: placeholder, coming-soon, under-construction. DNS-dead domains are dead, not this. |
blocked |
A live site we could not read on this crawl — anti-bot challenge, 403/429/451/5xx, timeout (including a DNS timeout or server failure), connection or TLS error, or, on a domain in popularity bucket 0 or 1 (best rank 1,000,000 or better on Tranco, Majestic or Umbrella; §10; not applied when rank_ladder is absent), a page that gave us no readable text in a capture made from 2026-07-31 — usually one rendered entirely by JavaScript, a redirect or a bot interstitial. This is not parking and not death, with one exception: a minority of those empty captures are hosting placeholders or parking pages our crawler could not recognize (more often in bucket 1 than in bucket 0; see the changelog). The lane's share of the feed moves with the crawler's re-crawl cadence for blocked sites; the changelog records material moves. A 304 Not Modified answer to our conditional request is not blocked: the site confirmed the content we hold is current, so from the 2026-09-17 set the row is laned as a successful crawl is — active where a label is on file, uncategorizable where no label exists, and parked or blocked by the rule above where the content it confirms extracted to no readable text. From 1.18.0 a few rows leave this lane for the role lane: where the crawl attempt failed and a newer DNS probe confirms a live, delegated zone with no website at its apex or www, the row ships uncategorizable with role = infrastructure (§15). |
dead |
The name does not resolve (a definitive NXDOMAIN or no-address answer, in crawls from 2026-09-23), or 404. Not a website. From 1.18.0 a row on either of those answers — never a 404 — can ship uncategorizable with role = infrastructure instead, where our DNS probe has confirmed a live, delegated zone with no website at its apex or www (§15). |
uncategorizable |
Crawled successfully but no label could be assigned; or the underlying crawl status was not recognized; or — from 1.18.0 — a live DNS zone with no website at its apex or www, which carries role = infrastructure (§15). Read role to tell them apart. The first two are a deliberate fail-loud bucket, never a silent default. |
Known issue (October 2026): some live sites ship as parked or inactive. A site whose
page our crawler captured empty — often one that builds its page with JavaScript — can carry
the parking-stub fingerprint and ship parked, with its label withheld (see
parked_stub_suppression in §10). From the 2026-11-03 deploy, on a domain
ranked in the top 1M (popularity bucket 0 or 1) such a capture made from 2026-07-31 ships
blocked instead, and the 2026-11-08 set is the first to carry every such row. Such a capture made
before 2026-07-31 stays parked until a crawl fetches the page again (a 304 Not Modified answer
keeps the earlier capture), and below that rank such rows stay parked.
Separately, some live sites were laned inactive by an earlier placeholder rule that
misread them, and they stay inactive, with their label in last_known_*, until the crawler reads them again. Do not
cull a domain on parked alone, nor on the real-time API's is_parked, which is true for
inactive answers too.
Three points that matter for integration:
blocked is not parked. A blocked domain is a real, live website that
defended itself against our crawler. Treating blocked as parked or dead will
discard live sites. This separation is a deliberate design invariant.uncategorizable is a fail-loud bucket, not a category. It means "we will not
guess". It covers a clean crawl that produced no label and an inventory status our
mapping does not recognize. It is never used to quietly absorb an unknown into
active. From 1.18.0 it also holds a classified subset — live DNS zones with no
website at their apex or www, each carrying role = infrastructure — which is still
not a category: read role to tell it apart, and filter role IS NULL to leave it out
(§15).active rows carry a current content category — but not every active row
carries one. On parked, inactive and dead rows every content column (columns
2–8 and 10–14, plus openrtb_cat, the reserved site_purpose_* label columns and the
three policy_* columns) is NULL by design. That is not missing data — see
§9. The implication runs one way only:
active says the crawl found a live site, not that a label was produced or kept for
it. A live site can reach the export with every content column NULL — because it was
never labelled, or because one of the suppression guards withheld the label it had
(the stub, no-evidence, template and demoted-label guards described below and in
§10). Test the columns you need
(tier1 IS NOT NULL); do not infer them from domain_status = 'active'.blocked rows are the exception worth knowing: they keep the category from a
previous successful crawl, because the site is presumed live and unchanged. That label
is real but was not re-verified on this crawl. Do not drop blocked rows as
category-free — after parked, this is the largest non-active lane. Most of those carried
labels are DOUBTFUL, as most labels on active rows are, and a model-adjudicated read of the
2026-10-18 set found them less often acceptable than labels on active rows, on a small sample: filter them on
calibrated_confidence_tier rather than keeping or dropping the lane wholesale.
uncategorizable rows likewise keep whatever label is already on file, if there is one:
where the lane was reached because the crawl status was not recognized, an existing label
is retained rather than blanked.
A role-lane row (from 1.18.0) is the exception: it never carries a label. Where one is on
file for its domain, the role lane keeps it out of the artifact, and the manifest counts
those rows in row_count_role_lane_category_superseded (§10).
From artifact 1.15.0 a blocked or uncategorizable row may carry a tier-1 category that was
never read from the page: the name-only LLM lane (model_name = llm-name/consensus_v1,
policy_confidence_basis = name_only; see §5 and §11) labels head
domains (Tranco rank ≤ 100,000) the crawler cannot read from the domain name, where two model
families agree. The torso tranche (rank 100,000–1,000,000) this lane first reserved was measured
and withheld for now — see the 1.15.0 notice in the changelog — and nothing ranked past
1,000,000 is ever labelled this way. The lane the crawl decided is unchanged by it: the label is
additional information about the domain, not evidence that the site was read. (When we checked on
2026-09-30, a small number of these rows, whose crawl produced no label from the page, would ship in the lane a label read from the page would give them: some as active, with the label, and others as parked, without it. From delta_2026-10-02_100000Z and the 2026-10-04 set they keep the lane the
crawl decided; the changelog's 2026-10-01 and 2026-10-02 entries have the detail. To find a
name-only label, test policy_confidence_basis or model_name, not domain_status.) On parked,
inactive and dead rows such a label is withheld like any other, with no last_known_* block.
One exception applies on every lane, including blocked and uncategorizable:
rows whose latest capture extracted to no readable text (see
parked_stub_suppression in §10) ship with NULL category columns
wherever they land. No label can be checked against such a capture — see the
no-content note in §5 — so it is withheld rather than retained,
with no last_known_* block either. Where the crawl itself succeeded and the label is not
a name-only one (which never decides the lane; see above), such a capture ships parked (it
is read as a parking-provider redirect stub), except that one made from 2026-07-31 on a domain
in popularity bucket 0 or 1 (§10) ships blocked (read as a live site we could not read; a
minority are hosting placeholders or parking pages). An export
without the buckets (rank_ladder absent) makes no such exception. So a blocked row with a NULL
category is either a site that was never labeled or one of these rows; in both cases the
NULL is deliberate.
curated/* rows: active is asserted, not crawl-observedRows whose model_name begins with curated/ (see §5) are
the one place where domain_status is not derived from a crawl. They always carry
active, because a human placed the domain on the curated list and that list is the
same first-tier lookup the real-time API answers from — so the feed and the API give
you the same answer for those domains. The list includes CDN, DNS and API zones that serve
no homepage (akamaiedge.net, cloudfront.net and googleapis.com, for example): their
active is the curated assertion, not a sign that a website answered.
Two consequences, stated plainly rather than hidden:
blocked verdict.status_observed_at on these rows still dates the last crawl attempt, and is
NULL where the domain has no crawl record at all. It is the one column on a
curated/* row that reports observation rather than assertion, so do not read it as
dating the active lane.Curated rows are a small share of the feed. If you need to exclude or specially handle them,
filter on model_name LIKE 'curated/%'; manifest.json publishes the count as
row_count_curated.
www. hosts: a curated domain ships once, under its bare (no-www.) name.
The www. spelling of a curated domain is not published at all — not as a curated row
and not as a model-produced one — so it cannot contradict the bare host. If your
lookup key can carry a www. prefix, strip it before matching, which is what our
real-time API does with the requested domain.
The feed carries two dates and they mean different things. This is the single most common misreading, so it is stated plainly:
| Column | What it dates |
|---|---|
classified_at |
When we processed the domain and produced or served the label. |
status_observed_at |
When the crawler observed the domain_status on that row. |
Neither is "when the website's content changed." Website Categorization does not publish a content-change date, and neither column should be used as a proxy for one.
classified_at records our processing, not the site's. In our own code
comments: "classified_at only records when webcat served or cached the verdict."
A recent classified_at does not mean the page was recently re-fetched.status_observed_at is the crawl-attempt timestamp behind domain_status. It is
populated on every row the crawler has observed, in every lane, and the exceptions are
visible rather than silent: a row whose inventory carries a status but no attempt
timestamp ships with it null and is counted, per lane,
in the manifest's row_count_by_status_age null bucket (artifact ≥ 1.14.0), and a
curated row carries it only where the inventory has a row for that domain. Use it to
judge how old a parked / blocked / dead verdict is.status_observed_at dates our DNS probe, not a crawl
attempt: it is the latest probe that confirmed the role claim, because the probe, not a
crawl, decided that row's lane (§15). classified_at is
NULL there, since no label was produced. Contrast a curated row, where
status_observed_at keeps dating the last crawl attempt
(§3).last_known_classified_at dates that
label, and is generally older than both of the above.row_count_by_status_age buckets every row, per
lane, by status_age_reference_date minus the UTC calendar day of
status_observed_at, and stale_fraction_active_gt_90d is the active lane's share
observed more than 90 days before that reference — see §10. The
reference is the artifact's key date (a snapshot's date, a delta's cutoff day),
never the time the export happened to run, so a re-published artifact reports the
same ages.classified_at is usually the later of the two. Outside the top 1M, a re-crawl
re-labels a row only when its content hash changed.We are here to listen. For a quick response, please select your request type. By submitting a request, you agree to our Terms of Service and Privacy Policy.