Website Categorization — the categories you get, and how they are built

Companion page to the sample data and the feed data dictionary. Taxonomy version iab_v3.1 · revision 2026-08-13


1. One taxonomy, 704 categories, four levels deep

Every category we emit comes from the IAB Content Taxonomy v3.1, an industry-standard tree maintained by IAB Tech Lab. We do not invent categories and we do not mix taxonomies — a label is always a node in this one tree.

Level Categories Example
Tier 1 37 Technology & Computing
Tier 2 323 Technology & Computing > Computing
Tier 3 275 … > Computing > Computer Networking
Tier 4 69 … > Computer Networking > Network Security
Total 704

The complete list is available for download as webcat_iab_v3.1_categories.csv — one row per category, with its id, its parent, and its full tier path. That file is the authoritative answer to "exactly which categories can we expect": nothing outside it is ever emitted.

How deep a label goes is not a measure of how much the page supported: for allow/block policy, key on tier 1 (or tiers 1 and 2) and treat deeper tiers as descriptive detail.

The 37 tier-1 categories

Attractions                          Genres                   Politics
Automotive                           Healthy Living           Pop Culture
Books and Literature                 Hobbies & Interests      Real Estate
Business and Finance                 Holidays                 Religion & Spirituality
Careers                              Home & Garden            Science
Communication                        Law                      Sensitive Topics
Crime                                Medical Health           Shopping
Disasters                            Personal Celebrations    Sports
Education                              & Life Events          Style & Fashion
Entertainment                        Personal Finance         Technology & Computing
Events                               Pets                     Travel
Family and Relationships                                      Video Gaming
Fine Art                                                      War and Conflicts
Food & Drink

Five of these are terminal (no children): Communication, Crime, Disasters, Law, War and Conflicts.

Relevant to policy enforcement: Sensitive Topics

For access-control and web-filtering use cases, the Sensitive Topics tier-1 branch is usually the one that matters. It has eleven tier-2 children:

Adult & Explicit Sexual Content
Arms & Ammunition
Crime & Harmful Acts to Individuals, Society & Human Right Violations
Death, Injury, or Military Conflict
Debated Sensitive Social Issues
Hate Speech and Acts of Aggression
Illegal Drugs, Tobacco, eCigarettes, Vaping, Alcohol
Obscenity and Profanity
Online Piracy
Spam or Harmful Content
Terrorism

Gambling is carried separately at Attractions > Casinos & Gambling, with illegal-gambling markers routed to Sensitive Topics > Spam or Harmful Content.


2. Three ways the same category is expressed

This is the point that most often causes confusion on a first read of a sample. A row carries one category, written three different ways — they are representations, not three separate classifications:

Representation Column(s) Example
Human-readable path tier1, tier2, tier3, tier4 Attractions › Museums & Galleries
Canonical IAB id iab_id IAB-155
OpenRTB / AdCOM ids openrtb_cat (array) the same node in OpenRTB encoding (cattax = 9)

Pick whichever your systems consume. They always agree; if they ever disagree, that is a bug and we want to hear about it.

Two further columns look like categories but are not the current label:

  • last_known_tier1..4 — a historical label retained for a domain that is no longer active (parked, expired, unreachable). It answers "what was this site, before it went dark?" It is populated on a small fraction of rows and always carries its own as-of date in last_known_classified_at.
  • confidence_tier / calibrated_confidence_tier — confidence bands, not categories. See §4.

3. How a label is produced

  domain
    │
    ├─ 1. Crawl            fetch the site; record DNS + HTTP outcome
    │                      → sets domain_status (the lane: active, parked, inactive,
    │                        blocked, dead, uncategorizable)
    │
    ├─ 2. Extract          pull the page's meaningful text; detect language
    │
    ├─ 3. Classify         route to the appropriate component (see below)
    │                      → tier1..tier4, iab_id, confidence, model_name
    │
    └─ 4. Calibrate        map the raw score onto an estimate that a strict blind judge
                           picks the same tier1
                           → calibrated_confidence, calibrated_confidence_tier

Only active domains with usable content reach step 3, apart from the name-only lane, which can label a blocked or uncategorizable Tranco top-100,000 domain from its name alone. A parked or dead domain gets a status row with no category — deliberately, rather than a guess.

The components that can decide a label

Website Categorization is not a single model. Nine families of component can produce a row, and the model_name column always tells you which one did:

Family What it is
Trained classifiers The main path — an ensemble of fine-tuned models (ensemble-5en-3nonen: English and non-English members voting together), plus a distilled student/teacher cascade for harder pages.
LLM teacher A large language model escalated to when the ensemble is unsure.
Sensitive-content filter A dedicated brand-safety model; always lands on Sensitive Topics.
Gambling filter A keyword-density rule for casino/betting sites.
Adult-content filter A multilingual rule layer that overrides a benign-looking label on an adult site.
Curated lists & rules Hand-maintained entries and TLD/schema rules for domains we want pinned.
Legacy upstream fallback A crosswalked label from the previous-generation WhoisXML categorization, where nothing newer exists.
Vendor-rule flip A deterministic rule that moves a software, SaaS, hosting, IT-services or telecom vendor from one of three vendor-heavy Business and Finance cells to Technology & Computing, keeping the classifier's own score.
Name-only LLM lane For high-traffic domains the crawler cannot read, a tier-1 category inferred from the domain name by two language models that agree; never read from the page.

The tier column (0–3) records how the row was decided, and is the conflict priority — higher wins:

tier Meaning
0 Status-only row (parked/inactive/dead). No content classification.
1 Bulk classification — the standard automated path.
2 Cascade — escalated to a stronger model.
3 Human-reviewed.

tier is provenance, not taxonomy depth. Taxonomy depth is the tier1..tier4 name columns. The collision of terms is unfortunate and we are stuck with it for schema compatibility.


4. Reading the confidence columns correctly

There are two, and they are not interchangeable.

confidence is the raw score from whichever component produced the row. Because nine different families write this column, the number does not mean the same thing on every row — a trained classifier writes a model probability, the gambling filter writes a keyword-density ratio, a curated rule writes a fixed trust value. We publish each component's own score unchanged rather than silently rescaling them onto a common axis.

Practical consequence: do not rank or threshold on confidence across the whole feed without also filtering on model_name. If you do, you are mixing eight scales.

calibrated_confidence is the column to use for a uniform cutoff. It is an estimate of the probability that a strict blind judge, reading the page, assigns the same tier1, fitted against judged samples, with bands in calibrated_confidence_tier (PROBABLE / TOSSUP / DOUBTFUL). The fitted map is versioned; manifest.json records which one produced a given snapshot, so you can tell whether two deliveries are comparable. The target is strict, so on average, though not in its top band, the value understates how often a label is acceptable to a reader: read it as a reliability ranking and a strict-match estimate, not as the chance that a label is usable.


5. What is not categorized, and why

A meaningful share of any domain corpus has no content to classify. We assign a status lane rather than fabricating a label — but only three of the six lanes are nulled:

Lane Meaning Carries a category?
active Live site with content. Usually
blocked A real, live site; anti-bot or geo restrictions blocked the fetch. Usually — an existing label is retained
uncategorizable Fail-loud bucket: crawled but no label could be assigned, or the crawl status was not recognized. Usually — an existing label is retained
parked Registered but showing a placeholder/for-sale page. Never
inactive Resolves but serves nothing usable. Never
dead Does not resolve. Never

Known issue (October 2026): some live sites ship as parked or inactive. The feed data dictionary describes it.

blocked and uncategorizable rows keep whatever label is already on file rather than being blanked — losing the ability to fetch a site today is not a reason to discard what we already knew about it.

So there are two different questions here, with different answers:

  • "Do you classify live sites?" — only active rows carry a current content category, but not every active row carries one.
  • "What share of delivered rows carry a category?" — lower, because parked, inactive and dead domains are present and, by design, uncategorized. Parked alone is roughly a quarter of the feed.

A completeness figure should always state its denominator. Reading it against the active lane alone in fact understates coverage, because it discards several million categorized blocked and uncategorizable rows.


6. Where to go next

  • webcat_iab_v3.1_categories.csv — every category, with ids and paths. Two rows list the parent their tier path implies, where IAB's published Parent column skips a level: IAB-376 Public Radio sits under IAB-371 Talk Radio, and IAB-497 Horse Racing under IAB-496 Equine Sports.
  • Feed data dictionary — the full 44-column contract, column by column, including null semantics, encoding, and the manifest.json schema.
  • Live taxonomy endpoint — GET /api/v4/taxonomy returns the same 704 categories as JSON. Its parent_id follows IAB's published Parent column, so for IAB-376 and IAB-497 it gives the tier-1 as the parent.

Questions on any column are welcome — particularly if something in the sample reads ambiguously. Several columns in this feed exist because an earlier reader asked exactly that kind of question.