Adding 225 TLDs to the list of web crawling
We are excited to announce that we just added 225 top-level domains (TLDs) to the list of web crawling! Below is the full list. The domain count per TLD represents the domains registered according to DomainTools. We group multiple TLDs into “buckets” to make it manageable – you can select these buckets in the Advanced menu next to the search button on intelx.io!
| TLD | Country | Bucket | Domain Count per TLD | Domain Count per Bucket |
|---|---|---|---|---|
| AT | Austria | web.public.de | 1,327,983 | 18,672,594 |
| DE | Germany | 15,106,819 | ||
| LU | Luxembourg | 89,822 | ||
| LI | Lichtenstein | 55,656 | ||
| CH | Switzerland | 2,092,314 | ||
| UA | Ukraine | web.public.ru | 566,476 | 5,599,820 |
| KZ | Kazakhstan | 148,691 | ||
| RU | Russia | 4,884,653 | ||
| KP | web.public.kp | 37 | 37 | |
| COM | web.public.com | 150,953,990 | 150,953,990 | |
| BIT | web.public.peer | |||
| BAZAR | ||||
| GOV | web.public.gov | 6,324 | 6,324 | |
| ORG | web.public.org | 10,317,554 | 10,317,554 | |
| NET | web.public.net | 13,291,542 | 13,291,542 | |
| INFO | web.public.info | 4,147,726 | 4,147,726 | |
| EU | web.public.eu | 3,597,950 | 3,597,950 | |
| CN | China | web.public.cn | 13,982,944 | 13,982,944 |
| UK | United Kingdom | web.public.nord | 10,300,887 | 14,713,717 |
| IE | Ireland | 288,539 | ||
| IS | Iceland | 65,529 | ||
| DK | Denmark | 1,306,074 | ||
| NO | Norway | 754,552 | ||
| FI | Finland | 483,625 | ||
| SE | Sweden | 1,508,703 | ||
| GL | Greenland | 5,808 | ||
| AD | Andorra | web.public.we | 2,053 | 15,862,803 |
| ES | Spain | 1,942,335 | ||
| FR | France | 3,593,494 | ||
| IT | Italy | 3,020,524 | ||
| MC | Monaco | 3,927 | ||
| MT | Malta | 15,515 | ||
| NL | Netherlands | 5,371,427 | ||
| BE | Belgium | 1,586,787 | ||
| PT | Portugal | 324,124 | ||
| SM | San Marino | 2,213 | ||
| VA | Vatican | 404 | ||
| AL | Albania | web.public.cee | 21,749 | 7,762,975 |
| BA | Bosnia & Herzigovina | 23,143 | ||
| BG | Bulgaria | 63,485 | ||
| BY | Belarus | 129,091 | ||
| CZ | Czech Republic | 1,271,461 | ||
| EE | Estonia | 128,346 | ||
| GR | Greece | 420,271 | ||
| HR | Croatia | 103,458 | ||
| HU | Hungary | 797,657 | ||
| LT | Lithuania | 198,736 | ||
| LV | Latvia | 113,289 | ||
| MD | Moldova | 25,032 | ||
| ME | Montenegro | 1,025,732 | ||
| MK | North Macedonia | 29,352 | ||
| PL | Poland | 2,240,128 | ||
| RO | Romania | 518,760 | ||
| RS | Serbia | 112,901 | ||
| SI | Slovenia | 124,981 | ||
| SK | Slovakia | 415,403 | ||
| AG | Antigua and Barbuda | web.public.ams | 19,104 | 13,829,992 |
| AI | Anguilla | 83,644 | ||
| AR | Argentina | 482,912 | ||
| AW | Aruba | 697 | ||
| BB | Barbados | 1,592 | ||
| BM | Bermuda | 3,106 | ||
| BO | Bolivia | 12,293 | ||
| BQ | Caribbean Netherlands | – | ||
| BR | Brazil | 3,837,643 | ||
| BS | Bahamas | 2,486 | ||
| BZ | Belize | 28,272 | ||
| CA | Canada | 2,871,681 | ||
| CL | Chile | 514,302 | ||
| CO | Colombia | 2,905,726 | ||
| CR | Costa Rica | 16,413 | ||
| CU | Cuba | 1,939 | ||
| CW | Curaçao | 875 | ||
| DM | Dominican Republic | 2,384 | ||
| DO | Dominican Repuplic | 26,031 | ||
| EC | Ecuador | 43,454 | ||
| FK | Falkland Islands | 145 | ||
| GD | Grenada | 2,456 | ||
| GF | French Guiana | 933 | ||
| GP | Guadeloupe | 2,173 | ||
| GT | Guatemala | 18,202 | ||
| GY | Guyana | 3,119 | ||
| HN | Honduras | 7,731 | ||
| HT | Haiti | 2,792 | ||
| JM | Jamaica | 7,386 | ||
| KN | Saint Kitts and Nevis | 829 | ||
| KY | Cayman Islands | 6,480 | ||
| LC | Saint Lucia | 3,729 | ||
| MQ | Martinique | 1,043 | ||
| MS | Montserrat | 8,970 | ||
| MX | Mexico | 929,819 | ||
| NI | Nicaragua | 4,695 | ||
| PA | Panama | 11,365 | ||
| PE | Peru | 112,857 | ||
| PR | Puerto Rico | 4,682 | ||
| PY | Paraguay | 22,897 | ||
| SR | Suriname | 3,165 | ||
| SV | El Salvador | 8,619 | ||
| SX | Sint Maarten | 2,761 | ||
| TC | Turks and Caicos Islands (UK) | 7,415 | ||
| TT | Trinidad and Tabago | 3,946 | ||
| US | United States | 1,661,201 | ||
| UY | Uruguay | 76,013 | ||
| VC | Saint Vincent and the Grenadines | 27,649 | ||
| VE | Venezuela | 26,307 | ||
| VG | US Virgin Islands | 4,481 | ||
| VI | British Virgin Islands | 1,578 | ||
| AC | Ascension | web.public.af | 8,294 | 16,663,118 |
| DZ | Algeria | 9,880 | ||
| AO | Angola | 6,618 | ||
| BJ | Benin | 1,396 | ||
| BW | Botswana | 8,242 | ||
| BF | Burkina Faso | 1,643 | ||
| BI | Burundi | 2,738 | ||
| CM | Cameroon | 49,827 | ||
| CV | Cape Verde | 2,452 | ||
| CF | Central African Republic | 3,918,575 | ||
| TD | Chad | 582 | ||
| KM | Comoros | 141 | ||
| CD | Democratic Republic of the Congo | 6,090 | ||
| DJ | Djibouti | 7,764 | ||
| EG | Egypt | 7,776 | ||
| GQ | Equatorial Guinea | 3,090,022 | ||
| ER | Eritrea | 94 | ||
| SZ | Eswatini | 1,790 | ||
| ET | Ethiopia | 2,583 | ||
| GA | Gabon | 4,227,941 | ||
| GM | Gambia | 1,451 | ||
| GH | Ghana | 2,966 | ||
| GN | Guinea | 416 | ||
| GW | Guinea-Bissau | 238 | ||
| CI | Ivory Coast | 9,447 | ||
| KE | Kenya | 83,325 | ||
| LS | Lesotho | 1,615 | ||
| LR | Liberia | 403 | ||
| LY | Libya | 17,330 | ||
| MG | Madagascar | 4,476 | ||
| MW | Malawi | 2,071 | ||
| ML | Mali | 3,511,703 | ||
| MR | Mauritania | 861 | ||
| MU | Mauritius | 9,938 | ||
| YT | Mayotte (France) | 3,689 | ||
| MA | Morocco | 83,579 | ||
| MZ | Mozambique | 7,129 | ||
| NA | Namibia | 4,564 | ||
| NE | Niger | 811 | ||
| NG | Nigeria | 119,413 | ||
| CG | Republic of the Congo | 1,650 | ||
| RW | Rwanda | 4,772 | ||
| RE | Réunion (France) | 28,150 | ||
| SH | Saint Helena | 14,350 | ||
| SN | Senegal | 5,935 | ||
| SC | Seychelles | 5,043 | ||
| SL | Sierra Leone | 1,992 | ||
| SO | Somalia | 11,183 | ||
| ZA | South Africa | 1,234,532 | ||
| SS | South Sudan | 26 | ||
| SD | Sudan | 3,520 | ||
| ST | São Tomé and Príncipe | 13,920 | ||
| TZ | Tanzania | 18,711 | ||
| TG | Togo | 1,657 | ||
| TN | Tunisia | 43,144 | ||
| UG | Uganda | 7,852 | ||
| EH | Western Sahara | – | ||
| ZM | Zambia | 4,152 | ||
| ZW | Zimbabwe | 42,656 | ||
| AF | Afghanistan | web.public.mea | 5,947 | 9,887,829 |
| AM | Armenia | 33,050 | ||
| AZ | Azerbaijan | 26,727 | ||
| BH | Bahrain | 5,040 | ||
| BD | Bangladesh | 25,528 | ||
| BT | Bhutan | 1,824 | ||
| BN | Brunei | 1,366 | ||
| KH | Cambodia | 3,208 | ||
| CY | Cyprus | 17,765 | ||
| GE | Georgia | 42,411 | ||
| HK | Hongkong | 232,505 | ||
| IN | India | 2,349,738 | ||
| ID | Indonesia | 375,112 | ||
| IR | Iran | 1,077,516 | ||
| IQ | Iraq | 2,029 | ||
| IL | Israel | 250,425 | ||
| JP | Japan | 1,506,391 | ||
| JO | Jordan | 4,521 | ||
| KW | Kuwait | 3,943 | ||
| KG | Kyrgyzstan | 11,123 | ||
| LA | Laos | 173,595 | ||
| LB | Lebanon | 4,539 | ||
| MO | Macao | 2,866 | ||
| MY | Malaysia | 247,237 | ||
| MV | Maldives | 5,108 | ||
| MN | Mongolia | 17,124 | ||
| MM | Myanmar | 4,202 | ||
| NP | Nepal | 78,200 | ||
| OM | Oman | 3,815 | ||
| PK | Pakistan | 93,709 | ||
| PH | Philippines | 116,685 | ||
| QA | Qatar | 19,218 | ||
| SA | Saudi Arabia | 64,201 | ||
| SG | Singapore | 175,686 | ||
| KR | Republic of Korea | 973,409 | ||
| LK | Sri Lanka | 68,505 | ||
| PS | State of Palestine | 7,639 | ||
| SY | Syria | 10,577 | ||
| TW | Taiwan | 686,428 | ||
| TJ | Tajikistan | 6,923 | ||
| TH | Thailand | 71,314 | ||
| TL | Timor-Leste | 1,717 | ||
| TR | Turkey | 398,104 | ||
| TM | Turkmenistan | 18,447 | ||
| AE | United Arab Emirates | 157,743 | ||
| UZ | Uzbekistan | 83,594 | ||
| VN | Vietnam | 419,911 | ||
| YE | Yemen | 1,164 | ||
| AS | American Samoa (US) | web.public.oc | 14,474 | 5,530,037 |
| AU | Australia | 2,982,455 | ||
| CC | Cocos (Keeling) Islands (Australia) | 719,256 | ||
| CK | Cook Islands (NZ) | 1,308 | ||
| FM | Federated States of Micronesia | 17,024 | ||
| FJ | Fiji | 3,317 | ||
| PF | French Polynesia (France) | 1,827 | ||
| GU | Guam (US) | 62 | ||
| KI | Kiribati | 553 | ||
| MH | Marshall Islands | 10 | ||
| NR | Nauru | 299 | ||
| NC | New Caledonia (France) | 5,891 | ||
| NZ | New Zealand | 686,488 | ||
| NF | Norfolk Island (Australia) | 1,074 | ||
| MP | Northern Mariana Islands (US) | 653 | ||
| PW | Palau | 400,979 | ||
| PG | Papua New Guinea | 2,147 | ||
| PN | Pitcairn Islands (UK) | 749 | ||
| WS | Samoa | 165,404 | ||
| SB | Solomon Islands | 1,619 | ||
| TO | Tonga | 25,215 | ||
| TV | Tuvalu | 495,755 | ||
| VU | Vanuatu | 1,471 | ||
| WF | Wallis and Futuna (France) | 2,007 | ||
| AQ | Antarctica | web.public.aq | 94 | 94 |
It is important to note that a minority of above mentioned TLDs, while formally added, are remaining in wait state because they do not meet our criteria of quality control. In this previous blog post Top-level domain considerations we explained why we are not accepting any TLD that exists (tl;dr: some TLDs such as “ZIP” don’t have enough domains to be worth in comparison to potential false positives).
Previously we were only crawling these TLDs:
- Germany DE, AT, LU, CH
- Russia: UA, KZ, RU
- North Korea: KP
- International: COM
- Decentralized TLDs: BIT, BAZAR
- US Government: GOV