We are excited to announce that we just added 225 top-level domains (TLDs) to the list of web crawling! Below is the full list. The domain count per TLD represents the domains registered according to DomainTools. We group multiple TLDs into “buckets” to make it manageable – you can select these buckets in the Advanced menu next to the search button on intelx.io!
| TLD | Country | Bucket | Domain Count per TLD | Domain Count per Bucket |
|---|---|---|---|---|
| AT | Austria | web.public.de | 1,327,983 | 18,672,594 |
| DE | Germany | 15,106,819 | ||
| LU | Luxembourg | 89,822 | ||
| LI | Lichtenstein | 55,656 | ||
| CH | Switzerland | 2,092,314 | ||
| UA | Ukraine | web.public.ru | 566,476 | 5,599,820 |
| KZ | Kazakhstan | 148,691 | ||
| RU | Russia | 4,884,653 | ||
| KP | web.public.kp | 37 | 37 | |
| COM | web.public.com | 150,953,990 | 150,953,990 | |
| BIT | web.public.peer | |||
| BAZAR | ||||
| GOV | web.public.gov | 6,324 | 6,324 | |
| ORG | web.public.org | 10,317,554 | 10,317,554 | |
| NET | web.public.net | 13,291,542 | 13,291,542 | |
| INFO | web.public.info | 4,147,726 | 4,147,726 | |
| EU | web.public.eu | 3,597,950 | 3,597,950 | |
| CN | China | web.public.cn | 13,982,944 | 13,982,944 |
| UK | United Kingdom | web.public.nord | 10,300,887 | 14,713,717 |
| IE | Ireland | 288,539 | ||
| IS | Iceland | 65,529 | ||
| DK | Denmark | 1,306,074 | ||
| NO | Norway | 754,552 | ||
| FI | Finland | 483,625 | ||
| SE | Sweden | 1,508,703 | ||
| GL | Greenland | 5,808 | ||
| AD | Andorra | web.public.we | 2,053 | 15,862,803 |
| ES | Spain | 1,942,335 | ||
| FR | France | 3,593,494 | ||
| IT | Italy | 3,020,524 | ||
| MC | Monaco | 3,927 | ||
| MT | Malta | 15,515 | ||
| NL | Netherlands | 5,371,427 | ||
| BE | Belgium | 1,586,787 | ||
| PT | Portugal | 324,124 | ||
| SM | San Marino | 2,213 | ||
| VA | Vatican | 404 | ||
| AL | Albania | web.public.cee | 21,749 | 7,762,975 |
| BA | Bosnia & Herzigovina | 23,143 | ||
| BG | Bulgaria | 63,485 | ||
| BY | Belarus | 129,091 | ||
| CZ | Czech Republic | 1,271,461 | ||
| EE | Estonia | 128,346 | ||
| GR | Greece | 420,271 | ||
| HR | Croatia | 103,458 | ||
| HU | Hungary | 797,657 | ||
| LT | Lithuania | 198,736 | ||
| LV | Latvia | 113,289 | ||
| MD | Moldova | 25,032 | ||
| ME | Montenegro | 1,025,732 | ||
| MK | North Macedonia | 29,352 | ||
| PL | Poland | 2,240,128 | ||
| RO | Romania | 518,760 | ||
| RS | Serbia | 112,901 | ||
| SI | Slovenia | 124,981 | ||
| SK | Slovakia | 415,403 | ||
| AG | Antigua and Barbuda | web.public.ams | 19,104 | 13,829,992 |
| AI | Anguilla | 83,644 | ||
| AR | Argentina | 482,912 | ||
| AW | Aruba | 697 | ||
| BB | Barbados | 1,592 | ||
| BM | Bermuda | 3,106 | ||
| BO | Bolivia | 12,293 | ||
| BQ | Caribbean Netherlands | – | ||
| BR | Brazil | 3,837,643 | ||
| BS | Bahamas | 2,486 | ||
| BZ | Belize | 28,272 | ||
| CA | Canada | 2,871,681 | ||
| CL | Chile | 514,302 | ||
| CO | Colombia | 2,905,726 | ||
| CR | Costa Rica | 16,413 | ||
| CU | Cuba | 1,939 | ||
| CW | Curaçao | 875 | ||
| DM | Dominican Republic | 2,384 | ||
| DO | Dominican Repuplic | 26,031 | ||
| EC | Ecuador | 43,454 | ||
| FK | Falkland Islands | 145 | ||
| GD | Grenada | 2,456 | ||
| GF | French Guiana | 933 | ||
| GP | Guadeloupe | 2,173 | ||
| GT | Guatemala | 18,202 | ||
| GY | Guyana | 3,119 | ||
| HN | Honduras | 7,731 | ||
| HT | Haiti | 2,792 | ||
| JM | Jamaica | 7,386 | ||
| KN | Saint Kitts and Nevis | 829 | ||
| KY | Cayman Islands | 6,480 | ||
| LC | Saint Lucia | 3,729 | ||
| MQ | Martinique | 1,043 | ||
| MS | Montserrat | 8,970 | ||
| MX | Mexico | 929,819 | ||
| NI | Nicaragua | 4,695 | ||
| PA | Panama | 11,365 | ||
| PE | Peru | 112,857 | ||
| PR | Puerto Rico | 4,682 | ||
| PY | Paraguay | 22,897 | ||
| SR | Suriname | 3,165 | ||
| SV | El Salvador | 8,619 | ||
| SX | Sint Maarten | 2,761 | ||
| TC | Turks and Caicos Islands (UK) | 7,415 | ||
| TT | Trinidad and Tabago | 3,946 | ||
| US | United States | 1,661,201 | ||
| UY | Uruguay | 76,013 | ||
| VC | Saint Vincent and the Grenadines | Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â | Â 27,649 | |
| VE | Venezuela | 26,307 | ||
| VG | US Virgin Islands | 4,481 | ||
| VI | British Virgin Islands | 1,578 | ||
| AC | Ascension | web.public.af | 8,294 | 16,663,118 |
| DZ | Algeria | 9,880 | ||
| AO | Angola | 6,618 | ||
| BJ | Benin | 1,396 | ||
| BW | Botswana | 8,242 | ||
| BF | Burkina Faso | 1,643 | ||
| BI | Burundi | 2,738 | ||
| CM | Cameroon | 49,827 | ||
| CV | Cape Verde | 2,452 | ||
| CF | Central African Republic | 3,918,575 | ||
| TD | Chad | 582 | ||
| KM | Comoros | 141 | ||
| CD | Democratic Republic of the Congo | Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â | Â 6,090 | |
| DJ | Djibouti | 7,764 | ||
| EG | Egypt | 7,776 | ||
| GQ | Equatorial Guinea | 3,090,022 | ||
| ER | Eritrea | 94 | ||
| SZ | Eswatini | 1,790 | ||
| ET | Ethiopia | 2,583 | ||
| GA | Gabon | 4,227,941 | ||
| GM | Gambia | 1,451 | ||
| GH | Ghana | 2,966 | ||
| GN | Guinea | 416 | ||
| GW | Guinea-Bissau | 238 | ||
| CI | Ivory Coast | 9,447 | ||
| KE | Kenya | 83,325 | ||
| LS | Lesotho | 1,615 | ||
| LR | Liberia | 403 | ||
| LY | Libya | 17,330 | ||
| MG | Madagascar | 4,476 | ||
| MW | Malawi | 2,071 | ||
| ML | Mali | 3,511,703 | ||
| MR | Mauritania | 861 | ||
| MU | Mauritius | 9,938 | ||
| YT | Mayotte (France) | 3,689 | ||
| MA | Morocco | 83,579 | ||
| MZ | Mozambique | 7,129 | ||
| NA | Namibia | 4,564 | ||
| NE | Niger | 811 | ||
| NG | Nigeria | 119,413 | ||
| CG | Republic of the Congo | 1,650 | ||
| RW | Rwanda | 4,772 | ||
| RE | Réunion (France) | 28,150 | ||
| SH | Saint Helena | 14,350 | ||
| SN | Senegal | 5,935 | ||
| SC | Seychelles | 5,043 | ||
| SL | Sierra Leone | 1,992 | ||
| SO | Somalia | 11,183 | ||
| ZA | South Africa | 1,234,532 | ||
| SS | South Sudan | 26 | ||
| SD | Sudan | 3,520 | ||
| ST | São Tomé and PrÃncipe | 13,920 | ||
| TZ | Tanzania | 18,711 | ||
| TG | Togo | 1,657 | ||
| TN | Tunisia | 43,144 | ||
| UG | Uganda | 7,852 | ||
| EH | Western Sahara | – | ||
| ZM | Zambia | 4,152 | ||
| ZW | Zimbabwe | 42,656 | ||
| AF | Afghanistan | web.public.mea | 5,947 | 9,887,829 |
| AM | Armenia | 33,050 | ||
| AZ | Azerbaijan | 26,727 | ||
| BH | Bahrain | 5,040 | ||
| BD | Bangladesh | 25,528 | ||
| BT | Bhutan | 1,824 | ||
| BN | Brunei | 1,366 | ||
| KH | Cambodia | 3,208 | ||
| CY | Cyprus | 17,765 | ||
| GE | Georgia | 42,411 | ||
| HK | Hongkong | 232,505 | ||
| IN | India | 2,349,738 | ||
| ID | Indonesia | 375,112 | ||
| IR | Iran | 1,077,516 | ||
| IQ | Iraq | 2,029 | ||
| IL | Israel | 250,425 | ||
| JP | Japan | 1,506,391 | ||
| JO | Jordan | 4,521 | ||
| KW | Kuwait | 3,943 | ||
| KG | Kyrgyzstan | 11,123 | ||
| LA | Laos | 173,595 | ||
| LB | Lebanon | 4,539 | ||
| MO | Macao | 2,866 | ||
| MY | Malaysia | 247,237 | ||
| MV | Maldives | 5,108 | ||
| MN | Mongolia | 17,124 | ||
| MM | Myanmar | 4,202 | ||
| NP | Nepal | 78,200 | ||
| OM | Oman | 3,815 | ||
| PK | Pakistan | 93,709 | ||
| PH | Philippines | 116,685 | ||
| QA | Qatar | 19,218 | ||
| SA | Saudi Arabia | 64,201 | ||
| SG | Singapore | 175,686 | ||
| KR | Republic of Korea | 973,409 | ||
| LK | Sri Lanka | 68,505 | ||
| PS | State of Palestine | 7,639 | ||
| SY | Syria | 10,577 | ||
| TW | Taiwan | 686,428 | ||
| TJ | Tajikistan | 6,923 | ||
| TH | Thailand | 71,314 | ||
| TL | Timor-Leste | 1,717 | ||
| TR | Turkey | 398,104 | ||
| TM | Turkmenistan | 18,447 | ||
| AE | United Arab Emirates | 157,743 | ||
| UZ | Uzbekistan | 83,594 | ||
| VN | Vietnam | 419,911 | ||
| YE | Yemen | 1,164 | ||
| AS | American Samoa (US) | web.public.oc | 14,474 | 5,530,037 |
| AU | Australia | 2,982,455 | ||
| CC | Cocos (Keeling) Islands (Australia) | 719,256 | ||
| CK | Cook Islands (NZ) | 1,308 | ||
| FM | Federated States of Micronesia | 17,024 | ||
| FJ | Fiji | 3,317 | ||
| PF | French Polynesia (France) | 1,827 | ||
| GU | Guam (US) | 62 | ||
| KI | Kiribati | 553 | ||
| MH | Marshall Islands | 10 | ||
| NR | Nauru | 299 | ||
| NC | New Caledonia (France) | 5,891 | ||
| NZ | New Zealand | 686,488 | ||
| NF | Norfolk Island (Australia) | 1,074 | ||
| MP | Northern Mariana Islands (US) | 653 | ||
| PW | Palau | 400,979 | ||
| PG | Papua New Guinea | 2,147 | ||
| PN | Pitcairn Islands (UK) | 749 | ||
| WS | Samoa | 165,404 | ||
| SB | Solomon Islands | 1,619 | ||
| TO | Tonga | 25,215 | ||
| TV | Tuvalu | 495,755 | ||
| VU | Vanuatu | 1,471 | ||
| WF | Wallis and Futuna (France) | 2,007 | ||
| AQ | Antarctica | web.public.aq | 94 | 94 |
It is important to note that a minority of above mentioned TLDs, while formally added, are remaining in wait state because they do not meet our criteria of quality control. In this previous blog post Top-level domain considerations we explained why we are not accepting any TLD that exists (tl;dr: some TLDs such as “ZIP” don’t have enough domains to be worth in comparison to potential false positives).
Previously we were only crawling these TLDs:
Intelligence X supports Peernet – Founder’s Statement I am excited to announce Peernet, a decentralized network that allows sharing of data freely without censorship and restrictions. Here is the pitch deck: https://peernet.org/dl/Peernet%20Deck.pdf Peernet is making quick progress from its inception as I am finalizing the whitepaper and developing the core library. I would like to
February 2021: Launch of the European Internet Archive The European Internet Archive just launched! 🎉🥳 ➡ https://archive.eu/ 225 TLDs added to the list of web crawling We have added 225 top-level domains (TLDs) to the list of web crawling. Find the full list and how we are categorizing them in this blog post. Our dataset
We have added the new category “Bot Logs”. It contains data collected by and leaked from viruses such as Azorult. Such data is often sold on marketplaces such as the Genesis Market. We decided to index such data into this new category to help filtering out relevant results. You can find this new category in