Adding 225 TLDs to the list of web crawling

Published on January 28, 2021 by

We are excited to announce that we just added 225 top-level domains (TLDs) to the list of web crawling! Below is the full list. The domain count per TLD represents the domains registered according to DomainTools. We group multiple TLDs into “buckets” to make it manageable – you can select these buckets in the Advanced menu next to the search button on intelx.io!

TLDCountryBucketDomain Count per TLDDomain Count per Bucket
ATAustriaweb.public.de                                            1,327,983                                         18,672,594
DEGermany                                          15,106,819 
LULuxembourg                                                   89,822 
LILichtenstein                                                   55,656 
CHSwitzerland                                             2,092,314 
UAUkraineweb.public.ru                                               566,476                                            5,599,820
KZKazakhstan                                                148,691 
RURussia                                             4,884,653 
KP web.public.kp                                                          37                                                          37
COM web.public.com                                       150,953,990                                       150,953,990
BIT web.public.peer 
BAZAR    
GOV web.public.gov                                                    6,324                                                    6,324
ORG web.public.org                                         10,317,554                                         10,317,554
NET web.public.net                                         13,291,542                                         13,291,542
INFO web.public.info                                            4,147,726                                            4,147,726
EU web.public.eu                                            3,597,950                                            3,597,950
CNChinaweb.public.cn                                         13,982,944                                         13,982,944
UKUnited Kingdomweb.public.nord                                         10,300,887                                         14,713,717
IEIreland                                                288,539 
ISIceland                                                   65,529 
DKDenmark                                             1,306,074 
NONorway                                                754,552 
FIFinland                                                483,625 
SESweden                                             1,508,703 
GLGreenland                                                     5,808 
ADAndorraweb.public.we                                                    2,053                                         15,862,803
ESSpain                                             1,942,335 
FRFrance                                             3,593,494 
ITItaly                                             3,020,524 
MCMonaco                                                     3,927 
MTMalta                                                   15,515 
NLNetherlands                                             5,371,427 
BEBelgium                                             1,586,787 
PTPortugal                                                324,124 
SMSan Marino                                                     2,213 
VAVatican                                                         404 
ALAlbaniaweb.public.cee                                                  21,749                                            7,762,975
BABosnia & Herzigovina                                                   23,143 
BGBulgaria                                                   63,485 
BYBelarus                                                129,091 
CZCzech Republic                                             1,271,461 
EEEstonia                                                128,346 
GRGreece                                                420,271 
HRCroatia                                                103,458 
HUHungary                                                797,657 
LTLithuania                                                198,736 
LVLatvia                                                113,289 
MDMoldova                                                   25,032 
MEMontenegro                                             1,025,732 
MKNorth Macedonia                                                   29,352 
PLPoland                                             2,240,128 
RORomania                                                518,760 
RSSerbia                                                112,901 
SISlovenia                                                124,981 
SKSlovakia                                                415,403 
AGAntigua and Barbudaweb.public.ams                                                  19,104                                         13,829,992
AIAnguilla                                                   83,644 
ARArgentina                                                482,912 
AWAruba                                                         697 
BBBarbados                                                     1,592 
BMBermuda                                                     3,106 
BOBolivia                                                   12,293 
BQCaribbean Netherlands                                                             –   
BRBrazil                                             3,837,643 
BSBahamas                                                     2,486 
BZBelize                                                   28,272 
CACanada                                             2,871,681 
CLChile                                                514,302 
COColombia                                             2,905,726 
CRCosta Rica                                                   16,413 
CUCuba                                                     1,939 
CWCuraçao                                                         875 
DMDominican Republic                                                     2,384 
DODominican Repuplic                                                   26,031 
ECEcuador                                                   43,454 
FKFalkland Islands                                                         145 
GDGrenada                                                     2,456 
GFFrench Guiana                                                         933 
GPGuadeloupe                                                     2,173 
GTGuatemala                                                   18,202 
GYGuyana                                                     3,119 
HNHonduras                                                     7,731 
HTHaiti                                                     2,792 
JMJamaica                                                     7,386 
KNSaint Kitts and Nevis                                                         829 
KYCayman Islands                                                     6,480 
LCSaint Lucia                                                     3,729 
MQMartinique                                                     1,043 
MSMontserrat                                                     8,970 
MXMexico                                                929,819 
NINicaragua                                                     4,695 
PAPanama                                                   11,365 
PEPeru                                                112,857 
PRPuerto Rico                                                     4,682 
PYParaguay                                                   22,897 
SRSuriname                                                     3,165 
SVEl Salvador                                                     8,619 
SXSint Maarten                                                     2,761 
TCTurks and Caicos Islands (UK)                                                     7,415 
TTTrinidad and Tabago                                                     3,946 
USUnited States                                             1,661,201 
UYUruguay                                                   76,013 
VCSaint Vincent and the Grenadines                                                   27,649
VEVenezuela                                                   26,307 
VGUS Virgin Islands                                                     4,481 
VIBritish Virgin Islands                                                     1,578 
ACAscensionweb.public.af                                                    8,294                                         16,663,118
DZAlgeria                                                     9,880 
AOAngola                                                     6,618 
BJBenin                                                     1,396 
BWBotswana                                                     8,242 
BFBurkina Faso                                                     1,643 
BIBurundi                                                     2,738 
CMCameroon                                                   49,827 
CVCape Verde                                                     2,452 
CFCentral African Republic                                             3,918,575 
TDChad                                                         582 
KMComoros                                                         141 
CDDemocratic Republic of the Congo                                                     6,090
DJDjibouti                                                     7,764 
EGEgypt                                                     7,776 
GQEquatorial Guinea                                             3,090,022 
EREritrea                                                           94 
SZEswatini                                                     1,790 
ETEthiopia                                                     2,583 
GAGabon                                             4,227,941 
GMGambia                                                     1,451 
GHGhana                                                     2,966 
GNGuinea                                                         416 
GWGuinea-Bissau                                                         238 
CIIvory Coast                                                     9,447 
KEKenya                                                   83,325 
LSLesotho                                                     1,615 
LRLiberia                                                         403 
LYLibya                                                   17,330 
MGMadagascar                                                     4,476 
MWMalawi                                                     2,071 
MLMali                                             3,511,703 
MRMauritania                                                         861 
MUMauritius                                                     9,938 
YTMayotte (France)                                                     3,689 
MAMorocco                                                   83,579 
MZMozambique                                                     7,129 
NANamibia                                                     4,564 
NENiger                                                         811 
NGNigeria                                                119,413 
CGRepublic of the Congo                                                     1,650 
RWRwanda                                                     4,772 
RERéunion (France)                                                   28,150 
SHSaint Helena                                                    14,350 
SNSenegal                                                     5,935 
SCSeychelles                                                     5,043 
SLSierra Leone                                                     1,992 
SOSomalia                                                   11,183 
ZASouth Africa                                             1,234,532 
SSSouth Sudan                                                           26 
SDSudan                                                     3,520 
STSão Tomé and Príncipe                                                   13,920 
TZTanzania                                                   18,711 
TGTogo                                                     1,657 
TNTunisia                                                   43,144 
UGUganda                                                     7,852 
EHWestern Sahara                                                            –   
ZMZambia                                                     4,152 
ZWZimbabwe                                                   42,656 
AFAfghanistanweb.public.mea                                                    5,947                                            9,887,829
AMArmenia                                                   33,050 
AZAzerbaijan                                                   26,727 
BHBahrain                                                     5,040 
BDBangladesh                                                   25,528 
BTBhutan                                                     1,824 
BNBrunei                                                     1,366 
KHCambodia                                                     3,208 
CYCyprus                                                   17,765 
GEGeorgia                                                   42,411 
HKHongkong                                                232,505 
INIndia                                             2,349,738 
IDIndonesia                                                375,112 
IRIran                                             1,077,516 
IQIraq                                                     2,029 
ILIsrael                                                250,425 
JPJapan                                             1,506,391 
JOJordan                                                     4,521 
KWKuwait                                                     3,943 
KGKyrgyzstan                                                   11,123 
LALaos                                                173,595 
LBLebanon                                                     4,539 
MOMacao                                                     2,866 
MYMalaysia                                                247,237 
MVMaldives                                                     5,108 
MNMongolia                                                   17,124 
MMMyanmar                                                     4,202 
NPNepal                                                   78,200 
OMOman                                                     3,815 
PKPakistan                                                   93,709 
PHPhilippines                                                116,685 
QAQatar                                                   19,218 
SASaudi Arabia                                                   64,201 
SGSingapore                                                175,686 
KRRepublic of Korea                                                973,409 
LKSri Lanka                                                   68,505 
PSState of Palestine                                                     7,639 
SYSyria                                                   10,577 
TWTaiwan                                                686,428 
TJTajikistan                                                     6,923 
THThailand                                                   71,314 
TLTimor-Leste                                                     1,717 
TRTurkey                                                398,104 
TMTurkmenistan                                                   18,447 
AEUnited Arab Emirates                                                157,743 
UZUzbekistan                                                   83,594 
VNVietnam                                                419,911 
YEYemen                                                     1,164 
ASAmerican Samoa (US)web.public.oc                                                  14,474                                            5,530,037
AUAustralia                                             2,982,455 
CCCocos (Keeling) Islands (Australia)                                                719,256 
CKCook Islands (NZ)                                                     1,308 
FMFederated States of Micronesia                                                   17,024 
FJFiji                                                     3,317 
PFFrench Polynesia (France)                                                     1,827 
GUGuam (US)                                                           62 
KIKiribati                                                         553 
MHMarshall Islands                                                           10 
NRNauru                                                         299 
NCNew Caledonia (France)                                                     5,891 
NZNew Zealand                                                686,488 
NFNorfolk Island (Australia)                                                     1,074 
MPNorthern Mariana Islands (US)                                                         653 
PWPalau                                                400,979 
PGPapua New Guinea                                                     2,147 
PNPitcairn Islands (UK)                                                         749 
WSSamoa                                                165,404 
SBSolomon Islands                                                     1,619 
TOTonga                                                   25,215 
TVTuvalu                                                495,755 
VUVanuatu                                                     1,471 
WFWallis and Futuna (France)                                                     2,007 
AQAntarcticaweb.public.aq                                                          94                                                          94

It is important to note that a minority of above mentioned TLDs, while formally added, are remaining in wait state because they do not meet our criteria of quality control. In this previous blog post Top-level domain considerations we explained why we are not accepting any TLD that exists (tl;dr: some TLDs such as “ZIP” don’t have enough domains to be worth in comparison to potential false positives).

Previously we were only crawling these TLDs:

  • Germany DE, AT, LU, CH
  • Russia: UA, KZ, RU
  • North Korea: KP
  • International: COM
  • Decentralized TLDs: BIT, BAZAR
  • US Government: GOV

Related articles

Newsletter 2021-02-20

Published on February 20, 2021 by

February 2021: Launch of the European Internet Archive The European Internet Archive just launched! 🎉🥳 ➡ https://archive.eu/ 225 TLDs added to the list of web crawling We have added 225 top-level domains (TLDs) to the list of web crawling. Find the full list and how we are categorizing them in this blog post. Our dataset


New Category “Bot Logs”

Published on February 10, 2021 by

We have added the new category “Bot Logs”. It contains data collected by and leaked from viruses such as Azorult. Such data is often sold on marketplaces such as the Genesis Market. We decided to index such data into this new category to help filtering out relevant results. You can find this new category in


Newsletter 2021-01-23

Published on January 23, 2021 by

January 2021: Capitol Hill Riots Archive, New Features Our dataset contains now 30 billion records and grows by +3 billion every month. Last year has been incredible for us. We are growing by any metric – all organically. Searches are up 113% Q-on-Q and our user base is growing sustainably at 18% month-on-month. Capitol Hill


Search the blog: