We are excited to announce that we just added 225 top-level domains (TLDs) to the list of web crawling! Below is the full list. The domain count per TLD represents the domains registered according to DomainTools. We group multiple TLDs into “buckets” to make it manageable – you can select these buckets in the Advanced menu next to the search button on intelx.io!
| TLD | Country | Bucket | Domain Count per TLD | Domain Count per Bucket |
|---|---|---|---|---|
| AT | Austria | web.public.de | 1,327,983 | 18,672,594 |
| DE | Germany | 15,106,819 | ||
| LU | Luxembourg | 89,822 | ||
| LI | Lichtenstein | 55,656 | ||
| CH | Switzerland | 2,092,314 | ||
| UA | Ukraine | web.public.ru | 566,476 | 5,599,820 |
| KZ | Kazakhstan | 148,691 | ||
| RU | Russia | 4,884,653 | ||
| KP | web.public.kp | 37 | 37 | |
| COM | web.public.com | 150,953,990 | 150,953,990 | |
| BIT | web.public.peer | |||
| BAZAR | ||||
| GOV | web.public.gov | 6,324 | 6,324 | |
| ORG | web.public.org | 10,317,554 | 10,317,554 | |
| NET | web.public.net | 13,291,542 | 13,291,542 | |
| INFO | web.public.info | 4,147,726 | 4,147,726 | |
| EU | web.public.eu | 3,597,950 | 3,597,950 | |
| CN | China | web.public.cn | 13,982,944 | 13,982,944 |
| UK | United Kingdom | web.public.nord | 10,300,887 | 14,713,717 |
| IE | Ireland | 288,539 | ||
| IS | Iceland | 65,529 | ||
| DK | Denmark | 1,306,074 | ||
| NO | Norway | 754,552 | ||
| FI | Finland | 483,625 | ||
| SE | Sweden | 1,508,703 | ||
| GL | Greenland | 5,808 | ||
| AD | Andorra | web.public.we | 2,053 | 15,862,803 |
| ES | Spain | 1,942,335 | ||
| FR | France | 3,593,494 | ||
| IT | Italy | 3,020,524 | ||
| MC | Monaco | 3,927 | ||
| MT | Malta | 15,515 | ||
| NL | Netherlands | 5,371,427 | ||
| BE | Belgium | 1,586,787 | ||
| PT | Portugal | 324,124 | ||
| SM | San Marino | 2,213 | ||
| VA | Vatican | 404 | ||
| AL | Albania | web.public.cee | 21,749 | 7,762,975 |
| BA | Bosnia & Herzigovina | 23,143 | ||
| BG | Bulgaria | 63,485 | ||
| BY | Belarus | 129,091 | ||
| CZ | Czech Republic | 1,271,461 | ||
| EE | Estonia | 128,346 | ||
| GR | Greece | 420,271 | ||
| HR | Croatia | 103,458 | ||
| HU | Hungary | 797,657 | ||
| LT | Lithuania | 198,736 | ||
| LV | Latvia | 113,289 | ||
| MD | Moldova | 25,032 | ||
| ME | Montenegro | 1,025,732 | ||
| MK | North Macedonia | 29,352 | ||
| PL | Poland | 2,240,128 | ||
| RO | Romania | 518,760 | ||
| RS | Serbia | 112,901 | ||
| SI | Slovenia | 124,981 | ||
| SK | Slovakia | 415,403 | ||
| AG | Antigua and Barbuda | web.public.ams | 19,104 | 13,829,992 |
| AI | Anguilla | 83,644 | ||
| AR | Argentina | 482,912 | ||
| AW | Aruba | 697 | ||
| BB | Barbados | 1,592 | ||
| BM | Bermuda | 3,106 | ||
| BO | Bolivia | 12,293 | ||
| BQ | Caribbean Netherlands | – | ||
| BR | Brazil | 3,837,643 | ||
| BS | Bahamas | 2,486 | ||
| BZ | Belize | 28,272 | ||
| CA | Canada | 2,871,681 | ||
| CL | Chile | 514,302 | ||
| CO | Colombia | 2,905,726 | ||
| CR | Costa Rica | 16,413 | ||
| CU | Cuba | 1,939 | ||
| CW | Curaçao | 875 | ||
| DM | Dominican Republic | 2,384 | ||
| DO | Dominican Repuplic | 26,031 | ||
| EC | Ecuador | 43,454 | ||
| FK | Falkland Islands | 145 | ||
| GD | Grenada | 2,456 | ||
| GF | French Guiana | 933 | ||
| GP | Guadeloupe | 2,173 | ||
| GT | Guatemala | 18,202 | ||
| GY | Guyana | 3,119 | ||
| HN | Honduras | 7,731 | ||
| HT | Haiti | 2,792 | ||
| JM | Jamaica | 7,386 | ||
| KN | Saint Kitts and Nevis | 829 | ||
| KY | Cayman Islands | 6,480 | ||
| LC | Saint Lucia | 3,729 | ||
| MQ | Martinique | 1,043 | ||
| MS | Montserrat | 8,970 | ||
| MX | Mexico | 929,819 | ||
| NI | Nicaragua | 4,695 | ||
| PA | Panama | 11,365 | ||
| PE | Peru | 112,857 | ||
| PR | Puerto Rico | 4,682 | ||
| PY | Paraguay | 22,897 | ||
| SR | Suriname | 3,165 | ||
| SV | El Salvador | 8,619 | ||
| SX | Sint Maarten | 2,761 | ||
| TC | Turks and Caicos Islands (UK) | 7,415 | ||
| TT | Trinidad and Tabago | 3,946 | ||
| US | United States | 1,661,201 | ||
| UY | Uruguay | 76,013 | ||
| VC | Saint Vincent and the Grenadines | Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â | Â 27,649 | |
| VE | Venezuela | 26,307 | ||
| VG | US Virgin Islands | 4,481 | ||
| VI | British Virgin Islands | 1,578 | ||
| AC | Ascension | web.public.af | 8,294 | 16,663,118 |
| DZ | Algeria | 9,880 | ||
| AO | Angola | 6,618 | ||
| BJ | Benin | 1,396 | ||
| BW | Botswana | 8,242 | ||
| BF | Burkina Faso | 1,643 | ||
| BI | Burundi | 2,738 | ||
| CM | Cameroon | 49,827 | ||
| CV | Cape Verde | 2,452 | ||
| CF | Central African Republic | 3,918,575 | ||
| TD | Chad | 582 | ||
| KM | Comoros | 141 | ||
| CD | Democratic Republic of the Congo | Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â | Â 6,090 | |
| DJ | Djibouti | 7,764 | ||
| EG | Egypt | 7,776 | ||
| GQ | Equatorial Guinea | 3,090,022 | ||
| ER | Eritrea | 94 | ||
| SZ | Eswatini | 1,790 | ||
| ET | Ethiopia | 2,583 | ||
| GA | Gabon | 4,227,941 | ||
| GM | Gambia | 1,451 | ||
| GH | Ghana | 2,966 | ||
| GN | Guinea | 416 | ||
| GW | Guinea-Bissau | 238 | ||
| CI | Ivory Coast | 9,447 | ||
| KE | Kenya | 83,325 | ||
| LS | Lesotho | 1,615 | ||
| LR | Liberia | 403 | ||
| LY | Libya | 17,330 | ||
| MG | Madagascar | 4,476 | ||
| MW | Malawi | 2,071 | ||
| ML | Mali | 3,511,703 | ||
| MR | Mauritania | 861 | ||
| MU | Mauritius | 9,938 | ||
| YT | Mayotte (France) | 3,689 | ||
| MA | Morocco | 83,579 | ||
| MZ | Mozambique | 7,129 | ||
| NA | Namibia | 4,564 | ||
| NE | Niger | 811 | ||
| NG | Nigeria | 119,413 | ||
| CG | Republic of the Congo | 1,650 | ||
| RW | Rwanda | 4,772 | ||
| RE | Réunion (France) | 28,150 | ||
| SH | Saint Helena | 14,350 | ||
| SN | Senegal | 5,935 | ||
| SC | Seychelles | 5,043 | ||
| SL | Sierra Leone | 1,992 | ||
| SO | Somalia | 11,183 | ||
| ZA | South Africa | 1,234,532 | ||
| SS | South Sudan | 26 | ||
| SD | Sudan | 3,520 | ||
| ST | São Tomé and PrÃncipe | 13,920 | ||
| TZ | Tanzania | 18,711 | ||
| TG | Togo | 1,657 | ||
| TN | Tunisia | 43,144 | ||
| UG | Uganda | 7,852 | ||
| EH | Western Sahara | – | ||
| ZM | Zambia | 4,152 | ||
| ZW | Zimbabwe | 42,656 | ||
| AF | Afghanistan | web.public.mea | 5,947 | 9,887,829 |
| AM | Armenia | 33,050 | ||
| AZ | Azerbaijan | 26,727 | ||
| BH | Bahrain | 5,040 | ||
| BD | Bangladesh | 25,528 | ||
| BT | Bhutan | 1,824 | ||
| BN | Brunei | 1,366 | ||
| KH | Cambodia | 3,208 | ||
| CY | Cyprus | 17,765 | ||
| GE | Georgia | 42,411 | ||
| HK | Hongkong | 232,505 | ||
| IN | India | 2,349,738 | ||
| ID | Indonesia | 375,112 | ||
| IR | Iran | 1,077,516 | ||
| IQ | Iraq | 2,029 | ||
| IL | Israel | 250,425 | ||
| JP | Japan | 1,506,391 | ||
| JO | Jordan | 4,521 | ||
| KW | Kuwait | 3,943 | ||
| KG | Kyrgyzstan | 11,123 | ||
| LA | Laos | 173,595 | ||
| LB | Lebanon | 4,539 | ||
| MO | Macao | 2,866 | ||
| MY | Malaysia | 247,237 | ||
| MV | Maldives | 5,108 | ||
| MN | Mongolia | 17,124 | ||
| MM | Myanmar | 4,202 | ||
| NP | Nepal | 78,200 | ||
| OM | Oman | 3,815 | ||
| PK | Pakistan | 93,709 | ||
| PH | Philippines | 116,685 | ||
| QA | Qatar | 19,218 | ||
| SA | Saudi Arabia | 64,201 | ||
| SG | Singapore | 175,686 | ||
| KR | Republic of Korea | 973,409 | ||
| LK | Sri Lanka | 68,505 | ||
| PS | State of Palestine | 7,639 | ||
| SY | Syria | 10,577 | ||
| TW | Taiwan | 686,428 | ||
| TJ | Tajikistan | 6,923 | ||
| TH | Thailand | 71,314 | ||
| TL | Timor-Leste | 1,717 | ||
| TR | Turkey | 398,104 | ||
| TM | Turkmenistan | 18,447 | ||
| AE | United Arab Emirates | 157,743 | ||
| UZ | Uzbekistan | 83,594 | ||
| VN | Vietnam | 419,911 | ||
| YE | Yemen | 1,164 | ||
| AS | American Samoa (US) | web.public.oc | 14,474 | 5,530,037 |
| AU | Australia | 2,982,455 | ||
| CC | Cocos (Keeling) Islands (Australia) | 719,256 | ||
| CK | Cook Islands (NZ) | 1,308 | ||
| FM | Federated States of Micronesia | 17,024 | ||
| FJ | Fiji | 3,317 | ||
| PF | French Polynesia (France) | 1,827 | ||
| GU | Guam (US) | 62 | ||
| KI | Kiribati | 553 | ||
| MH | Marshall Islands | 10 | ||
| NR | Nauru | 299 | ||
| NC | New Caledonia (France) | 5,891 | ||
| NZ | New Zealand | 686,488 | ||
| NF | Norfolk Island (Australia) | 1,074 | ||
| MP | Northern Mariana Islands (US) | 653 | ||
| PW | Palau | 400,979 | ||
| PG | Papua New Guinea | 2,147 | ||
| PN | Pitcairn Islands (UK) | 749 | ||
| WS | Samoa | 165,404 | ||
| SB | Solomon Islands | 1,619 | ||
| TO | Tonga | 25,215 | ||
| TV | Tuvalu | 495,755 | ||
| VU | Vanuatu | 1,471 | ||
| WF | Wallis and Futuna (France) | 2,007 | ||
| AQ | Antarctica | web.public.aq | 94 | 94 |
It is important to note that a minority of above mentioned TLDs, while formally added, are remaining in wait state because they do not meet our criteria of quality control. In this previous blog post Top-level domain considerations we explained why we are not accepting any TLD that exists (tl;dr: some TLDs such as “ZIP” don’t have enough domains to be worth in comparison to potential false positives).
Previously we were only crawling these TLDs:
At Intelligence X we categorize data sources into buckets. Buckets can be used as filters and to broadly identify the source of individual search results. For example, the bucket “Darknet Tor” indicates the result origins from some a Tor hidden service (.onion domain) and was collected by our Tor crawler. Buckets have human readable names
We just added support for an additional 152 top-level domains (TLDs), increasing the support to 511 TLDs in total. Support means that you can search for those domains across intelx.io and APIs, and internally that our backend supports processing them. While you can start searching for them immediately, it will take some time until our
Earlier today at 11:24 The Guardian Journalist Shaun Walker posted the security procedure and the security token used to pass makeshift checkpoints in Ukraine related to the Russian Ukrainian war: This is a reminder to journalists – and the public – to take OPSEC (operations security) seriously and not endanger people on the ground. Posting