Top-level domain considerations

Published on September 21, 2020 by

We want to give some insight into top-level domains (TLDs) supported by Intelligence X. Even though there are a total of 1,508 available TLDs (full list by IANA), we choose to only support certain TLDs in order to prevent spam, false positives, and maintain a high quality data set.

Validation of top-level domains appears in these places:

  1. Search term input: Detect when the user enters a valid domain (as well as email address, URL)
  2. Indexing of search results: Detect when a valid domain name appears in text

False Positives

“ZIP” is an example TLD that we intentionally do not support. It has only 2 domains (according to DomainTools), but supporting it would mean millions (if not billions) of false positives, due to its common use as file extension for ZIP files. “test.zip” as filename is on its own indistinguishable from “test.zip” as domain name.

If you try to search for a domain with an unsupported TLD, you will get this message:

Count of supported TLDs

2 years ago we last updated the list of supported TLDs and included 362 domains. Today, we added 29 that meet our criteria for quality. Interestingly, there are some TLDs that met the criteria 2 years ago, but now do not.

Crawling

Intelligence X operates public web crawlers. At the time of writing, 8 TLDs are currently being crawled and more are in preparation.

Related articles

Newsletter 2021-12-24

Published on December 24, 2021 by

December 2021: Christmas Special 🎄🎁 Merry Christmas! To celebrate, we are offering a special 1-month license for € 50. This offer is valid for a week. It provides the same type of access as the Pro license that costs € 2000 /year. To take advantage of this Christmas special, go to https://intelx.io/order. License Changes in


Intelligence X welcomes seasoned Security Expert to Advisory Board

Published on December 2, 2021 by

Co-Founder of QuoScient GmbH and QuoLab Technologies Inc., former Global Head of Malware Research and Response at Deutsche Bank joins Intelligence X’s newly created advisory board as the company is gaining momentum. “We are very happy to welcome Fabien today to our advisory board at a time when we keep expanding on our services beyond


Newsletter 2021-06-29

Published on June 29, 2021 by

June 2021: New Usenet data category We added the new data category Usenet. It contains historical and current data from Usenet, which is “a worldwide distributed discussion system”. Today, Usenet is mostly used for piracy. This new category stores currently 209,469,453 selectors and is expected to grow substantially. Improved inline statistics We have improved the


Search the blog: