Certain actors spam Tor by creating many duplicate websites under different .onion domains and then linking them to each other. The cost of doing that is pretty low, considering that all you need is creating a new public key pair (the onion domain is the hash of the public key). In theory anyone can create an infinite number of onion domains.
Sadly, bad actors are using this spam technique also for promoting websites with child exploitation content. The motivation behind creating many onion domains for essentially the same website (sometimes with rotating the content slightly for each copy) is likely to increase coverage by Tor search engines.
Since those spam websites provide 0 value and often host illegal content, we have decided to completely delete them from our search index. We are also reporting child exploitation content to organizations that work together with law enforcement. There are also technical considerations why we want to refrain from indexing spam content: Our crawlers should be busy with indexing actual onion websites and storage and system resources should not be wasted for content that has no value.
Detecting Spam Onion Domains
The spam websites are typically SEO optimized – after all that is why the spam technique is used in the first place. This means that they have descriptive meta tags in the HTML data, as well as domain names that may indicate the type of content.
Therefore, our algorithms take the following into consideration to fingerprint websites to classify as spam:
Our algorithms have removed:
August 2020: Latest News & Statistics Public API keys are retired We have retired public API keys. All 3rd party tools and integrations must use per-user API keys. Note: This also affects SpiderFoot users. We have updated our SDK to reflect that change. You can find your personal API key here: https://intelx.io/account?tab=developer In related news,
At Intelligence X, we value quality over quantity. Our goal is continuous improvement, sustainability, and stability. As we cross the mark of 25 billion records with 100+ TB of storage, it is time to set sail for 100 billion records. Counting records A single record is an extracted selector (search term) like “test.com”. A search
August 2020: Recap of 3rd-party OSINT tools and integrations We are listing all approved 3rd-party integrations here: https://intelx.io/integrations h8mail: “an email OSINT and breach hunting tool using different breach and reconnaissance services” Maltego Transform subfinder: “subdomain discovery tool that discovers valid subdomains for websites by using passive online sources” theHarvester: “The tool gathers emails, names,