Combating spam websites from Tor

Published on April 27, 2020 by

Certain actors spam Tor by creating many duplicate websites under different .onion domains and then linking them to each other. The cost of doing that is pretty low, considering that all you need is creating a new public key pair (the onion domain is the hash of the public key). In theory anyone can create an infinite number of onion domains.

Sadly, bad actors are using this spam technique also for promoting websites with child exploitation content. The motivation behind creating many onion domains for essentially the same website (sometimes with rotating the content slightly for each copy) is likely to increase coverage by Tor search engines.

Since those spam websites provide 0 value and often host illegal content, we have decided to completely delete them from our search index. We are also reporting child exploitation content to organizations that work together with law enforcement. There are also technical considerations why we want to refrain from indexing spam content: Our crawlers should be busy with indexing actual onion websites and storage and system resources should not be wasted for content that has no value.

Detecting Spam Onion Domains

The spam websites are typically SEO optimized – after all that is why the spam technique is used in the first place. This means that they have descriptive meta tags in the HTML data, as well as domain names that may indicate the type of content.

Therefore, our algorithms take the following into consideration to fingerprint websites to classify as spam:

  • HTML tag <title>
  • HTML tag <meta name=”description”>
  • HTML tag <meta name=”keywords”>
  • Subdomain name
  • Text of outgoing links (<a> tags)
  • “alt” attribute of <img> tags (= alt text of pictures)

Statistics

Our algorithms have removed:

  • 209,081 unique onion domains (this number includes sub-domains)
  • About 500 GB of archived text pages
  • About 34 million archived text pages and related index files
  • 28% of our overall Tor index

Related articles

Newsletter 2021-12-24

Published on December 24, 2021 by

December 2021: Christmas Special 🎄🎁 Merry Christmas! To celebrate, we are offering a special 1-month license for € 50. This offer is valid for a week. It provides the same type of access as the Pro license that costs € 2000 /year. To take advantage of this Christmas special, go to https://intelx.io/order. License Changes in


Intelligence X welcomes seasoned Security Expert to Advisory Board

Published on December 2, 2021 by

Co-Founder of QuoScient GmbH and QuoLab Technologies Inc., former Global Head of Malware Research and Response at Deutsche Bank joins Intelligence X’s newly created advisory board as the company is gaining momentum. “We are very happy to welcome Fabien today to our advisory board at a time when we keep expanding on our services beyond


Newsletter 2021-06-29

Published on June 29, 2021 by

June 2021: New Usenet data category We added the new data category Usenet. It contains historical and current data from Usenet, which is “a worldwide distributed discussion system”. Today, Usenet is mostly used for piracy. This new category stores currently 209,469,453 selectors and is expected to grow substantially. Improved inline statistics We have improved the


Search the blog: