A few days ago a new search engine DarkSearch for Tor launched, adding to the mix of other existing search engines out there like Ahmia, Torch, Not Evil, and Haystack – it’s time for a feature comparison!
No search engine can cover 100% of the pages due to the nature of Tor. There is no central .onion repository so the first challenge is to find the .onion links. Other challenges when running a search engine include data size (and associated storage and processing power), data formats, and many smaller challenges like depth of crawling (i.e. how many sub-pages, how to behave when there are infinite sup-pages).
The following graph shows our index of Tor (dark blue) and I2P (light blue). As of April 2019, we have 10,197,379 items indexed for Tor and 1,557,915 items for I2P. An item can be any supported file format – including HTML, text, PDF, office documents (Word, Excel, and PowerPoint files), and since yesterday, even eBooks.
We have 2,250,020 .onion addresses in our index, although only a small fraction is actually active. For I2P our index has 3,565 .i2p domains listed.
March 2020 Updates How we prepared for the current global situation We have ordered and deployed 180 TB worth of enterprise storage to be prepared for upcoming price increases and shortages of hard disks. We are reading reports that warn of upcoming delivery delays due to disruptions in the supply chain. We tweeted a picture
“The Death Master File (DMF) is a computer database file made available by the United States Social Security Administration since 1980″ according to Wikipedia. It is available here https://ladmf.ntis.gov/ but costs $2,930.00 anually. The file has since been posted on the internet for free, including here: http://ssdmf.info/download.html November 30, 2011 http://cancelthesefunerals.com/ May 31, 2013 https://archive.org/details/DeathMasterFile May 31, 2013 This file can
On December 11, 2019 we have received 31,866 HTTP requests from the IP 81.171.107.57. Below are few sample log entries: 81.171.107.57 – – [11/Dec/2019:13:52:37 +0000] “POST /login HTTP/1.1” 200 7448 81.171.107.57 – – [11/Dec/2019:13:52:37 +0000] “POST /login HTTP/1.1” 200 7443 81.171.107.57 – – [11/Dec/2019:13:52:37 +0000] “POST /login HTTP/1.1” 200 7447 81.171.107.57 – – [11/Dec/2019:13:52:37 +0000]