We have retired public API keys. All 3rd party tools and integrations must use per-user API keys. Note: This also affects SpiderFoot users. We have updated our SDK to reflect that change.
You can find your personal API key here: https://intelx.io/account?tab=developer
In related news, the tool Intel Owl v1.5.0, which “analyze files, domains, IPs in multiple ways from a single API at scale”, was just released and supports our Phonebook API for observables: https://github.com/intelowlproject/IntelOwl/releases/tag/v1.5.0.
We have added support for Bitmessage addresses. Bitmessage is a “decentralized, encrypted, P2P protocol to send encrypted messages”. Bitmessage addresses are now being indexed across all of our data sources and made searchable.
In March, we started indexing public documents from Sci-Hub. We are now finished! In total, we indexed 84,374,229 documents which equals to 74 TB of data. Here is a breakdown of the selectors:
Total selectors: 237,111,135. Unique:
🔹 52,996,669 URLs
🔹 14,070,158 email addresses
🔹 4,678,421 domains
🔹 476 Bitcoin addresses
🔹 24 IPFS hashes
Current statistics of Tor hidden services (.onion domains):
🔹 51,498 active
🔹 3,566,621 inactive
🔹 239,867 spam/deleted/blacklisted
Active means alive within the last 7 days.
An interesting observation is that there are more spam domains than active ones. One recent example of removed spam was 29,630 hidden services having “owncloud” in its subdomain name with 375,814 associated website copies from our index using 1,267 MB of storage.
As we cross the mark of 25 billion records with 100+ TB of storage, it is time to set sail for 100 billion records. Read more in this blog post.
There is a hidden feature of Intelligence X to find phishing copies of darknet websites, even if they are slightly modified – similarity hashes. You can find the similarity hash of websites when you click on a search result and click on the History tab.
Here is an example search: https://intelx.io/?s=8822b95eb6ab37be
Kleissner Investments s.r.o., Na Strzi 1702/65, 14000 Prague, Czech Republic
If you don’t wish to receive this newsletter anymore, please click here to unsubscribe.
June 2021: New Usenet data category We added the new data category Usenet. It contains historical and current data from Usenet, which is “a worldwide distributed discussion system”. Today, Usenet is mostly used for piracy. This new category stores currently 209,469,453 selectors and is expected to grow substantially. Improved inline statistics We have improved the
Intelligence X supports Peernet – Founder’s Statement I am excited to announce Peernet, a decentralized network that allows sharing of data freely without censorship and restrictions. Here is the pitch deck: https://peernet.org/dl/Peernet%20Deck.pdf Peernet is making quick progress from its inception as I am finalizing the whitepaper and developing the core library. I would like to
February 2021: Launch of the European Internet Archive The European Internet Archive just launched! 🎉🥳 ➡ https://archive.eu/ 225 TLDs added to the list of web crawling We have added 225 top-level domains (TLDs) to the list of web crawling. Find the full list and how we are categorizing them in this blog post. Our dataset