Open source intelligence tools and resources handbook 2020 pdf

The following is a list of links shared by those attending the 2021 Open-Source Intelligence Summit. SANS loves the positivity and contributions from the Summit attendees, and that members of the OSINT community openly shared these URL links for use by others in the OSINT community.

Notice of Non-Affiliation and Disclaimer: SANS is not affiliated with, and does not control or endorse these links in any way. The tools, websites or information provided here are to be used at your own risk. We suggest that those interested in using these resources do their due diligence and assess them carefully before use.

We truly appreciate the full participation of all the presenters and attendees of the 2021 SANS Open-Source Intelligence Summit – the OSINT community is what made it great!

Micah Hoffman & John TerBush

General 2021 SANS OSINT Summit Links:

OSINT Community Groups:

https://sec487.info/summitdiscord

https://discord.gg/WSt7yUj9

Home

https://www.facebook.com/groups/Boolean.Strings

https://osint.team/home

https://searchlight.community/

https://www.tracelabs.org/get-involved

Free OSINT Resource and Tool Compilations:

http://www.sans.org/free

https://start.me/p/DPYPMz/the-ultimate-osint-collection

https://www.osintdojo.com/resources/

https://github.com/sinwindie/OSINT

https://www.osintcombine.com/tools

http://osintframework.com

http://osintframework.de

https://start.me/p/OmExgb/terrorism-radicalisation-research-dashboard

http://bit.ly/bcattools

150+ Top Sourcing / #OSINT Tools

Free OSINT tools:

http://privacytools.io

https://github.com/evilsocket/ditto

https://www.whatsmyname.app/

https://blog.bushidotoken.net/2021/02/using-discord-server-as-personal-cti.html

https://outline.com/

https://dfir.blog/unfurl/

https://gchq.github.io/CyberChef/

https://github.com/ping/instagram_private_api

http://DarkSearch.io

https://canarytokens.org/generate

https://github.com/azmatt/gamera

https://exiftool.org/

https://guardianproject.info/apps/org.witness.proofmode/

https://www.thexifer.net/

http://metricsparrow.com/toolkit/email-permutator/

https://namecombiner.com/

https://github.com/sham00n/buster

https://github.com/j3ssie/Osmedeus

https://scrapy.org/

https://www.maltego.com/ce-registration/

http://howmanyofme.com/search/

https://haveibeenpwned.com/NotifyMe

https://www.fakepersongenerator.com/

Free Translation Tools:

https://www.excite.co.jp/world/english_japanese/

https://papago.naver.com/

Free Note-Taking Tools:

https://obsidian.md/

http://draw.io

Online Databases of use for OSINT:

http://Shodan.io

https://wigle.net/

https://archive.org/web/

https://czds.icann.org/home

https://honeydb.io/

http://domainbigdata.com

https://www.whoxy.com/

OSINT Browser Plug-ins:

https://chrome.google.com/webstore/detail/instant-data-scraper/ofaokhiedipichpaobibbnahnkdoiiah

https://chrome.google.com/webstore/detail/search-by-image/cnojnbdhbhnkbcieeekonklommdnndci

Paid OSINT Tools:

https://www.hunch.ly/

Vortimo HomeP

https://www.notion.so/

https://www.maltego.com/

OSINT Books:

https://inteltechniques.com/book1.html

https://www.ialeia.org/docs/Psychology_of_Intelligence_Analysis.pdf

https://www.scribd.com/book/287523777/Google-Hacking-for-Penetration-Testers

https://www.wiley.com/en-us/Hunting+Cyber+Criminals%3A+A+Hacker%27s+Guide+to+Online+Intelligence+Gathering+Tools+and+Techniques-p-9781119540991

https://arxiv.org/pdf/1703.03107.pdf

OSINT Videos:

https://www.youtube.com/c/SANSBlueTeamOps

OSINT Blogs and Articles:

https://sector035.nl/articles/category:week-in-osint

https://www.linkedin.com/pulse/useful-websites-your-investigation-chinese-individuals-shu-han

https://www.secjuice.com/introduction-to-open-source-intelligence-osint/

https://medium.com/secjuice/osint-as-a-mindset-7d42ad72113d

How to trace social media users across multiple platforms

Vicarious trauma and OSINT – a practical guide

Using RESET for better OSINT

https://medium.com/@benjamindbrown/finding-mcafee-a-case-study-on-geoprofiling-and-imagery-analysis-6f16bbd5c219

Fugitive John McAfee’s location revealed by photo meta-data screw-up

https://pentestmag.com/product/pentest-osint-on-pentest-targets/

OSINT Map: A MindMap for Your Investigations

https://www.cybrary.it/blog/0p3n/google-dorks-easy-way-of-hacking/

https://www.sans.org/blog/cyber-camp-blog-in-real-time/

How to land an OSINT job

https://sector035.nl/articles/osint-movie-time-for-the-holidays

https://www.troyhunt.com/heres-how-i-verify-data-breaches/

https://securitytrails.com/blog/cybersecurity-red-blue-team

https://securitytrails.com/blog/what-is-osint-how-can-i-make-use-of-it

https://inteltechniques.com/podcast.html

Advanced Search Engine Usage:

https://www.exploit-db.com/google-hacking-database

https://kit.exposingtheinvisible.org/en/how/google-dorking.html

https://github.com/BullsEye0/google_dork_list

https://programmablesearchengine.google.com/about/

https://support.google.com/programmable-search/answer/4513882?hl=en

https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers

https://search.google.com/test/mobile-friendly

https://www.etools.ch/

The Full List of 21 Google Search Operators

https://seosly.com/yandex-search-operators/

Image Search and Analysis:

https://pimeyes.com/en

https://www.si.edu/openaccess

https://www.adl.org/education-and-resources/resource-knowledge-base

https://www.splcenter.org/fighting-hate/intelligence-report/2006/look-racist-skinhead-symbols-and-tattoos

https://www.emilyboda.com/post/instagram-runs-image-recognition-software-on-every-post

Social Network OSINT:

http://map.snapchat.com/

http://pipl.com

https://namecheckup.com/

http://snapchat.com/add/username

https://botometer.osome.iu.edu/

Mobile OSINT:

https://github.com/Genymobile/scrcpy

https://letsview.com/

https://www.airsquirrels.com/reflector

https://sundowndev.github.io/PhoneInfoga/formatting/

https://support.truecaller.com/hc/en-us/articles/360001167949-Who-viewed-my-profile-

https://sync.me/

https://www.android-x86.org/

https://www.genymotion.com/fun-zone/

https://www.osboxes.org/android-x86/

Tor & VPNs:

https://4n6lady.medium.com/tor-over-vpn-or-vice-versa-4a23eb40757d

https://protonvpn.com/blog/tor-vpn/

https://www.safetydetectives.com/best-vpns/#simple-vpn-comparison

https://www.techradar.com/news/tor-and-vpn-how-well-do-they-mix

https://www.vpnmentor.com/blog/top-5-logless-vpns-privacy-seeker/

Operating Systems for Anonymity:

https://tails.boum.org/index.en.html

https://tails.boum.org/install/

https://tails.boum.org/install/index.en.html

https://www.whonix.org/

Dark Web:

https://www.cybrary.it/blog/2017/04/exploring-dark-web-dont-venture-alone/

https://www.cybrary.it/blog/dark-web-search-introduction-and-instruction/

https://www.cybrary.it/blog/how-to-safely-access-the-dark-web/

https://www.ma-no.org/en/security/onion-search-engine-how-to-browse-the-deep-web-without-tor

https://www.sans.org/webcasts/dark-web-solutions-forum-illuminating-dark-web-harvesting-osint-data-dark-web-resources-110200

https://www.justice.gov/criminal-ccips/page/file/1252341/download

Browser Bookmarklets:

https://one-plus.github.io/Bookmarks

http://com.hemiola.com/bookmarklet/

https://www.secjuice.com/osint-bookmarklet-tools/

Browser Developer Tools:

https://developer.mozilla.org/en-US/

https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

Coding and APIs:

https://register.automatingosint.com/python-course/

https://www.w3schools.com/

https://www.coursera.org/learn/python

https://www.codecademy.com/

https://anvil.works/blog/python-in-the-browser-talk

https://code.visualstudio.com/download

https://datatables.net/

https://fontawesome.com/

https://github.com/getify/You-Dont-Know-JS

https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js

https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css

https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js

https://www.30secondsofcode.org/

https://www.javatpoint.com/javascript-tutorial

https://www.oreilly.com/library/view/javascript-the-definitive/9781491952016/

http://tabulator.info/

https://docs.microsoft.com/en-us/learn/paths/introduction-python-space-exploration-nasa/

https://snakify.org/en/

http://Udemy.com

https://learning.postman.com/docs/getting-started/introduction/

https://developer.twitter.com/en/docs/twitter-api

Think Python 2e

https://www.learnpython.org/

http://www.automatingosint.com/blog/2017/04/building-a-keyword-monitoring-pipeline-with-python-pastebin-and-searx/

http://realpython.com

https://rstudio.com/

Computer Science Basics:

https://cs50.harvard.edu/x/2021/

https://tools.ietf.org/html/

https://twit.tv/shows/security-now

Home

https://automatetheboringstuff.com/

2021 SANS OSINT Summit Speaker Links:

Apurv Singh Gautam

https://apurvsinghgautam.me/#talks

https://github.com/apurvsinghgautam/dark-web-osint-tools

Irina Shamaeva

https://booleanstrings.com

https://booleanbook.com

Sourcing Services

Chris Poulter

http://osintcombine.com

Curtis Hanson

https://www.riskint.blog/

http://www.gsxt.gov.cn/index.html

http://www.iranchamber.com/calendar/converter/iranian_calendar_converter.php

Heather Honey

Home

http://smmart.us

Jeff Lomas

https://www.sans.org/profiles/jeffrey-lomas/

Azat Kashparov & Andrew Kulikov

Matt Edmondson

https://github.com/azmatt

Matthias Wilson

https://gosintcon.de/

https://keyfindings.blog/

Nico Dekens

https://dutchosintguy.com/

Steven Harris

https://nixintel.info/

Ygor Maximo

https://github.com/mxm0z

OSINT Curious Panel

https://osintcurio.us/

https://www.technisette.com/

https://twitter.com/ChristinaLekati/

Michael James’ Headphones of Impressive Appearance:

OneOdio® Pro-10 Over Ear Wired Headphones, Best Seller

OSINT Summit Sponsors

http://domaintools.com/

https://www.skopenow.com/

The 2020 edition of our Open Source Intelligence Tools and Resources Handbook has been completely revised and updated to reflect changes in the technical and operational domains analysts and OSINT practitioners have to work in.

The Handbook is not just for experienced investigators and information security professionals. Rather, it is for anyone wishing to improve the quality of their research, regardless of where they are in their career. Newcomers to OSINT can use the different tool categories to orient their learning, while those who manage them can use the same categories to explore the state-of-the-art.

All tools have been vetted by the i-intelligence team prior to their inclusion. Nevertheless, the usual security precautions apply: always experiment on a dedicated desktop or laptop PC, and be sure to firewall any devices you use for research and investigative purposes.

We hope the Handbook is of value to you and your colleagues. We welcome any feedback you may have, as well as any suggestions on new tools or those we may have overlooked.

Best of luck with your work!

Download the OSINT Handbook 2020

Open source intelligence (OSINT) from the surface, deep or dark web is invaluable to threat intelligence investigations. Find the shortcuts to improve your research.

What is OSINT?

“Open source intelligence” doesn’t just refer to the accessibility of information. OSINT is the practice of collecting information from publicly available sources.

OSINT grew out of spycraft as it shifted away from clandestine methods of information gathering (think phone tapping, tails) and toward scouring publicly available information like newspapers and files or databases open to the public.

With the advent of the internet, vastly more information became publicly available and OSINT became increasingly useful not just to sophisticated government agencies and law enforcement, but to financial crime analysts, fraud and brand misuse investigations and particularly cybersecurity.

Cybersecurity teams frequently use OSINT for OPSEC (operational security) by understanding what of their company’s information is publicly available. This information may be on assets they control that are designed to be public-facing or become so through error, or on assets outside the company perimeter, like social media or third-party websites that may accidentally leak information.

OSINT on the deep and dark web

The examples given are where companies may perform OSINT on the surface web (i.e., the internet most of us use every day). But OSINT can also be conducted on the deep or dark web.

The deep web is a layer below the surface web that requires login or subscription services. These sites can include academic journals, court record databases or even services like Netflix. OSINT can still be applied even to sites requiring login or subscription — as long as analysts can access the information legally, without hacking.

And, that extends to the dark web.

While the surface and deep web can be accessed by any common browser, the dark web requires specific software, like Tor (The Onion Router). Once inside, there’s lots of information that can be beneficial to threat intelligence gathering and other investigations.

If you’re using the dark web for OSINT, it’s important to remember:

  • Paying for hacked/stolen items can qualify as OSINT, but there are lots of practical, ethical and legal considerations one should make before engaging in such a purchase (the DOJ CCIPS has good guidance here)
  • Any website could introduce malicious code to your computer, but this is especially true on the dark web, where site owners often set boobytraps to track potential adversaries
  • There is some anonymity to using the dark web, but there are still lots of details given to site owners about your identity — you’ll need to control your digital fingerprint

Learn more: 3 things to consider before you start your dark web investigation >

How is OSINT used in threat intelligence gathering?

As discussed above, OSINT is a valuable technique for OPSEC, but it can also be used to gather threat intelligence to proactively reduce cyber risks.

OSINT is used to analyze, monitor and track cyberthreats from targeted or indiscriminate attacks against an organization by malware and bad actors. There are typically one of two sources that trigger a cyber OSINT investigation:

  1. A flag or item of interest identified from a threat intelligence platform (TIP) or subscription service
  2. A new threat, vulnerability or data breach is identified from an OSINT news source
  3. A threat hunter identifying a potential advanced persistent threat (APT) within the network

In the case of an issue caught by a TIP, while the initial indicator is valuable, the level of detail and specificity to the organization often will require enrichment to understand how significant it is. Conducting OSINT across the surface, deep and dark web can enrich the indicator to understand urgency and scope. For example, a TIP may flag that email addresses and passwords are in a breach package or on a forum or dark web site. An analyst will want to go and see the full breach package to understand potential high-ranking targets for phishing attacks.

Additionally, the analyst can provide more detailed information regarding the breached information to include who may be impacted at their organization along with how the breach occurred for more amplifying information. 

In the case where a threat hunter identifies an anomaly on the internal network, they need to understand if it’s malicious. This often requires a lot of research into current attacker tactics, techniques, procedures (TTPs). This may require researching and collecting info in areas where attackers reside like forums.

When it comes to the identification of a new threat or vulnerability that was reported by a news organization or cybersecurity news/research organization, there is the need for the analyst to confirm the reports. This is done by not only looking on the surface and deep web for additional reporting and details, but it may also include looking on the dark web for information on where this new threat or vulnerability will be conducted or has been conducted. This is where having the knowledge and ability to access the deep and dark web becomes important for a cyberthreat or cybersecurity analyst.

OSINT Techniques

When searching for information on the surface web, the websites themselves hold several keys about who might be behind the content. (On the dark web, you won’t be so lucky as site operators and owners are anonymous.) These services provide user-friendly protocols for retrieving that information from the databases that house domain data.

Learn more: Essential tools for improve surface and dark web research >

    Identifying site owners through WHOIS

    WHOIS records provide top-level domain (the .com or .org root of the URL) information. This includes addresses, names and phone numbers used to register the domain, the date of registration and details about where it is hosted.

    By combining WHOIS query and response protocols with additional search tools, investigators can uncover more information.

    URLscan.io

    URLscan.io is a service that provides the end user with analysis of the IP address information and HTTP connections made during the site’s retrieval. The result panels include a top-level survey of what country the site is hosted in, what links are included on the main page and the IP location details. Details about how many subdomains it contains and what external links it contains can be found as well. 

    Through WHOIS analysis, hosting details can also be discovered. This can help lead investigators to find servers that host multiple sites or share webmasters, as well as valuable owner information.

      Urlscanio sample

      
      

      DomainIQ

      DomainIQ operates similarly to URLscan.io and can provide identifying details about the site owner, host and what other pages they may be operating.

      Utilizing advanced search engine techniques

      By using advanced search engine techniques, we can search the identifying data from WHOIS records (such as emails, names, servers or IP addresses) and find additional clues or information that may be lurking on other sites.

      Carbon Date

      Carbon Date uses the advanced search engine technique of “carbon dating” that analyzes a website and gives the earliest known creation date of the page. You can also view previous versions of the page, including the first known scrape through archive.org.

      Google Dorking

      “Google Dorking” is the process of using advanced search parameters on Google. There are several techniques that can be used ranging from simple to more advanced. Some of the most common Bolean logic search operators are using quotes to search for exact phrasing or the dash symbol (-) to exclude specific words. You can also use Google to search specific file types or recent caches of a specific site.

      These techniques can be used to find identifying information about moderators or search a site for identifying pieces. It can also be used to string together sites sharing specific information.

      Common Google Dorking techniques include:

      • Intitle: identifies any mention of search text in the web page title
      • Allintitle: only identifies pages with all of the search text in the web page title
      • Inurl: identifies any mention of search text in the web page URL
      • Intext: only identifies pages with all of the search text in the web page URL
      • Site: limits results to the specified file type
      • Filetype: limits results to only the specified file type
      • Cache: shows the most recent cache of a site specified
      • Around (X): searches for two different words within X words of one another

      All of these tools can help investigate ownership and hosting information about the sites relevant to your research. Using WHOIS records and advanced search engine techniques can reveal identifying details on the host, moderator and IP, as well as what other sites might be sourced from the same owners.

      Learn more about WHOIS records analysis, advanced search engine use and real-world examples of these techniques in action in our flash report, Investigating Site Ownership and History >

      Top OSINT research tools

      There are tons of tools available to aid OSINT for threat intelligence gathering, many of which are free to use. Below are some of our top go-to’s for conducting OSINT on the surface and dark web.

      Learn more: 21 OSINT research tools for threat intelligence investigations >

      OSINT Framework: find free OSINT resources

      OSINT Framework: Find free OSINT resources

      
      

      WHAT IT IS

      OSINT Framework indexes a multitude of connections to different URLs, recommending where to look next when conducting an investigation. It also provides suggestions on what services can help analysts find specific data that might aid in their research.

      USE CASE

      When you plug a piece of data (such as an email address, phone number, name, etc.) into the framework, it returns all known online sources that contain information relevant to that data. OSINT Framework also offers a list of potential resources where more information related to that particular source can be found.

      Maltego Transform Hub: mine, merge and map information

      Maltego Transform Hub: Mine, Merge and Map Information

      
      

      WHAT IT IS

      Integrate data from public sources, commercial vendors and internal sources via the Maltego Transform Hub.All data comes pre-packaged as Transforms, ready to be used in investigations. Maltego takes one artifact and finds more.

      USE CASE

      A user feeds Maltego domain names, IP addresses, domain records, URLs or emails. The service finds connections and relationships within the data and allows users to create graphs in an intuitive point- and-click logic.

      Shodan: the search engine for the IoT

      Shodan: The Search Engine for the IoT

      
      

      WHAT IT IS

      Websites are just one part of the internet. Shodanallows analysts to discover which of their devices are connected to the internet, where they are located and who is using them.

      USE CASE

      Shodan helps researchers monitor all devices within their network that are directly accessible from the internet and therefore vulnerable to attacks.

      ThreatMiner: IOC lookup and contextualization

      ThreatMiner: IOC Lookup and Contextualization

      
      

      WHAT IT IS

      ThreatMiner is a threat intelligence portal designed to enable an analyst to research indicators of compromise (IOCs) under a single interface. That interface allows for not only looking up IOCs but also providing the analyst with contextual information. With this context, the IOC is not just a data point but a useful piece of information and potentially intelligence.

      USE CASE

      Identify and enrich indicators of compromise to have a better understanding of attack origins.

      Torch search engine: explore the darknet

      Torch Search Engine: Explore the DarkNet

      
      

      WHAT IT IS

      Torch, or TorSearch, is a search engine designed to explore the hidden parts of the internet. Torch claims to have over a billion darknet pages indexed and allows users to browse the dark web uncensored and untracked.

      USE CASE

      Torch promises peace of mind to researchers who venture into the dark web to explore .onion sites. It also doesn’t censor results — so investigators can find all types of information and join discussion forums to find out more about current malware, stolen data for sale or groups who might be planning a cyberattack.

      Dark.fail: go deeper into the darknet

      Dark.fail: Go Deeper into the Darknet

      
      

      WHAT IT IS

      Dark.fail has been crowned the new hidden wiki. It indexes every major darknet site and keeps track of all domains linked to a particular hidden service.

      USE CASE

      Tor admins rely on Dark.fail to disseminate links in the wake of takedowns of sites like DeepDotWeb. Researchers can use Dark.fail when exploring sites that correlate with the hidden service.

      To learn more about Silo for Research, check out our experience Silo page!

      Threat intelligence

      Written by Jane