Top 10 Popular Open Source

Intelligence (OSINT) Tools

Introduction
In this article, we will be discussing various OSINT tools that are available in the
market. When we search the internet there are multiple pages of results that are
presented. We just have a look at the first page and research and if we do not
get what we are expecting, we stop right? But have you ever wondered what lies
in those hundreds of pages of result? "Information”! Let's get this free information
using various tools. Tools are important but not knowing the usage of a tool will
leave the user helpless. Before digging into the tools let's have a fair idea of what
OSINT is and what can be achieved out of it.

What is Open Source Intelligence?

OSINT stands for open source intelligence. The Internet is an ocean of data which
is an advantage as well as a disadvantage.

Pros are that the internet is free and accessible to everyone unless restricted by
an organization or law. The Internet has all the information readily available for
anyone to access. Cons are that the information is available that can be misused
by someone with a malicious intent. Collection and correlation of information
using these tools are referred to as open source intelligence. Information can be
in various forms like audio, video, image, text, file etc. Below is the bird's eye view
of the data categories available on the internet:

1. Social media websites like Twitter, Facebook etc. hold a lot of user data.
2. Public facing web servers: Websites that hold information about various
users and organizations.
3. Newsletters and articles.
4. Code repositories: Software and code repositories like Codechef, Github
hold a lot of information but we only see what we are searching for.

Why do we need tools?

Getting to know that the information is available is one thing. Collection of the
information is second and making an analysis or intelligence out of them is the
third. The information can be gathered manually as well but that will take the time
that can instead be used in the later stages. Tools can help us gather the data
from hundreds of sites in minutes and thus easing the collection phase. Let us
say that the task is to identify whether a username is present and if so, on which
all social media websites. One way is to log in to all the social media websites (I
bet you don't know all of them!) and then testing the username in that. Another
way is to use an open source tool that is connected to various websites more
than what we can remember and checks the usernames presence on all the
websites at once. This is done just in seconds. Run multiple tools to gather all
target related information that can be correlated and used later.

You may also like: Fundamentals of Website Security for Online Retailers

OSINT Tools
1. Maltego

Maltego is developed by Paterva and is used by security professionals and

forensic investigators for collecting and analyzing open source intelligence. It can
easily collect Information from various sources and use various transforms to
generate graphical results. The transforms are inbuilt and can also be customized
based on the requirement. Maltego is written in Java and comes pre-packaged in
Kali Linux. To use Maltego, user registration is required, the registration is free.
Once registered users can use this tool to create the digital footprint of the target
on the internet.

2. Shodan
Google is the search engine for all but shodan is the search engine for
hackers. Instead of presenting the result like other search engines it will show
the result that will make more sense to a security professional. As a certified
information security professional one of the important entity is digital asset and
network. Shodan provides you a lot of information about the assets that have
been connected to the network. The devices can vary from computers, laptops,
webcams, traffic signals, and various IOT devices. This can help security analysts
to identify the target and test it for various vulnerabilities, default settings or
passwords, available ports, banners, and services etc.

You may also like: Brute Force Attacks: Prominent Tools to Tackle Such
Attacks

3. Google Dorks

Google is one of the most commonly used search engine when it comes to finding
stuff on the internet. For a single search, the results can be of various hundred
pages sorted in order of relevance. The results vary from ads, websites, social
media posts, images etc. Google Dorks can help a user to target the search or
index the results in a better and more efficient way. Let us say that the user wants
to search for the word usernames but only requires the results with PDF files and
not websites. This is done as below:
<Filetype: searches for a particular string in a pdf file>

Some of the other indexing options are:

 Inurl: search for a string in URL of the page.

 Intitle: To search the title for a keyword.
 Ext: To search for a particular extension.
 Intext: Search for a particular text in a page.

Sometimes it is also referred to as Google hacking.

4. The Harvester
A harvester is an excellent tool for getting email and domain related information.
This one is pre-bundled in Kali and can be very useful in fetching information.
Below is an example of the output when we try to search for emails for Microsoft
in PGP server. You can explore more as per requirement.

E.g the harvester –d Microsoft.com –b pgp

5. Metagoofil
Metagoofil is written by Christian Martorella and is a command line tool that is
used to gather metadata of public documents. The tool is pre-bundled in Kali
Linux and has a lot of features searching for the document type on the target,
local download, extraction of metadata and reporting the results. For example:
Users can scan for a particular kind of documents on a particular domain.
Metagoofil –d nmap.org –t pdf.

6. Recon-ng
Recon-ng is a great tool for target information collection. This is also pre-bundled
in Kali. The power of this tool lies in the modular approach. For those who have
used Metasploit will know the power of modular tools. Different modules can be
used on the target to extract information as per need. Just add the domains in
the workspace and use the modules. For starters, here is a sample of the tool
helping you.

You may also like: Top 15 Prominent Wireless Hacking Tools to watch out
for in 2018

7. Check Usernames
Social networking websites hold a lot of information but it will be really boring and
time taking task if you need to check whether a particular username is present on
any social media website. To get such information there is a
website www.checkusernames.com. It will search for the presence of a particular
username on more than 150 websites. The users can check for the presence of
a target on a particular website so as to make the attack more targeted.

A more advanced version of the website is https://knowem.com which has a more

wide database of more than 500 websites along with a few more services?

8. TinEye
Tineye is used to perform an image related search on the web. It has various
products like tineye alert system, color search API, mobile engine etc. You can
search if an image has been available online and where that image has appeared.
Tineye uses neural networks, machine learning, and pattern recognition to get
the results. It uses image matching, watermark identification, signature matching
and various other parameters to match the image rather than keyword matching.
The website offers API extensions and browser extensions as well. You can
simply visit the image and right click on it to select search on tineye.
Link: https://www.tineye.com

9. Searchcode

Searching for text is easy as compared to searching for a code snippet. Try
searching for a code sample on google and you will be prompted with no results
or irrelevant results. Search code offers you a feature to search for a line of code
which could have been present in various code sharing websites like Github etc.
Users can search for functions or methods, variables, operations, security flaws
and anything that can constitute a code segment. Users can search for strings as
simple as "a++" too complex methods. The search results can be further filtered
basis a particular repository or language. Do consider a few things before you hit
search.

10. Recorded Future

Recorded Future is an AI-based solution to trend prediction and big data

analysis. It uses various AI algorithms and both structured and unstructured
data to predict the future. The users can get past trends and future trends basis
OSINT data.

The utility of tools within the organization

Using the information that is freely available is an important task for the
organization (maybe not for all but mostly). Let's take a few examples where
these tools and manual means come to our rescue and generate something
that makes sense to the organization.

Another interesting read: Penetration Testing: Step-by-Step Guide, Stages,

Methods and Application

CASE 1: Hiring an employee

Company X wants to hire some employees for a team that is handling the data
which is sensitive to the organization. Usually, organizations do a background
check for the employees before hiring them. A background check will include
referring to various kinds of information to check the integrity of the information.
An employee might say that he has passed from B University presenting the
certificates as well. How do we ensure that the university is there or not?
Various universities offer search systems online that can be checked for
certificate validation. Police verification for criminal records and searching
various job portals to get the information is also a form of information.
Organisations share candidate feedback on the job portals as well just In case
they have to blacklist someone. Will you be interested in hiring a candidate that
is being blacklisted? Maybe yes, but this needs to be thought twice.
CASE 2: We are being attacked

Cybersecurity is booming because of increasing cyber threats. Let's consider a

scenario where an organization is getting large traffic from unexpected
countries. This raises a concern that this is a DOS attack. Referring to various
databases that are freely available on the internet can help you to get a clear
idea of the incident. Is the IP blacklisted? Is this traffic expected? Various
details about the source can be gathered from the internet using OSINT
techniques. A lot of free network-based tools are available on the internet that
has been connected to various databases for this kind of information.

CASE 3: We are the ORIGINALS

Organisations that deal with content can refer the Open intelligence to identify
the genuineness of the content.

 Universities can check for copied submissions of assignments and

projects.
 Industries can check for copied codes.
 Track usage of copyright images and content.
 Identify plagiarised articles and literature.

Explore OWASP- Top 10 Vulnerabilities in web applications (updated for

2018)

Conclusion
In a nutshell, the article has tried to cover 2 aspects of OSINT. One is the
technology bit of how OSINT can help. Second is the power of OSINT that can
help us in our day to day tasks. With all that information freely available multiple
actors can accomplish various tasks. A security professional can use the
information for data protection, security testing, incident handling, threat detection
etc. A threat actor, on the other hand, can gain information to perform phishing
attacks, targeted information gathering, DDOS attacks and much more.

Here we have discussed only a few aspects of the OSINT model but there is more
to it like Usernames, Emails, IP addresses, Domains, Media/videos, news,
articles, Social networks, people search, telephone numbers, transport, maps,
archives, metadata, search engines, dark web, digital currency, tools, malicious
content, documentation etc. The list can go on and on, it's just the requirement
that can help in selecting the right tools and techniques. Since this is all free,
users can make their own list as well where they can find information. A few
curated lists of such tools are already available on the internet, thanks to OSINT.
Check them out on Github or other reports. Good luck, Happy hunting!!!
Complementando o conhecimento de ferramentas

20 Recon and Intel Gathering Tools used by InfoSec

Professionals
Important note before we start: remember that you should never use
these tools on external networks/systems without previous authorization.
These tools are presented here in order to help IT security researchers and
private/public infosec investigators during the first phase of information
gathering, which is one of the most important parts of a cybersecurity
investigation.

1. OSINT Framework

While OSINT Framework isn't a tool to be run on your servers, it's a very
useful way to get valuable information by querying free search engines,
resources, and tools publicly available on the Internet. They are focused
on bringing the best links to valuable sources of OSINT data.

While this web application was originally created focused on IT security,

with the time it has evolved and today you can get other kinds of information
from other industries as well. Most of the websites it uses to query the
information are free, but some may require paying a low fee.
2. CheckUserNames

CheckUserNames is an online tool that can help you to find usernames

across over 170 social networks. This is especially useful if you are running
an investigation to determine the usage of the same username on different
social networks.

It can be also used to check for brand company names, not only individuals.

3. HaveIbeenPwned

HaveIbeenPwned can help you to check if your account has been

compromised in the past. This site was developed by Troy Hunt, one of the
most respected IT security professionals of this market, and it's been
serving accurate reports since years.
If you suspect your account has been compromised, or want to verify for
3rd party compromises on external accounts, this is the perfect tool. It can
track down web compromise from many sources like Gmail, Hotmail,
Yahoo accounts, as well as LastFM, Kickstarter, Wordpress.com, Linkedin
and many other popular websites.

Once you introduce your email address, the results will be displayed,
showing something like:

4. BeenVerified

BeenVerified is another similar tool that is used when you need to search
people on public internet records. It can be pretty useful to get more
valuable information about any person in the world when you are
conducting an IT security investigation and a target is an unknown person.

After done, the results page will be displayed with all the people that match
the person's name, along with their details, geographic location, phone
number, etc. Once found, you can build your own reports.

The amazing thing about BeenVerified it's that it also includes information
about criminal records and official government information as well.

BeenVerified background reports may include information from multiple

databases, bankruptcy records, career history, social media profiles and
even online photos.

5. Censys

Censys is a wonderful search engine used to get the latest and most
accurate information about any device connected to the internet, it can be
servers or domain names.

You will be able to find full geographic and technical details about 80 and
443 ports running on any server, as well as HTTP/S body content & GET
response of the target website, Chrome TLS Handshake, full SSL
Certificate Chain information, and WHOIS information.

6. BuiltWith

BuiltWith is a cool way to detect which technologies are used at any

website on the internet.

It includes full detailed information about CMS used like Wordpress,

Joomla, Drupal, etc, as well as full depth Javascript and CSS libraries like
jquery, bootstrap/foundation, external fonts, web server type (Nginx,
Apache, IIS, etc), SSL provider as well as web hosting provider used.
BuiltWith also lets you find which are the most popular technologies running
right now, or which ones are becoming trending.

Without any doubt, it is a very good tool to gather all the possible technical
details about any website.

7. Google Dorks

While investigating people or companies, a lot of IT security newbies forget

the importance of using traditional search engines for recon and intel
gathering.

In this case, Google Dorks can be your best friend. They have been there
since 2002 and can help you a lot in your intel reconnaissance.

Google Dorks are simply ways to query Google against certain information
that may be useful for your security investigation.

Search engines index a lot of information about almost anything on the

internet, including individual, companies, and their data.

Some popular operators used to perform Google Dorking:

 Filetype: you can use this dork to find any kind of filetypes.
 Ext: can help you to find files with specific extensions (eg. .txt, .log,
etc).
 Intext: can perform queries helps to search for specific text inside
any page.
 Intitle: it will search for any specific words inside the page title.
 Inurl: will look out for mentioned words inside the URL of any
website.

Log files aren't supposed to be indexed by search engines, however, they

do, and you can get valuable information from these Google Dorks, as you
see below:
Now let's focus on other more practical tools used by the most respected
InfoSec professionals:

8. Maltego

Is an amazing tool to track down footprints of any target you need to match.
This piece of software has been developed by Paterva, and it's part of the
Kali Linux distribution.

Using Maltego will allow you to launch reconnaissance testes against

specific targets.

One of the best things this software includes is what they call 'transforms'.
Transforms are available for free in some cases, and on others, you will
find commercial versions only. They will help you to run a different kind of
tests and data integration with external applications.

In order to use Maltego you need to open a free account on their website,
after that, you can launch a new machine or run transforms on the target
from an existing one. Once you have chosen your transforms, Maltego app
will start running all the transforms from Maltego servers.
Finally, Maltego will show you the results for the specified target, like IP,
domains, AS numbers, and much more.

If you need to explore more Kali Linux utilities, check out this article: Top
25 Kali Linux Tools

9. Recon-Ng

Recon-ng comes already built in the Kali Linux distribution and is another
great tool used to perform quickly and thoroughly reconnaissance on
remote targets.

This web reconnaissance framework was written in Python and includes

many modules, convenience functions and interactive help to guide you on
how to use it properly.

The simple command-based interface allows you to run common

operations like interacting with a database, run web requests, manage API
keys or standardizing output content.

Fetching information about any target is pretty easy and can be done within
seconds after installing. It includes interesting modules like
google_site_web and bing_domain_web that can be used to find valuable
information about the target domains.

While some recon-ng modules are pretty passive as they never hit the
target network, others can launch interesting stuff right against the remote
host.
10. theHarvester

theHarvester is another great alternative to fetch valuable information

about any subdomain names, virtual hosts, open ports and email address
of any company/website.

This is especially useful when you are in the first steps of a penetration test
against your own local network, or against 3rd party authorized networks.
Same as previous tools, theHarvester is included inside Kali Linux distro.

theHarvester uses many resources to fetch the data like PGP key servers,
Bing, Baidu, Yahoo and Google search engine, and also social networks
like Linkedin, Twitter and Google Plus.

It can also be used to launch active penetration test like DNS brute force
based on dictionary attack, DNS reverse lookups and DNS TLD expansion
using dictionary brute force enumeration.
11. Shodan

Shodan is a network security monitor and search engine focused on the

deep web & the internet of things. It was created by John Matherly in 2009
to keep track of publicly accessible computers inside any network.

It is often called the 'search engine for hackers', as it lets you find and
explore a different kind of devices connected to a network like servers,
routers, webcams, and more.

Shodan is pretty much like Google, but instead of showing you fancy
images and rich content / informative websites, it will show you things that
are more related to the interest of IT security researchers like SSH, FTP,
SNMP, Telnet, RTSP, IMAP and HTTP server banners and public
information. Results will be shown ordered by country, operating system,
network, and ports.

Shodan users are not only able to reach servers, webcams, and routers. It
can be used to scan almost anything that is connected to the internet,
including but not limited to traffic lights systems, home heating systems,
water park control panels, water plants, nuclear power plants, and much
more.

12. Jigsaw

Jigsaw is used to gather information about any company employees. This

tool works perfectly for companies like Google, Linkedin, or Microsoft,
where we can just pick up one of their domain names (like google.com),
and then gather all their employee's emails on the different company
departments.

The only drawback is that these queries are launched against Jigsaw
database located at jigsaw.com, so, we depend entirely on what
information they allow us to explore inside their database. You will be able
to find information about big companies, but if you are exploring a not so
famous startup then you may be out of luck.

13. SpiderFoot

SpiderFoot is one of the best reconnaissance tools out there if you want to
automate OSINT and have fast results for reconnaissance, threat
intelligence, and perimeter monitoring.

It was written by our friend Steve Micallef, who did a great job building this
app and writing the SecurityTrails Addon for Splunk
This recon tool can help you to launch queries over 100 public data sources
to gather intelligence on generic names, domain names, email addresses,
and IP addresses.

Using Sipiderfoot is pretty much easy, just specify the target, choose which
modules you want to run, and Spiderfoot will do the hard job for you
collecting all the intel data from the modules.

14. Creepy

Creepy is a geo-location OSINT tool for infosec professionals. It offers the

ability to get full geolocation data from any individuals by querying social
networking platforms like Twitter, Flickr, Facebook, etc.

If anyone uploads an image to any of these social networks with

geolocation feature activated, then you will be able to see a full active mal
where this person has been.

You will be able to filter based on exact locations, or even by date. After
that, you can export the results in CSV or KML format.

15. Nmap

Nmap is one of the most popular and widely used security auditing tools,
its name means "Network Mapper". Is a free and open source utility utilized
for security auditing and network exploration across local and remote
hosts.

Some of the main features include:

  Host detection: Nmap has the ability to identify hosts inside any
network that have certain ports open, or that can send a response to
ICMP and TCP packets.
 IP and DNS information detection: including device type, Mac
addresses and even reverse DNS names.
 Port detection: Nmap can detect any port open on the target network,
and let you know the possible running services on it.
 OS detection: get full OS version detection and hardware
specifications of any host connected.
 Version detection: Nmap is also able to get application name and
version number.

16. WebShag

WebShag is a great server auditing tool used to scan HTTP and HTTPS
protocols. Same as other tools, it's part of Kali Linux and can help you a lot
in your IT security research & penetration testing.

You will be able to launch a simple scan, or use advanced methods like
through a proxy, or over HTTP authentication.

Written in Python, it can be one of your best allies while auditing systems.

Main features include:

 Port Scan
 URL scanning
 File fuzzing
 Website crawling

In order to avoid getting blocked by remote server security systems, it uses

an intelligent IDS evasion system by launching random requests per HTTP
proxy server, so you can keep auditing the server without being banned.

17. OpenVAS

OpenVAS (Open Vulnerability Assessment System) is a security

framework that includes particular services and tools for infosec
professionals.

This is an open source vulnerability scanner & security manager that was
built after the famous Nessus switched from open source to private source.
Then, the original developers of the Nessus vulnerability scanner decided
to fork the original project and create OpenVAS.

While it is a little bit more difficult to setup than the old Nessus, it's quite
effective while working with it to analyze the security of remote hosts.
The main tool included in OpenVAS is OpenVAS Scanner, a highly efficient
agent that executes all the network vulnerability tests over the target
machine.

On the other hand, another main component is called OpenVAS Manager,

which is basically vulnerability management solution that allows you to
store scanned data into an SQLite database, so then you can search, filter
and order the scan results in a fancy and easy way.

18. Fierce

Fierce is an IP and DNS recon tool written in PERL, famous for helping IT
sec professionals to find target IPs associated with domain names.

It was written originally by RSnake along with other members of the old
http://ha.ckers.org/. It's used mostly targetting local and remote corporate
networks.

Once you have defined your target network, it will launch several scans
against the selected domains and then it will try to find misconfigured
networks and vulnerable points that can later leak private and valuable
data.

The results will be ready within a few minutes, a little bit more than when
you perform any other scan with similar tools like Nessus, Nikto,
Unicornscan, etc.

19. Unicornscan

Unicornscan is one of the top intel gathering tools for security research. It
has also a built-in correlation engine that aims to be efficient, flexible and
scalable at the same time.
Main features include:

 Full TCP/IP device/network scan.

 Asynchronous stateless TCP scanning (including all TCP Flags
variations).
 Asynchronous TCP banner detection.
 UDP Protocol scanning.
 A/P OS identification.
 Application and component detection.
 Support for SQL Relational Output

20. Foca

FOCA (Fingerprinting Organizations with Collected Archives) is a tool

written by ElevenPaths that can be used to scan, analyze, extract and
classify information from remote web servers and their hidden information.

Foca has the ability to analyze and collect valuable data from MS Office
suite, OpenOffice, PDF, as well as Adobe InDesign and SVG and GIF files.
This security tool also works actively with Google, Bing and DuckDuckGo
search engines to collect additional data from those files. Once you have
the full file list, it starts extracting information to attempt to identify more
valuable data from the files.

As you can see, there are a lot of recon and intel gathering tools out there.
On this particular post, we mentioned only 20 of the most popular tools, but
there is much more to discover. Start digging around and testing other
useful infosec tools.

Want to try one of the best intel gathering tools in the market? Start
using SecurityTrails, our intelligent security toolkit built for InfoSec
professionals who need to gather the precise IP, DNS and Domain
information to protect their companies.

Or sign up for a free API access today to integrate your apps with our
intelligent security engine!