CRAWLER

1.0 Introduction:-

A web crawler (also known as robot or a spider) is a system for bulk downloading of web pages.Web
crawlers are used for variety of purposes.Most prominently, they are one of the main component of search
engine. System that assemble a corpus of web pages,index them, and allow users to issue queries against
index and find web pages that matches queries A related use is a web archiving, where large set of web
pages are periodically collected and archived for posterity.another use is web data mining, web pages are
analysed for statistical properties.or where data analytics is perform on them.
Web indexing (or Internet indexing) refers to various methods for indexing the contents of a website
or of the Internet as a whole. Individual websites or intranets may use a back-of-the-book index, while search
engines usually use keywords and metadata to provide a more useful vocabulary for Internet or onsite
searching. With the increase in the number of periodicals that have articles online, web indexing is also
becoming important for periodical websites.
Back-of-the-book-style web indexes may be called "web site A-Z indexes". The implication with "A-Z"
is that there is an alphabetical browse view or interface. This interface differs from that of a browse through
layers of hierarchical categories (also known as a taxonomy) which are not necessarily alphabetical, but are
also found on some web sites. Although an A-Z index could be used to index multiple sites, rather than the
multiple pages of a single site, this is unusual.
A Web crawler starts with a list of Urls to visit, called the seeds. As the crawler visits these URLs, it
identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier.
URLs from the frontier are recursively visited according to a set of policies.

The large volume implies that the crawler can only download a limited number of the Web pages within
a given time, so it needs to prioritize its downloads. The high rate of change implies that the pages might have
already been updated or even deleted.
The number of possible crawlable URLs being generated by server-side software has also made it
difficult for web crawlers to avoid retrieving duplicate content. Endless combinations of HTTP GET (URLbased) parameters exist, of which only a small selection will actually return unique content. For example, a
simple online photo gallery may offer three options to users, as specified through HTTP GET parameters in
the URL. If there exist four ways to sort images, three choices of thumbnail size, two file formats, and an
option to disable user-provided content, then the same set of content can be accessed with 48 different URLs,
all of which may be linked on the site. This mathematical combination creates a problem for crawlers, as they
must sort through endless combinations of relatively minor scripted changes in order to retrieve unique
content.

Key design goals:

outstanding features:
Content-based indexing.
Breadth first search to create a broad index.
Crawler behavior to include as many web servers as possible.

How does a web crawler work?

A typical web crawler starts by parsing a specified web page: noting any hypertext links on that page that
point to other web pages. The Crawler then parses those pages for new links, and so on, recursively. A
crawler is a software or script or automated program which resides on a single machine. The crawler simply
sends HTTP requests for documents to other machines on the Internet, just as a web browser does when the
user clicks on links. All the crawler really does is to automate the process of following links.
This is the basic concept behind implementing web crawler, but implementing this concept is not
merely a bunch of programming. The next section describes the difficulties involved in implementing an
efficient web crawler.

1.2) Presently available crawlers:-

The following is a list of published crawler architectures for general-purpose crawlers (excluding focused web
crawlers), with a brief description that includes the names given to the different components and outstanding
features:
Yahoo! Slurp is the name of the Yahoo! Search crawler.
Bingbot is the name of Microsoft's Bing webcrawler. It replaced Msnbot.
FAST Crawler is a distributed crawler, used by Fast Search & Transfer, and a general description of its
architecture is available.
Googlebot is described in some detail, but the reference is only about an early version of its architecture,
which was based in C++ and Python. The crawler was integrated with the indexing process, because text
parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of
URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL
server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL
server.
PolyBot is a distributed crawler written in C++ and Python, which is composed of a "crawl manager", one or
more "downloaders" and one or more "DNS resolvers". Collected URLs are added to a queue on disk, and
processed later to search for seen URLs in batch mode. The politeness policy considers both third and
second level domains (e.g.: www.example.com and www2.example.com are third level domains) because
third level domains are usually hosted by the same Web server.
RBSE was the first published web crawler. It was based on two programs: the first program, "spider"
maintains a queue in a relational database, and the second program "mite", is a modified www ASCII browser
that downloads the pages from the Web.
WebCrawler was used to build the first publicly available full-text index of a subset of the Web. It was based
on lib-WWW to download pages, and another program to parse and order URLs for breadth-first exploration
of the Web graph. It also included a real-time crawler that followed links based on the similarity of the anchor
text with the provided query.
World Wide Web Worm was a crawler used to build a simple index of document titles and URLs. The index
could be searched by using the grep Unix command.
WebFountain is a distributed, modular crawler similar to Mercator but written in C++. It features a "controller"
machine that coordinates a series of "ant" machines. After repeatedly downloading pages, a change rate is
inferred for each page and a non-linear programming method must be used to solve the equation system for
maximizing freshness. The authors recommend to use this crawling order in the early stages of the crawl, and
then switch to a uniform crawling order, in which all pages are being visited with the same frequency.
WebRACE is a crawling and caching module implemented in Java, and used as a part of a more generic
system called eRACE. The system receives requests from users for downloading web pages, so the crawler
acts in part as a smart proxy server. The system also handles requests for "subscriptions" to Web pages that
must be monitored: when the pages change, they must be downloaded by the crawler and the subscriber
must be notified. The most outstanding feature of WebRACE is that, while most crawlers start with a set of
"seed" URLs, WebRACE is continuously receiving new starting URLs to crawl from.
In addition to the specific crawler architectures listed above, there are general crawler architectures published
by Cho and Chakrabarti.

Open-source crawlers

DataparkSearch is a crawler and search engine released under the GNU General Public License.
GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used
to mirror Web and FTP sites.
GRUB is an open source distributed search crawler that Wikia Search used to crawl the web.
Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large
portion of the Web. It was written in Java.
ht://Dig includes a Web crawler in its indexing engine.
HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released
under the GPL.
ICDL Crawler is a cross-platform web crawler written in C++ and intended to crawl Web sites based on
Website Parse Templates using computer's free CPU resources only.
mnoGoSearch is a crawler, indexer and a search engine written in C and licensed under the GPL (Linux
machines only)
Norconex HTTP Collector is a web spider, or crawler, written in Java, that aims to make Enterprise Search
integrators and developers's life easier (licensed under GPL).
Nutch is a crawler written in Java and released under an Apache License. It can be used in conjunction with
the Lucene text-indexing package.
Open Search Server is a search engine and web crawler software release under the GPL.
PHP-Crawler is a simple PHP and MySQL based crawler released under the BSD License. Easy to install it
became popular for small MySQL-driven websites on shared hosting.
the tkWWW Robot, a crawler based on the tkWWW web browser (licensed under GPL).
Scrapy, an open source webcrawler framework, written in python (licensed under BSD).
Seeks, a free distributed search engine (licensed under Affero General Public License).
YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL).

1.3) Need for New System

we make crawler for personal interest & assistant.

Why do we need a web crawler?

Following are some reasons to use a web crawler:

To maintain mirror sites for popular Web sites.
To test web pages and links for valid syntax and structure.
To monitor sites to see when their structure or contents change.
To search for copyright infringements.
To build a special-purpose index.for example, one that has some understanding of the content stored in
multimedia files on the Web.

How Do Search Engines Work?

Before a search engine can do anything it must first discover web pages. This is the task of the search engine
spiders, also known as bots, robots or crawlers; the spiders for the three major search engines are MSNbot
2 (Bing), Googlebot 2.1 (Google) and Slurp (Yahoo!), but there are many, many more and they all perform
much the same task.
These spiders are pieces of software that follow links around the internet. Each page they access is sent back
to a data centre, the data centre is a vast warehouse containing thousands of computers. Once a page is
stored in a data centre, the search engine can begin to analyse it and thats where the magic starts to happen.
Conceptually, each spider will have started from a single page on the internet (historically the DMOZ directory
was the starting point for many), and will have been crawling pages by following links from that day to the
present. This is a massive, constant task, involving accessing and storing billions of pages every day, and the
scale of the problem is one of the reasons there are so few major search engines around today.
Its important to note that at this stage in the search engine process there is no itelligence or clever algorithm
at work. The spiders are relatively simple bits of software, they follow links, harvest whatever data they can,
and send it back to the data centre, then follow the next set of links, and so on. Its all very robotic, which is
why search engines can so easily be stymied by non standard content or navigation, such as Flash movies or
forms and the like.
Key points to remember about crawling:
The job of the crawlers is to discover new content. They do this by following links.
Crawling is a massive, constant process, and the search engines crawl billions of pages every day, finding
new content and recrawling old content to check if its changed.
Search engine crawlers arent smart; they are simple bits of software programmed to singlemindedly collect
data and send it back to the search engine data centres.

Caching
Once a page has been crawled search engines will typically take a cache of the page. This means the entire
page, including its content, images, styles, scripts and source code, is stored by the search engine verbatim.
This cache usually becomes available, via the cached link in the search results or via the cache: search

operator, a few days after the crawl date, allowing users to access the page as it existed at the time it was
crawled and as the search engine spider saw it.

In practice the cache functionality is rarely used by most users, but for those interested in SEO it can be
invaluable because it serves to highlight accessibility issues with your pages. For example, if your pages dont
have a cache, or the cache is significantly different to what you see in your own browser perhaps it has
some key areas of content missing, or there is no clickable navigation to other pages on your site then you
know there is a good chance that there is some sort of issue preventing the spiders from properly accessing
your pages or their content.
Key points to remember about caching:
The cache is a literal copy of what the spiders crawled; its not searchable, but is sometimes useful for
assessing how spiders interact with your site.
If the search engines dont have a cache of your pages, or if your cache differs in important ways from what
you see in your browser, you could have an accessibility problem.

Indexing

Although the cache is a useful tool, to a search engine it has limited applications. In this state, a page cant be
searched by the search engine. To be searchable, a page must be indexed. This is the next stage and
involves deconstructing the page to its constituent parts, and databasing it so it can be easily located and
retrieved by the search engine later on, and compared to other pages.
It helps to think of a reference book in this respect. When looking for a specific piece of information, you dont
leaf through every page looking for mentions of a key word or phrase, instead you look at the index. Search
engine indexes function in a similar way, only across billions of documents rather than the few hundred pages
that a typical book index might cover.
You can check if a page is indexed by a search engine by performing a site: query on it. If this search returns
no results your page isnt indexed and, assuming enough time has passed and youd ordinarily expect the
page to be indexed, you may have an accessibility problem preventing the page from being crawled and/or
indexed.
Key points to remember about indexing:

Search engine indexes are analagous to the index youll find at the back of most reference books, which
allows you to quickly flip to the right page when you are looking for information on a specific topic.
They are the key to search engines being able to search hundreds of thousands, if not millions, of documents
so quickly.
Your pages must be in the index before they can rank in the search engine results. If they arent, you may
have an accessibility problem.

Retrieval

Having crawled, cached and indexed a page, its then ready to be returned by the search engine in response
to a users search query.
Lets assume that somebody searches for search engine optimisation. The first thing the search engine will
do is access a data centre (typically the nearest one) and retrieve every document that has been indexed that
the engine considers to be relevant for the term search engine optimisation. This often amounts to hundreds
of thousands or even millions of documents. This is the pool of results that will then be sorted by the search
engine in the final stage.
This then is the second hurdle for SEO; once youve ensured your content is accessible in order that it can
be crawled, cached and indexed, you must also make sure its relevant for the terms that your target market
is typing into search engines. The easiest way to be considered relevant for a search term is to include that
term on one of your pages. Some other signals, such as the text of links pointing to your pages, may be
considered, but the vast majority of retrieved pages simply use the term in question in their content.
Key points to remember about retrieval:
After you click Search, every document that is relevant to the term you searched for is retrieved from the
nearest data centre.
If your pages arent relevant to the terms people search for, they cant be retrieved by the search engine or
considered during the next and final stage, ranking.

Sorting

In the final stage, search engines take all of the documents they retrieved in the previous step and pass them
through their algorithms in order to sort the documents into the order that they think best serves the users
intent. The sorted documents are returned in a SERP, or search engine results page.
For all the scale and complexity of the previous stages, the algorithms that do the sorting are the real
workhorses. They analyse dozens or even hundreds of factors about each page, and they do this in mere
fractions of a second. Needless to say search engines dont reveal any specific details about their algorithms,
although we do know the general concepts behind them, which are similar for all of the major engines.
For a large number of searches most of the sorting will again be done based on the content of the pages
retrieved. This is classic information retrieval based on the frequency of words appearing on the page, how
they are emphasised and so on. So, again, relevance is of paramount importance.
For more competitive search terms, search engines will find that too many documents are all equally as
relevant as each other. At this point they will have to look to other signals to distinguish between the
documents, and the most important of these is links from other sites. These confer credibility to your pages.
This then is the final piece of the SEO puzzle.
Key points to remember about sorting:
Every document retrieved during the previous stage is fed through a complex formula, the algorithm, in order
to sort them into the order that the search engine thinks is best for your needs.
All of the major search engines use similar principles in their algorithms, differing only in the details and the
emphasis they apply to individual elements.
For many search terms, being relevant isnt always enough to ensure high rankings; you also need credibility
in the form of links from other websites.
Once sorted according to the algorithm, the results are returned to the user via the search engine results
pages, or SERPs for short.

ARC

ARC is the cornerstone of Greenlights SEO methodology and, as weve explained, its three parts
accessibility, relevancy and credibility correspond with how search engines work so that every potential SEO
issue is considered logically and methodically. Remember:
Your pages and content must be accessible to search engines before they can be crawled, cached and
indexed.

Search engines consider the relevancy of your pages during the retrieval and sorting processes.
Credibility is important to distinguish your site from the thousands of others sites that are equally as
accessible and relevant.
If you think your site falls down in one or more of these areas, we can help. Please get in touch.

1.4) Project Scope:There are basically three steps that are involved in the web crawling procedure. First, the search bot
starts by crawling the pages of your site. Then it continues indexing the words and content of the site, and
finally it visit the links (web page addresses or URLs) that are found in your site. When the spider doesnt find
a page, it will eventually be deleted from the index. However, some of the spiders will check again for a
second time to verify that the page really is offline.
Crawlers are typically programmed to visit sites that have been submitted by their owners as new or
updated. Entire sites or specific pages can be selectively visited and indexed. Crawlers apparently gained the
name because they crawl through a site a page at a time, following the links to other pages on the site until all
pages have been read.

Crawling policy

The behavior of a Web crawler is the outcome of a combination of policies:

a selection policy that states which pages to download,
a re-visit policy that states when to check for changes to the pages,
a politeness policy that states how to avoid overloading Web sites, and
a parallelization policy that states how to coordinate distributed web crawlers.

From above we are going to decide our crawlers policies and its functionality.

(The proposed use case will be given by saurabh by the time.)

http://en.wikipedia.org/wiki/Web_crawler
http://www.slideshare.net/hustwj/10111639242
http://www.sciencedaily.com/articles/w/web_crawler.htm
http://viralpatel.net/blogs/how-to-write-a-web-crawler-in-java/
http://wiki.answers.com/Q/What_are_the_disadvantages_of_web_crawlers
http://wiki.answers.com/Q/What_are_advantages_of_web_crawlers
http://www.princeton.edu/~achaney/tmve/wiki100k/docs/Web_crawler.html
http://ijcsi.org/papers/IJCSI-8-6-1-309-313.pdf
http://www.devbistro.com/articles/Misc/Implementing-Effective-Web-Crawler
http://www.wpthemesplanet.com/2009/09/how-does-web-crawler-spider-work/

S/W & H/W Requirements:H/W:- At least 20 GB HDD,1 GB RAM

S/W:FrontEnd:-AWT,Swing
BackEnd:-java
Technology:- JSP-Servelet,java
Software:- JDK (1.5 or above),Event