TEXT

AUTOMATIC TEMPLATE EXTRACTION FROM HETEROGENEOUS WEB PAGES

AIM
The main aim of this project is to provide reliable and fast

webpage in many websites are automatically populated by using the

common templates with contents.

SYNOPSIS
In this paper, we present novel algorithms for extracting templates from a large number of web documents which are generated from heterogeneous templates. We cluster the web documents based on the similarity of underlying template structures in the documents so that the template for each cluster is extracted

simultaneously. We develop a novel goodness measure with its fast

approximation for clustering and provide comprehensive analysis of our algorithm. Our experimental results with real-life data sets confirm the effectiveness and robustness of our algorithm compared to the state of the art for template detection algorithms.

EXISTING SYSTEM
Due to the assumption of all documents being generated from a single common template, solutions for this problem are applicable only when all documents are guaranteed to conform to a common template. However, in real applications, it is not trivial to classify massively crawled documents into homogeneous partitions

in order to use these techniques.. If we use only URLs to group

pages, these pages from the different templates will be included in the same cluster.

PROPOSED SYSTEM
Our work is different from the existing content

discovery schemes for storage-forwarding systems in the following:

In this paper, in order to alleviate the limitations of the state-of-the-art technologies, we investigate the problem of detecting the templates from heterogeneous web documents and present novel algorithms called TEXT (automatic template extraction).

1) Our goal is to manage an unknown number of templates

and to improve the efficiency and scalability of template detection and extraction algorithms. To deal with the unknown number of templates and select good partitioning from all possible partitions of web documents, we employ Rissanens Minimum Description Length (MDL) principle. 2) In our method, document clustering and template

extraction are done together at once. Since a large number of web

documents are massively crawled from the web quickly, so that a large number of documents can be processed.

MODULES

Template Architecture Design.

Template Extraction. Clustering.

Template Architecture Design

In this module we interact with the user to collect the user informations. This module is used to develop the GUI

design for the clients, which is easily understood to interact with this
project. This module developed by servlet package, which is present in J2EE.

Template Extraction
If any of a query searched throughout the networks previously servers organize only URL if it matches transfer the control to those templates. Over here we extract multiple temples from multiple sites. And finally extract which one is properly suite to our query fully extracted and frame it on common template. This form of formation is simply called as template extraction.

Clustering
TEXT-MDL is an agglomerative hierarchical

clustering which starts with each input document as an individual

cluster.
When we merge clusters hierarchically, we select two clusters which maximize the reduction of the MDL cost by merging them. Given a cluster ci, if a cluster cj maximizes the reduction of the MDL cost, we call cj the nearest cluster of ci. In order to efficiently find the nearest cluster of ci.

ARCHITECTURE DIAGRAM

Clients

Search

Server

TEXT Extraction of related URL

Auto Template Formation

Clustering

SOFTWARE REQUIREMENTS
Windows XP

JDK 1.6
Servlet, JSP Apache Tomcat

HARDWARE REQUIREMENTS
Hard Disk: 20GB and Above RAM: 512MB and Above Processor: Pentium III and Above