A web crawler is a relatively simple automated program that methodically scans or "crawls" through Internet pages to create an index of the data it is looking for. Process is called Web crawling or spidering.Basically web crawler provides scope to search engine, for the purpose of finding web pages from the World Wide Web in simply manner. In today's world each search engine has its own crawler and crawling techniques. In this paper we introduce search engines, different types of crawling

more »

... ques, crawler architecture, crawler algorithms, diff. types of crawler issues and types of crawler. Need of web crawling is To Test web pages and links for valid syntax and structure and search for copyright permissions. A front-end user interface and a supporting querying engine which queries the database and presents the results of searches so in this way web crawling works with search engine.