This paper researches feature of different data sources such as web site and social media, and proposes a load balancing method for distributed web crawlers by calculating weights of crawling data from various sources. Firstly, a seeding links allocation strategy is proposed based on analyzing differences of statistical data update frequency of different data sources. Then with the allocation strategy a data crawling solution using domain names and keywords as its task unit is given. Finally,

more »

... adjusting allocation among distributed web crawlers with calculating time expenditures of domain names and keywords as its weights, a load balancing method is proposed.