Load Balancing Scheme on the Basis of Huffman Coding for P2P Information Retrieval

Hisashi KURASAWA, Atsuhiro TAKASU, Jun ADACHI
2009 IEICE transactions on information and systems  
Hisashi KURASAWA †a) , Atsuhiro TAKASU † †b) , and Jun ADACHI † †c) , Members SUMMARY Although a distributed index on a distributed hash table (DHT) enables efficient document query processing in Peer-to-Peer information retrieval (P2P IR), the index costs a lot to construct and it tends to be an unfair management because of the unbalanced term frequency distribution. We devised a new distributed index, named Huffman-DHT, for P2P IR. The new index uses an algorithm similar to Huffman coding
more » ... a modification to the DHT structure based on the term distribution. In a Huffman-DHT, a frequent term is assigned to a short ID and allocated a large space in the node ID space in DHT. Throuth ID management, the Huffman-DHT balances the index registration accesses among peers and reduces load concentrations. Huffman-DHT is the first approach to adapt concepts of coding theory and term frequency distribution to load balancing. We evaluated this approach in experiments using a document collection and assessed its load balancing capabilities in P2P IR. The experimental results indicated that it is most effective when the P2P system consists of about 30,000 nodes and contains many documents. Moreover, we proved that we can construct a Huffman-DHT easily by estimating the probability distribution of the term occurrence from a small number of sample documents. key words: peer-to-peer, information retrieval, load balancing, Huffmancoding
doi:10.1587/transinf.e92.d.2064 fatcat:v4k6oiwtnzggrmru6vxol7dt6u