Evaluating Retrieval Effectiveness by Sustainable Rank List
The Internet of Things (IoT) and Big Data are among the most popular emerging fields of computer science today. IoT devices are creating an enormous amount of data daily on a different scale; hence, search engines must meet the requirements of rapid ingestion and processing followed by accurate and fast extraction. Researchers and students from the field of computer science query the search engines on these topics to reveal a wealth of IoT-related information. In this study, we evaluate the
... we evaluate the relative performance of two search engines: Bing and Yandex. This work proposes an automatic scheme that populates a sustainable optimal rank list of search results with higher precision for IoT-related topics. The proposed scheme rewrites the seed query with the help of attribute terms extracted from the page corpus. Additionally, we use newness and geo-sensitivity-based boosting and dampening of web pages for the re-ranking process. To evaluate the proposed scheme, we use an evaluation matrix based on discounted cumulative gain (DCG), normalized DCG (nDCG), and mean average precision (MAP n ). The experimental results show that the proposed scheme achieves scores of MAP@5 = 0.60, DCG 5 = 4.43, and nDCG 5 = 0.95 for general queries; DCG 5 = 4.14 and nDCG 5 = 0.93 for time-stamp queries; and DCG 5 = 4.15 and nDCG 5 = 0.96 for geographical location-based queries. These outcomes validate the usefulness of the suggested system in helping a user to access IoT-related information. For example, to handle this burst of data, existing storage mechanisms are gradually replaced by new emerging technologies such as Platform-as-a-Service (PaaS) or cloud Spark. On a daily basis, new devices are added to the Internet world and they share a huge amount of data. This addition of devices is making the data analysis to derive information increasingly difficult. Data analysis is a necessary operation to find the hidden patterns and trends in data. Nowadays, almost every Internet user seems to be interested in topics related to IoT. The question that arises here is whether present search engines can find this information or not. The IoT produces a different scale of data, so the search engines should be able to meet the requirements of rapid ingestion and processing followed by accurate and fast extraction. In large-scale IoT systems, search engine techniques are crucial for the efficient retrieval of desired data. Nowadays, half of the world population uses the Internet and most of them use search engines  . The information related to the latest breakthroughs of IoT is also searched with the help of search engines. The principal challenge for search engines is to achieve an optimal ranking of returned web pages based on users' information need regarding IoT. It has become very important to observe the quality of search results in dynamic environments of IoT and Big Data. The performance-related statistics found in such studies can be used by search engine industries to offer improved and up-to-date services. Measuring the effectiveness for a larger database of search engines by using human judgment is becoming impossible owing to cost, time, and regular scheduling. With these points in mind, relevance assessments should be transformed toward automatic schemes. This study examined the ranking performance of search engines against IoT-related queries. Additionally, we proposed a method of relative performance evaluation of search engines using an automatically prepared sustainable optimal rank list for IoT-related searches. Mostly automatic relevance assessments have performed using two approaches: a black box approach and a clear box approach. The black box approach gave search engines the freedom to use different methods. To evaluate the performance of a web search engine, we need a set of web pages, a set of queries, and relevance-judgment-related information about each web page. Usually, it is difficult to obtain relevance responses for all indexed pages. Therefore, pooling (fraction of top similar documents regarding queries and relevance assessment) is used by the Text Retrieval Conference (TREC) to solve the problem of size  . Zobel studied the reliability of pooling while evaluating the performance of Information Retrieval (IR) for a large set of web pages  . Harter and Voorhees [6, 7] found that assessors usually disagree while assessing relevance but this disagreement does not matter for the assessment of the relative effectiveness of IR. Modern search engines consume diverse kinds of signals to categorize web pages, to find locations, to read behavioral aspects, etc. Search engine rank completes the task of ranking with the help of various signals while searching. Scoring: Modern search engines manipulate hundreds of signals. A variety of algorithms used these signals as input in different combinations for calculating scores for the ranking. Boosting: These search engines consider multiple factors in the calculation of the final score. They give some factors more weight than others, and if a page holds such factors than it will be promoted to the rank. For example, fast mobile sites are boosted while user searching from the Google app. Dampening: These factors can affect the higher rank of a web page. It is taken as a penalty; this factor stops a page from getting a top rank if it does not satisfy the condition. The scheme described in this paper is not the first work related to the topic of automatic search engine performance evaluation. Thus far, researchers     have presented automatic evaluation systems for web search based on rank aggregation, data fusion, AWSEEM, and wisdom of crowds, among others. As we discussed above, IoT is the main emerging technology. Here, we proposed an automatic performance evaluation scheme for search engines based on Pseudo Relevance Feedback (PRF) for IoT-related user queries. We prepared information needs related to the top 10 most Googled IoT topics. We followed the topics given in study  , where the rank presents the sum for number Sustainability 2017Sustainability , 9, 1203 3 of 20 of searches at Google, Twitter, and LinkedIn. We then prepared 10 queries related to each of these topics with the help of students. After performing some processing on these 100 information needs, we selected 60 of them. We translated each of these information needs using an offline-made page corpus. After translation, the scheme executed the top-K translated queries on search engines and prepared a pool of documents. Afterward, the scheme calculated the score of text similarity among queries and web pages. Using this PRF-based scheme, we prepared a sustainable optimal rank list to measure the relative performance of the search engines. Lastly, the proposed PRF scheme performed boosting based on location and freshness of web pages to achieve a sustainable optimal rank. The proposed approach automatically figures out the relevance of web pages. To the best of our knowledge, this is the first study that examines the ranking performance of search engines for IoT-related queries. This study verifies the effectiveness of the proposed scheme for sustainable optimality with the help of evolution metrics based on discounted cumulative gain (DCG), normalized DCG (nDCG), and Mean Average Precision (MAP n ). In this work, we compare the proposed scheme results with two different search engines (Bing and Yandex). The experimental results showed that proposed scheme achieves satisfactory results for all the metrics used in this work. Table 1 below shows all the terms and abbreviations used in this work.