Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics.

However, this viewpoint diversity is not trivial to assess. In this paper we use existing and novel ranking fairness metrics to evaluate viewpoint diversity in search result rankings. ... This paper lays out important ground work for future research to measure and assess viewpoint diversity in real search result rankings. ... In this paper, we investigate whether ranking fairness metrics can be used to assess viewpoint diversity in search results. ...

arXiv:2010.14531v1 fatcat:mivwonmrt5axpgvq6w5mx7tvzu

Multiple Versions

One result of such biases is the search engine manipulation effect (SEME): when a list of search results relates to a debated topic (e.g., veganism) and promotes documents pertaining to a particular viewpoint ... (e.g., by ranking them higher), users tend to adopt this advantaged viewpoint [5] . ... The first contribution to RQ2 was demonstrating how to assess viewpoint-related ranking bias in search results using ranking fairness metrics for categorical viewpoint labels and evaluated which specific ...

doi:10.1145/3404835.3463273 fatcat:3b6kh43fwrforky7a2gebahy7e

Our offline evaluation indicates that the proposed method is capable of enhancing the viewpoint diversity of recommendation lists according to a diversity metric from literature. ... While viewpoint diversity is often considered synonymous with source diversity in communication science domain, in this paper, we take a computational view. ... We would like to thank the team at Blendle for their support in this research, as well as the anonymous users that participated in our user study. ...

doi:10.1145/3442188.3445911 fatcat:holad4zstzfkbjqqlxyhsppll4

While several methods in information retrieval, data clustering, and similarity searching have tackled the problem of adding diversity into result sets, the experimental comparison of their performances ... We conclude diversity is better addressed by a set of measures rather than a single quality index and introduce the concept of Diversity Features Model (DFM), which combines the viewpoints of biased metrics ... , coined Diversity Features Model (DFM), which combines previous relevant metrics from data clustering and similarity-searching for assessing the quality of diversified similarity searching. ...

doi:10.5753/jidm.2021.1917 fatcat:syer3zrikbhsxcbtezjjhijpou

Search queries are often ambiguous and/or underspecified. To accomodate different user needs, search result diversification has received attention in the past few years. ... Our results show that diversified IR experiments with a given number of topics can be as reliable as traditional IR experiments with the same number of topics, provided that the right metrics are used. ... [8] proposed a metric called α-nDCG for evaluating diversified search results. ...

doi:10.1145/2009916.2010055 dblp:conf/sigir/SakaiS11 fatcat:rh6vfuva3fcmjbpgrujhuvmnf4

For instance, a viewpoint's stance usually has a degree of strength (e.g., mild or strong), and, even if two viewpoints support or oppose something to the same degree, they may use different logics of ... By analyzing the resulting data set and conducting a user study, we further show that the two-dimensional viewpoint representation we propose allows for more meaningful analyses and diversification interventions ... Such labels can broadly categorize viewpoints while allowing for cheap computation of metrics and algorithms (e.g., related to ranking fairness [16, 60, 62] ). ...

doi:10.1145/3498366.3505812 fatcat:q2r7dxikpff63dynecwn2u2rvq

Several companies use the amount of deliveries as a metric of performance evaluation of the developer. ... Metrics Productivity Past experiences Specialization (expert in some technology or tool) Generalization (diversity of skills) Solve complex problems Metrics The main behavior of the developer ... Survey 1) Goal-Question-Metric We used an approach called Goal Question Metric (GQM) [27] that helped us define our survey. ...

doi:10.18293/seke2015-56 dblp:conf/seke/TangariM15 fatcat:nr3ctklctvatteudhy2t5bz4qu

In particular, we show through experimentation how the current practice in Information Retrieval evaluation of using a measurement depth larger than the pooling depth increases uncertainty during system ... This new methodology aims to identify potential uncertainty during system comparison that may result from incompleteness. ... This measures the concordance in the ranking of all systems in terms of Performance and Assessment metrics. ...

doi:10.1016/j.ipm.2007.04.002 fatcat:gabq7nsuyrhwjewo2eph2zi4rm

Afterward, we provide a survey of how research in this area is currently operationalized, for example, in terms of the general research methodology, fairness metrics, and algorithmic approaches. ... a fair recommendation in the context of a given application. ... We discuss evaluation metrics next. Evaluation Metrics. We found that in offline experiments a wide range of computational metrics is used to assess the fairness of a set of recommendations. ...

arXiv:2205.11127v2 fatcat:qcq5iuwlevg2dh54i4on5jj4hi

Open Access Multiple Versions

The results show numerous bias and unfairness detection and mitigation approaches for ML technologies, with clearly defined metrics in the literature, and varied metrics can be highlighted. ... This study aims to examine existing knowledge on bias and unfairness in Machine Learning models, identifying mitigation methods, fairness metrics, and supporting tools. ... : Planning, Scoping, Searching, Assessing, Synthesizing. ...

arXiv:2202.08176v2 fatcat:m7rexaf26ra3zadl2dsqaqmjgq

Open Access Multiple Versions

Citation

Tiago Palma Pagano, Rafael Bessa Loureiro, Maira Matos Araujo, Fernanda Vitoria Nascimento Lisboa, Rodrigo Matos Peixoto, Guilherme Aragao de Sousa Guimaraes, Lucas Lisboa dos Santos, Gustavo Oliveira Ramos Cruz, Ewerton Lopes Silva de Oliveira, Marco Cruz, Ingrid Winkler, Erick Giovani Sperandio Nascimento. "Bias and unfairness in machine learning models: a systematic literature review." arXiv (2022)

The prospective site quality measurements could be used as a site assessment system in order to assess functional sites and dispense superiority scores. ... In this paper the website superiority metrics and means to compute the website interface has been proposed. ... The functionality assessment result is depicted in Figure 3 . For the sake of fair evaluation of the degree of functionality in a webpage, every sub-attribute has to specify the weight. ...

doi:10.17485/ijst/2016/v9i40/100229 fatcat:fxbrimhxbvd73aksvy6olio2zu

We describe cases where real recommender systems were modified in the service of various human values such as diversity, fairness, well-being, time well spent, and factual accuracy. ... Instead, we look to AI alignment work for approaches that could learn complex values directly from stakeholders, and identify four major directions: useful measures of alignment, participatory design and ... Suites of metrics for "well-being" are used in public policy (O'Donnell et al., 2014 ) and many of those most applicable to AI systems have been collected into an IEEE standard for well-being assessment ...

arXiv:2107.10939v1 fatcat:e2mzj4l3lbg5ter5yo37pbki3m

Open Access

Search algorithms based on evolutionary multiobjective optimization (EMO) are then proposed to obtain diverse solutions in the feasible solution space. ... We first design simple heuristics to obtain the extreme points in the solution space, which however, could not produce diverse solutions. ... Kumar and Rockett [20] proposed use of rank-histograms to assess movement of solution-front towards convergence. ...

doi:10.1145/1143997.1144274 dblp:conf/gecco/KumarSB06 fatcat:q2a2l2q33vfvrn6esxy6bvncqm

Transparency is important for facilitating fairness management as well as explainability in algorithms; however, the concept of diversity, and its relationship to bias and transparency, has been largely ... As the role of algorithmic systems and processes increases in society, so does the risk of bias, which can result in discrimination against individuals and social groups. ... This was used as a benchmark of gender bias in the search engine results (presented to US-based users). ...

doi:10.5281/zenodo.5783298 fatcat:mjvzjbt5czboracqgfnqvfowx4

Open Access

In addition, I propose several algorithms each approaching the popularity bias mitigation from a different angle and compare their performances using several metrics with some other state-of-the-art approaches ... The effectiveness of these approaches, however, has not been assessed in multi-stakeholder environments. ... The algorithm, re-ranks an initial list of search results so it contains a fair proportion of web pages related to each of these two different aspects. ...

arXiv:2008.08551v1 fatcat:yiuamp6lcnc2bmhzkklakq6ui4

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics [article]

Preserved Fulltext

Understanding How Algorithmic and Cognitive Biases in Web Search Affect User Attitudes on Debated Topics

Preserved Fulltext

Operationalizing Framing to Support Multiperspective Recommendations of Opinion Pieces

Preserved Fulltext

An empirical assessment of quality metrics for diversified similarity searching

Preserved Fulltext

Evaluating diversified search results using per-intent graded relevance

Preserved Fulltext

Comprehensive Viewpoint Representations for a Deeper Understanding of User Interactions With Debated Topics

Preserved Fulltext

Developers' importance from the leader perspective

Preserved Fulltext

Evaluating epistemic uncertainty under incomplete assessments

Preserved Fulltext

A Survey of Research on Fair Recommender Systems [article]

Preserved Fulltext

Other Versions

Bias and unfairness in machine learning models: a systematic literature review [article]

Preserved Fulltext

Other Versions

Implementation of a Framework for Website Quality Evaluation: Himachal Pradesh University Website

Preserved Fulltext

What are you optimizing for? Aligning Recommender Systems with Human Values [article]

Preserved Fulltext

Biobjective evolutionary and heuristic algorithms for intersection of geometric graphs

Preserved Fulltext

Towards Algorithmic Transparency: A Diversity Perspective

Preserved Fulltext

Popularity Bias in Recommendation: A Multi-stakeholder Perspective [article]

Preserved Fulltext