5,991 Hits in 4.7 sec

Performance Metric Elicitation from Pairwise Classifier Comparisons [article]

Gaurush Hiranandani, Shant Boodaghians, Ruta Mehta, Oluwasanmi Koyejo
2019 arXiv   pre-print
In particular, we focus on eliciting binary classification performance metrics from pairwise feedback, where a practitioner is queried to provide relative preference between two classifiers.  ...  Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation.  ...  By construction, pairwise classifier comparisons may be conceptually represented by their associated pairwise confusion matrix comparisons.  ... 
arXiv:1806.01827v2 fatcat:d2jguj5qhndrjkmjinge2vnnuq

Quadratic Metric Elicitation for Fairness and Beyond [article]

Gaurush Hiranandani, Jatin Mathur, Harikrishna Narasimhan, Oluwasanmi Koyejo
2021 arXiv   pre-print
Metric elicitation is a recent framework for eliciting performance metrics that best reflect implicit user preferences based on the application and context.  ...  We further extend this strategy to eliciting polynomial metrics -- thus broadening the use cases for metric elicitation.  ...  Using such pairwise comparison queries, ME aims to recover the oracle's metric. Figure 1 (reproduced from Hiranandani et al. [18] ) depicts the ME framework.  ... 
arXiv:2011.01516v2 fatcat:ew2rk6u4jzbhzcwxr6sj2677ma

Fair Performance Metric Elicitation [article]

Gaurush Hiranandani, Harikrishna Narasimhan, Oluwasanmi Koyejo
2020 arXiv   pre-print
The use of metric elicitation enables a practitioner to tune the performance and fairness metrics to the task, context, and population at hand.  ...  We consider the choice of fairness metrics through the lens of metric elicitation -- a principled framework for selecting performance metrics that best reflect implicit preferences.  ...  To this end, we extend the ME framework from eliciting multiclass classification metrics [21] to the task of eliciting fair performance metrics from pairwise preference feedback in the presence of multiple  ... 
arXiv:2006.12732v3 fatcat:74dfxqqdn5butnw3nbxx3iyzca

Operationalizing Individual Fairness with Pairwise Fair Representations [article]

Preethi Lahoti, Krishna P. Gummadi, Gerhard Weikum
2019 arXiv   pre-print
A central challenge in operationalizing their approach is the difficulty in eliciting a human specification of a similarity metric.  ...  We elicit fairness judgments from a variety of sources, including human judgments for two real-world datasets on recidivism prediction (COMPAS) and violent neighborhood prediction (Crime & Communities)  ...  This ability crucially distinguishes our problem from the classical metric labeling problem [27] , where the goal is to classify a given set of individuals with pairwise relationships (i.e., costs for  ... 
arXiv:1907.01439v1 fatcat:lfkxpjpeynetbpjxpuptbkxs4q

An Ontology Selection and Ranking System Based on the Analytic Hierarchy Process

Adrian Groza, Irina Dragoste, Iulia Sincai, Ioana Jimborean, Vasile Moraru
2014 2014 16th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing  
The system applies different metrics on ontologies in order to feed the Analytic Hierarchy Process with facts.  ...  Selecting the desired ontology from a collection of available ones is essential for ontology reuse.  ...  The user preferences are modeled using a pairwise comparison matrix [19] . Definition 1.  ... 
doi:10.1109/synasc.2014.47 dblp:conf/synasc/GrozaDSJM14 fatcat:qxcldxstkzbmre2ynnnce2hcmq

Using the Case-Based Ranking Methodology for Test Case Prioritization

Paolo Tonella, Paolo Avesani, Angelo Susi
2006 Conference on Software Maintenance, Proceedings of the  
CBR elicits just relative priority information from the user, in the form of pairwise test case comparisons.  ...  CBR learns the target ranking from two inputs: a set of possibly partial indicators of priority and pairwise comparisons elicited from the user (cases).  ...  Setting Since the number of pairwise comparisons that can be reasonably elicited from the user is limited, we expect that the performance of the proposed technique depends critically on the test suite  ... 
doi:10.1109/icsm.2006.74 dblp:conf/icsm/TonellaAS06 fatcat:dxkmc3vfqzeonm4zzcjeyndxaa

The usefulness of metrics in the quantification of speech rhythm

Amalia Arvaniti
2012 Journal of Phonetics  
Eight participants per language produced speech using three elicitation methods, spontaneous speech, story reading and reading a set of sentences divided into "uncontrolled" sentences from original works  ...  decisions, making crosslinguistic comparisons and rhythmic classifications based on metrics unsafe at best.  ...  For the other metrics, indicative results from the pairwise comparisons are given.  ... 
doi:10.1016/j.wocn.2012.02.003 fatcat:gkhr5tafb5f35apm2cq67nwg7u

Retrieving Categorical Emotions Using a Probabilistic Framework to Define Preference Learning Samples

Reza Lotfian, Carlos Busso
2016 Interspeech 2016  
We evaluate the approach over cross-corpus experiments, showing improved performance over binary classifiers and rank-based classifiers trained with consensus labels.  ...  Instead of predicting the underlying emotional class of a sample, this framework relies on pairwise comparisons to rank-order the testing data according to an emotional dimension.  ...  Direct annotation of these relative labels is expensive and time demanding as the number of pairwise comparison is N (N − 1)/2 for N samples.  ... 
doi:10.21437/interspeech.2016-1052 dblp:conf/interspeech/LotfianB16 fatcat:3iocz6bwvfcy7bjdl5cji7qjvm

A Comparison of Classification Techniques to Predict Brain-Computer Interfaces Accuracy Using Classifier-Based Latency Estimation

Md Rakibul Mowla, Jesus D. Gonzalez-Morales, Jacob Rico-Martinez, Daniel A. Ulichnie, David E. Thompson
2020 Brain Sciences  
To investigate the role of latency jitter on BCI system performance, we proposed the classifier-based latency estimation (CBLE) method.  ...  Our results showed that on the whole, CBLE worked regardless of the classification method and electrode count; by contrast the effect of the number of electrodes on BCI performance was classifier dependent  ...  Table 2 shows the p-values of pairwise multiple comparisons using the Wilcoxon signed-ranks test.  ... 
doi:10.3390/brainsci10100734 pmid:33066374 fatcat:4poxye7abndyjbltl7eli7hjum

Recording from the same neurons chronically in motor cortex

George W. Fraser, Andrew B. Schwartz
2012 Journal of Neurophysiology  
Although these metrics are individually weak, together they produce a strong classifier. Some neurons were recorded for Ͼ100 days.  ...  Pairwise cross-correlograms, the autocorrelogram, waveform shape, and mean firing rate were used together as identifying features of a neuron.  ...  Single-metric performance is on the diagonal. 2-Metric performance is indicated by the combination of row and column labels.  ... 
doi:10.1152/jn.01012.2010 pmid:22190623 pmcid:PMC3331662 fatcat:zyvuxhfrz5fzrhwjwrsh2l64kq

Arabic speaker emotion classification using rhythm metrics and neural networks

Ali Mefiah, Yousef A. Alotaibi, Sid-Ahmed Selouani
2015 2015 23rd European Signal Processing Conference (EUSIPCO)  
In this paper, rhythm metrics are calculated and used to classify five Arabic speech emotions; namely, neutral, sad, happy, surprised, and angry.  ...  The results of a neural network-based automatic emotion recognition system using rhythm metrics were similar to the human perception test results, although less accurate.  ...  This is because rhythm metric computation depends heavily on various comparisons between consonantal and vocalic intervals in the given speech utterance.  ... 
doi:10.1109/eusipco.2015.7362619 dblp:conf/eusipco/MefiahAS15 fatcat:eqmx4wbburd6pgt525itu4z7ia

Partial Multi-Label Learning via Credible Label Elicitation

Jun-Peng Fang, Min-Ling Zhang
Extensive experiments on synthetic as well as real-world data sets clearly validate the effectiveness of credible label elicitation in learning from PML examples.  ...  In light of this major difficulty, a novel two-stage PML approach is proposed which works by eliciting credible labels from the candidate label set for model induction.  ...  The basic idea of pairwise label ranking is to transform the original learning problem into a number of pairwise comparison problems, one for each label pair (y u , y z ) (1 ≤ u < z ≤ q).  ... 
doi:10.1609/aaai.v33i01.33013518 fatcat:edp3usvzmbagfl6te2b2pgkzuq

Using an SMT solver for interactive requirements prioritization

Francis Palma, Angelo Susi, Paolo Tonella
2011 Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering - SIGSOFT/FSE '11  
An interactive approach has been proposed to the problem of prioritization based on Satisfiability Modulo Theory (SMT) techniques and pairwise comparisons.  ...  The approach has been evaluated on a set of requirements from a real health-care project.  ...  pairwise comparisons are to be elicited next.  ... 
doi:10.1145/2025113.2025124 dblp:conf/sigsoft/PalmaST11 fatcat:ymyk74ud45h45cisd5umlozo5a

Linear Discriminant Analysis Achieves High Classification Accuracy for the BOLD fMRI Response to Naturalistic Movie Stimuli

Hendrik Mandelkow, Jacco A. de Zwart, Jeff H. Duyn
2016 Frontiers in Human Neuroscience  
All classifiers performed best when given input features from a large region of interest comprising around 25% of the voxels that responded significantly to the visual stimulus.  ...  In any case, the number FIGURE 2 | Comparison of classifier performance.  ...  By comparison, the nearest-mean classifiers (NMC, NME, GNB, Table 1 ) based on averaged class templates performed considerably worse, close to 60%, irrespective of the distance metric.  ... 
doi:10.3389/fnhum.2016.00128 pmid:27065832 pmcid:PMC4815557 fatcat:hbuyeuwv2zc2pjv6o72ycx3g7q

Perception of speech rhythm in second language: the case of rhythmically similar L1 and L2

Mikhail Ordin, Leona Polyanskaya
2015 Frontiers in Psychology  
However, when native speakers have to classify the timing patterns characteristic of L2 English of German learners at different proficiency levels, they attend to speech rate cues and ignore the differences  ...  We are also thankful to Ferenc Bunta and David Ingram for sharing the picture prompts that we used for the sentence elicitation task.  ...  0.692 C < 0.0005 0.154 %V 0.691 0.068 TABLE 4 | 4 Significance for comparisons of rhythm metrics between proficiency levels pairwise (with Hochberg's correction).  ... 
doi:10.3389/fpsyg.2015.00316 pmid:25859228 pmcid:PMC4373255 fatcat:2uibrpmb2nawnblini5gonrzpm
« Previous Showing results 1 — 15 out of 5,991 results