Many recent publications in audio research present subjective evaluations of audio quality based on the Recommendation ITU-R BS.1534-1 (MUSHRA, MUltiple Stimuli with Hidden Reference and Anchor). This is a very welcome trend because it enables researchers to assess the implications of their developments. The evaluation of listening tests, however, sometimes sufers from an incomplete understanding of the underlying statistics. The present paper aims at identifying the causes for the pitfalls and misconceptions in MUSHRA evaluations. It exemplifes the impact of falsely used or even misused statistics. Subsequently, schemes for evaluating the listeners' judgments that are well-grounded on statistical considerations comprising an understanding of the concepts of statistical power and efect size are proposed.

Authors: Nagel, Frederik; Sporer, Thomas; Sedlmeier, Peter
Affiliations: Fraunhofer Institute for Digital Media Technology IDMT, Ilmenau, Germany; Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany; Technical University of Dresden, Dresden, Grmany(See document for exact affiliation information.)
AES Convention: 128 (May 2010) Paper Number: 8146
Publication Date: May 1, 2010 Import into BibTeX
Subject: Listening Tests and Evaluation Psychoacoustics
Permalink: https://www.aes.org/e-lib/browse.cfm?elib=15442

Click to purchase paper as a non-member or login as an AES member. If your company or school subscribes to the E-Library then switch to the institutional version. If you are not an AES member and would like to subscribe to the E-Library then Join the AES!

This paper costs $33 for non-members and is free for AES members and E-Library subscribers.

Learn more about the AES E-Library

E-Library Location: (CD 128Papers)   /128/8146.pdf

Start a discussion about this paper!