In this paper we study the matter of perceptual evaluation data collection for the purposes of machine learning. Well established listening test methods have been developed and standardised in the audio community over many years. This papers looks at the specific needs for machine learning and seeks to establish efficient data collection methods, that address the requirements of machine learning, whilst also providing robust and repeatable perceptual evaluation results. Following a short review of efficient data collection techniques, including the concept of data augmentation and introduce the new concept of pre-augmentation as an alternative efficient data collection approach. Multiple stimulus presentation style listening tests are then presented for the evaluation of a wide range of audio quality devices (headphones) evaluated by a panel of trained expert assessors. Two tests are presented using a traditional full factorial design and a pre-augmented design to enable the performance comparison of these two approaches. The two approaches are statistically analysed and discussed. Finally, the performance of the two approaches for building machine learning models are reviewed, comparing the performance of a range of baseline models.

Authors: Volk, Christer P.; Nordby, Jon; Stegenborg-Andersen, Tore; Zacharov, Nick
Affiliations: FORCE Technology, SenseLab, Hørsholm, Denmark; Soundsensing, Oslo, Norway(See document for exact affiliation information.)
AES Convention: 150 (May 2021) Paper Number: 10488
Publication Date: May 24, 2021 Import into BibTeX
Subject: Audio Quality/Standards
Permalink: https://www.aes.org/e-lib/browse.cfm?elib=21081

Download Now (795 KB)

This paper is Open Access which means you can download it for free.

Learn more about the AES E-Library

E-Library Location:   /conv/150/10488.pdf

Start a discussion about this paper!