Selection of stimuli for listening tests by software-assisted ranking

Sammanfattning: Within the academic field of audio technology, which is the context of this thesis, research questions related to various forms of perceptual audio evaluation of different types of audio systems are important and commonly encountered. Methods for systematic evaluations of listeners’ perceived audio quality, based on different experimental designs, have evolved over a time span of several decades and are collectively referred to as listening tests. Typically, some 20 to 30 subjects take part as listeners in such studies, where the task for the subjects normally involves listening to a number of different audio stimuli that have been processed by the audio systems under study. The assessment of the perceived audio quality can be made in several ways, and can for example aim for a ranking of the systems under study or ask subjects to grade them on one or several scales.The selection of which audio stimuli to include in a listening test is of great importance and can greatly effect the results. This is well known from the literature, but it is also known that such selection normally is both time-consuming and labour-intensive and can be difficult to conduct in a systematic way. However, publications on systematic selection procedures are rare. International listening test recommendations call for the use of so called critical audio stimuli, with only a very brief further explication that critical stimuli are such stimuli that stress the systems under test to reveal differences among them.In this thesis, custom made software was developed to enable experimental listening studies to find out if a group of listeners could discern different audio stimuli using the notion critical. The objective of the experiments was to investigate if a group of listeners that were asked to rank a set of audio stimuli according to their criticality would produce any significant differences between the median rankings of the different audio stimuli. The specific ranking method used was ranking-by-elimination. Two experiments were performed – one using a randomized complete block design (RCBD), and one using a balanced incomplete block design (BIBD). The results indicated a valid and meaningful use of the notion critical in listening test contexts, and that stimuli could be rank ordered according to their degree of being more or less critical. As a basis for a software-assisted method of selecting critical audio stimuli for listening tests, such a resulting rank order constitutes an important contribution to a systematic strategy for the selection process.The developed software is planned to be made public in the future, to encourage and promote the use of open-source software in audio research as a basis for better control of software audio quality in demanding experimental contexts such as listening tests.