Comparison of Full Factorial and Optimal Experimental Design for Perceptual Evaluation of Audiovisual Quality
×
Cite This
Citation & Abstract
RA. FE. Frans, N. Zacharov, and S. Forchhammer, "Comparison of Full Factorial and Optimal Experimental Design for Perceptual Evaluation of Audiovisual Quality," J. Audio Eng. Soc., vol. 71, no. 1/2, pp. 4-19, (2023 January.). doi: https://doi.org/10.17743/jaes.2022.0063
RA. FE. Frans, N. Zacharov, and S. Forchhammer, "Comparison of Full Factorial and Optimal Experimental Design for Perceptual Evaluation of Audiovisual Quality," J. Audio Eng. Soc., vol. 71 Issue 1/2 pp. 4-19, (2023 January.). doi: https://doi.org/10.17743/jaes.2022.0063
Abstract: Perceptual evaluation of immersive audiovisual quality is often very labor-intensive and costly because numerous factors and factor levels are included in the experimental design. Therefore, the present study aims to reduce the required experimental effort by investigating the effectiveness of optimal experimental design (OED) compared to classical full factorial design (FFD) in the study using compressed omnidirectional video and ambisonic audio as examples. An FFD experiment was conducted and the results were used to simulate 12 OEDs consisting of D-optimal and I-optimal designs varying with replication and additional data points. The fraction of design space plot and the effect test based on the ordinary least-squares model were evaluated, and four OEDs were selected for a series of laboratory experiments. After demonstrating an insignificant difference between the simulation and experimental data, this study also showed that the differences in model performance between the experimental OEDs and FFD were insignificant, except for some interacting factors in the effect test. Finally, the performance of the I-optimal design with replicated points was shown to outperform that of the other designs. The results presented in this study open new possibilities for assessing perceptual quality in a much more efficient way.
@article{frans2023comparison,
author={frans, randy fela and zacharov, nick and forchhammer, søren},
journal={journal of the audio engineering society},
title={comparison of full factorial and optimal experimental design for perceptual evaluation of audiovisual quality},
year={2023},
volume={71},
number={1/2},
pages={4-19},
doi={https://doi.org/10.17743/jaes.2022.0063},
month={january},}
@article{frans2023comparison,
author={frans, randy fela and zacharov, nick and forchhammer, søren},
journal={journal of the audio engineering society},
title={comparison of full factorial and optimal experimental design for perceptual evaluation of audiovisual quality},
year={2023},
volume={71},
number={1/2},
pages={4-19},
doi={https://doi.org/10.17743/jaes.2022.0063},
month={january},
abstract={perceptual evaluation of immersive audiovisual quality is often very labor-intensive and costly because numerous factors and factor levels are included in the experimental design. therefore, the present study aims to reduce the required experimental effort by investigating the effectiveness of optimal experimental design (oed) compared to classical full factorial design (ffd) in the study using compressed omnidirectional video and ambisonic audio as examples. an ffd experiment was conducted and the results were used to simulate 12 oeds consisting of d-optimal and i-optimal designs varying with replication and additional data points. the fraction of design space plot and the effect test based on the ordinary least-squares model were evaluated, and four oeds were selected for a series of laboratory experiments. after demonstrating an insignificant difference between the simulation and experimental data, this study also showed that the differences in model performance between the experimental oeds and ffd were insignificant, except for some interacting factors in the effect test. finally, the performance of the i-optimal design with replicated points was shown to outperform that of the other designs. the results presented in this study open new possibilities for assessing perceptual quality in a much more efficient way.},}
TY - paper
TI - Comparison of Full Factorial and Optimal Experimental Design for Perceptual Evaluation of Audiovisual Quality
SP - 4
EP - 19
AU - Frans, Randy Fela
AU - Zacharov, Nick
AU - Forchhammer, Søren
PY - 2023
JO - Journal of the Audio Engineering Society
IS - 1/2
VO - 71
VL - 71
Y1 - January 2023
TY - paper
TI - Comparison of Full Factorial and Optimal Experimental Design for Perceptual Evaluation of Audiovisual Quality
SP - 4
EP - 19
AU - Frans, Randy Fela
AU - Zacharov, Nick
AU - Forchhammer, Søren
PY - 2023
JO - Journal of the Audio Engineering Society
IS - 1/2
VO - 71
VL - 71
Y1 - January 2023
AB - Perceptual evaluation of immersive audiovisual quality is often very labor-intensive and costly because numerous factors and factor levels are included in the experimental design. Therefore, the present study aims to reduce the required experimental effort by investigating the effectiveness of optimal experimental design (OED) compared to classical full factorial design (FFD) in the study using compressed omnidirectional video and ambisonic audio as examples. An FFD experiment was conducted and the results were used to simulate 12 OEDs consisting of D-optimal and I-optimal designs varying with replication and additional data points. The fraction of design space plot and the effect test based on the ordinary least-squares model were evaluated, and four OEDs were selected for a series of laboratory experiments. After demonstrating an insignificant difference between the simulation and experimental data, this study also showed that the differences in model performance between the experimental OEDs and FFD were insignificant, except for some interacting factors in the effect test. Finally, the performance of the I-optimal design with replicated points was shown to outperform that of the other designs. The results presented in this study open new possibilities for assessing perceptual quality in a much more efficient way.
Perceptual evaluation of immersive audiovisual quality is often very labor-intensive and costly because numerous factors and factor levels are included in the experimental design. Therefore, the present study aims to reduce the required experimental effort by investigating the effectiveness of optimal experimental design (OED) compared to classical full factorial design (FFD) in the study using compressed omnidirectional video and ambisonic audio as examples. An FFD experiment was conducted and the results were used to simulate 12 OEDs consisting of D-optimal and I-optimal designs varying with replication and additional data points. The fraction of design space plot and the effect test based on the ordinary least-squares model were evaluated, and four OEDs were selected for a series of laboratory experiments. After demonstrating an insignificant difference between the simulation and experimental data, this study also showed that the differences in model performance between the experimental OEDs and FFD were insignificant, except for some interacting factors in the effect test. Finally, the performance of the I-optimal design with replicated points was shown to outperform that of the other designs. The results presented in this study open new possibilities for assessing perceptual quality in a much more efficient way.
Open Access
Authors:
Frans, Randy Fela; Zacharov, Nick; Forchhammer, Søren
Affiliations:
SenseLab, FORCE Technology, Hørsholm, Denmark; Department of Electrical and Photonics Engineering, Technical University of Denmark, Kgs. Lyngby, Denmark; Meta Reality Labs., Paris, France; Department of Electrical and Photonics Engineering, Technical University of Denmark, Kgs. Lyngby, Denmark(See document for exact affiliation information.) JAES Volume 71 Issue 1/2 pp. 4-19; January 2023
Publication Date:
January 16, 2023Import into BibTeX
Permalink:
http://www.aes.org/e-lib/browse.cfm?elib=22027