AES New York 2019
Game Audio & XR Track Event Details
Wednesday, October 16, 9:00 am — 10:30 am (1E06)
Game Audio & XR: GA01 - 4-Pi Reverb Effects for In-Game Sounds
Presenters:Tomoya Kishi, CAPCOM Co., Ltd. - Japan
Steve Martz, THX Ltd. - San Rafael, CA, USA
Masataka Nakahara, ONFUTURE Ltd. - Tokyo, Japan; SONA Corp. - Tokyo, Japan
Kazutaka Someya, beBlue Co., Ltd. - Tokyo, Japan
In a video game, sound fields of virtual spaces are created in a 4-pi field which is free from channel restrictions. That is, in a video game, there are no picture frames, which cut out a part of sound fields, nor channel borders, which divide sound fields into finite numbers of areas. The workshop introduces how to create 4-pi channel-free reverberations for in-game sounds both from current and future technical points of the views. Demonstrations will be also provided, and reverberations that are generated by proposed methods will be listened to be compared with conventional reverb sounds that are created by a skilled mixing engineer.
 | This session is presented in association with the AES Technical Committee on Audio for Games and AES Technical Committee on Spatial Audio |
Wednesday, October 16, 9:15 am — 10:45 am (1E08)
Game Audio & XR: GA02 - Abbey Road Spatial Audio Forum—Music Production in VR and AR
Chair:Gavin Kearney, University of York - York, UK
Panelists:
Stephen Barton, Respawn Entertainment/EA - Los Angeles, CA, USA; Afterlight Inc. - Los Angeles, CA, USA
Etienne Corteel, L-Acoustics - Marcoussis, France
Oliver Kadel, 1.618 Digital - London, UK; University Of West London - London, UK
Muki Kulhan, Muki International - UK
Hyunkook Lee, University of Huddersfield - Huddersfield, UK
Mirek Stiles, Abbey Road Studios - London, UK
Virtual and augmented reality offers a new platform for the creation, production, and consumption of immersive music experiences. Immersive technologies now have the power to create experiences that transform how we experience music, from transporting the listener to the original recording studio in VR or even bringing the musicians to their own living room in an AR scenario. However, the creation of such audio experiences has many challenges. With different parts of the immersive audio production chain being developed by various third parties, there is a danger of confusing the producer/musician and perhaps scaring off talent before we even get off the ground. Can we do better? What are the barriers and how can they be broken down? What are the strengths and weaknesses of existing tools? Can we achieve better clarity in the different formats that are available, and should we move towards standardization? In this open panel discussion of the Abbey Road Spatial Audio Forum, we will be looking at workflow challenges for recording, mixing, and distributing music for VR and AR.
| This session is presented in association with the AES Technical Committee on Audio for Games |
Wednesday, October 16, 10:45 am — 12:00 pm (1E06)
Immersive & Spatial Audio: IS02 - Music Production in Immersive Formats: Alternative Perspectives
Presenters:Thomas Aichinger, scopeaudio - Austria
Zachary Bresler, University of Agder - Kristiansand S, Vest-Agder, Norway
Sally Kellaway, Microsoft - Seattle, WA, USA
Jo Lord, University of West London - London, UK
Evidenced by the large volume of presentations on immersive audio at the previous AES conventions in Dublin and New York and the AES conferences on Immersive and Interactive Audio, Virtual and Augmented Reality, and Spatial Reproduction, 3D audio is of growing interest within our field. As Anastasia Devana of Magic Leap stated in her keynote at the IIA conference, it is the “bleeding edge” of our industry. In spatial audio, there are many competing ideas and technologies from delivery formats to production standards and aesthetics. From the perspective of music creators, the norms of music production and the terms used to describe practice are often clumsy or not helpful when applied in 3D. It is in this context that we propose this workshop on immersive music production. We will discuss several questions from the perspective of creatives in immersive and interactive music content. What are the changing ways that creators use and exploit 3D technologies? How do we describe the way that we make content for such systems? What are the practices, standards, and formats that creators use, and which ones should they use in the future? What are the interesting use-cases for 3D audio that challenge the way we think about music and audio production?
Zachary Bresler, Ph.D. fellow, University of Agder. Research explores music production in immersive formats and the staging of listeners in immersive compositional design.
Jo Lord, Ph.D. student, University of West London. Research investigates the development, practical application, and aesthetic suitability of 3D mix technique for record production.
Dr. Eve Klein, senior lecturer in music technology and popular music, University of Queensland. Currently researches within the VR/AR space, creating large-scale immersive festival experiences.
Thomas Aichinger, founder, scopeaudio. Studio specialized in sound design and post-production, focusing on spatial audio productions for VR and 360° videos.
Wednesday, October 16, 10:45 am — 11:45 am (1E13)
Game Audio & XR: GA03 - Interactive AR Audio Using Spark
Moderator:Dren McDonald, Facebook
Panelists:
Kristi Boeckmann, Facebook - Menlo Park, CA, USA
Matt Nichols, Facebook
Grey Robbins, Facebook
In the past year, over a billion people have used SparkAR experiences (https://developers.facebook.com/videos/f8-2019/day-1-keynote/ at 1 hour 24min in). It’s a remarkable number. In apps such as Messenger, Messenger Kids, Instagram, the Facebook app, and in devices such as Portal, AR is becoming mainstream. All of those experiences were generated with Spark AR, a free tool that anyone can use to develop AR effects. The Facebook AR Audio team would like to demonstrate some of our AR effects and discuss the challenges we faced in bringing our ideas to fruition. Topics within the AR audio design discussion will include dynamic music, ambience, audio triggers based on face and hand gestures, asset variation/randomization, microphone DSP capabilities, target plane tracking triggers, importing beat maps/beat detection, and javascripting. We can demonstrate the Spark audio workflow using the patch editor and also demonstrate effects that were designed using javascript. Several AR games have been developed in Spark and we’d like to show how those were created, in addition to showing off the interactive AR stories on Portal called Story Time. Spark AR is free and available on both Windows and MacOS. While there is some documentation about audio within Spark AR, we feel that by showing effects that we have built and prototypes that we are developing, we can show what is possible within Spark AR. Hopefully we can inspire others in the interactive audio community to dive into AR audio to see what they can create. This short talk from F8 highlights topics that we will go into a deep dive on with this talk (https://developers.facebook.com/videos/2019/ar-music/)
| This session is presented in association with the AES Technical Committee on Audio for Games |
Wednesday, October 16, 1:30 pm — 2:30 pm (1E13)
Game Audio & XR: GA04 - Real-time Mixing and Monitoring Best Practices for Virtual, Mixed, and Augmented Reality
Moderator:Scott Selfon, Audio Experiences Lead, Facebook Reality Labs (Oculus Research) - Redmond, WA, USA
Presenting sound to a person experiencing a dynamic virtual reality experience is, by definition, a just-in-time activity. How do we take advantage of more than a century of mixing and monitoring practices based on linear content—and more than 20 years of interactive game mixing—to create a coherent, believable, and emotionally satisfying soundscape for these new realities? This talk discusses the current state of the art for mixing and monitoring techniques, from the actual process, to ever-evolving standards, to robustly handling the variety of authored and implemented content, playback environments, and scenarios.
| This session is presented in association with the AES Technical Committee on Audio for Games |
Wednesday, October 16, 2:45 pm — 4:15 pm (1E08)
Game Audio & XR: GA05 - Spatial Storytelling in Games
Moderator:Rob Bridgett, Eidos Montreal - Montreal, Canada
Panelists:
Cedric Diaz, Senior Sound Designer, People Can Fly - New York, NY, USA
Jason Kanter, Audio Director, Avalanche Studios - New York, NY, USA
Phillip Kovats, WWS Sound, Sony Interactive Entertainment
Mark Petty, Gearbox Software
In this exciting panel discussion join several industry experts, all with deep experience in authoring narrative content for spatial audio entertainment platforms, in discussing some of the incredible opportunities and challenges of bringing stories to life using spatial elements. Our goal is to discuss the techniques, thinking, approaches, and execution of how 3D spaces interface with and infiltrate our storytelling practices. As audio directors, sound designers, mixers and storytellers, we will focus on how we are able to leverage spatial audio to bring a greater level of engagement, spectacle, and immersion for audiences inside our story worlds.
Wednesday, October 16, 4:30 pm — 5:30 pm (1E08)
Game Audio & XR: GA06 - Simulating Real World Acoustic Phenomena: From Graphics to Audio
Presenter:Christophe Tornieri, Audiokinetic
Simulating real world acoustic phenomena in virtual environments drastically enhances immersion. However, computing high order reflections and convincing diffraction towards more dynamic and realistic audio scenes is a challenging task to achieve in real-time. In this talk we propose an approach derived from modern 3D graphic rendering techniques used in CGI. We will first introduce the general concepts behind ray tracing and stochastic methods, then present the adaptation of these techniques to spatial audio, focusing on how to compute reflections and diffraction while maintaining time and spatial coherence.
| This session is presented in association with the AES Technical Committee on Audio for Games and AES Technical Committee on Spatial Audio |
Thursday, October 17, 9:00 am — 10:30 am (1E21)
Game Audio & XR: GA07 - MPEG-H 3D Audio Goes VR
Chair:Jürgen Herre, International Audio Laboratories Erlangen - Erlangen, Germany; Fraunhofer IIS - Erlangen, Germany
Panelists:
Adrian Murtaza, Fraunhofer Institute for Integrated Circuits IIS - Erlangen, Germany
Nils Peters, Qualcomm, Advanced Tech R&D - San Diego, CA, USA
The MPEG-H 3D Audio is a recent MPEG standard that was designed to represent and render 3D audio experiences while supporting all known production paradigms (channel-based, object-based, and Higher Order Ambisonics based audio) and reproduction setups (loudspeaker, headphone/binaural). As the audio production world moves forward to embrace Virtual and Augmented Reality (VR/AR), MPEG-H found considerable adoption and re-use in recently finalized VR standards, such as MPEG-I OMAF(Omnidirectional Media Format), VR Industry Forum (VR-IF) Guidelines as well as 3GPP "VRStream" (Virtual Reality profiles for streaming applications) where it was selected as the audio standard for VR content delivered over 5G networks.
| This session is presented in association with the AES Technical Committee on Coding of Audio Signals |
Thursday, October 17, 10:15 am — 11:45 am (1E08)
Game Audio & XR: GA08 - Introduction to Machine Learning for Game Audio
Presenters:Krystle Becknauld
John Byrd, Gigantic Software - Santa Ana, CA, USA
In the past four years, every tech industry has put machine learning systems into production. Virtual assistants, cybersecurity systems, and self-driving cars are all built on machine learning. However, none of these novel algorithms are being used in the interactive entertainment industries.
This talk summarizes the most important new research in this exciting new field and describes novel applications of machine learning, specifically for interactive audio. You'll see demonstrations and code for: automatic feature extraction, classification, silence removal, tempo extraction, automatic speaker detection, emotion detection, convolutional and recurrent neural networks, variational autoencoders, and WaveNet. Additionally, this talk will show you the state of the art in interactive audio, and it will also tell you what will be possible, until the year 2030.
| This session is presented in association with the AES Technical Committee on Audio for Games |
Thursday, October 17, 1:15 pm — 2:15 pm (1E13)
Game Audio & XR: GA09 - Audio Productivity in Mixed Reality
Presenters:Sally Kellaway, Microsoft - Seattle, WA, USA
Joe Kelly, Microsoft
Hearing is a human sense that subconsciously enables humans to detect and categorize stimulus to make decisions and take actions. Virtual, Augmented and Mixed Reality has allowed audio designers and developers in general to explore what framework of audio is needed to have mostly entertainment focused immersive experiences. This has enabled users to have an immense capacity for understanding and empathy with a myriad of experiences. In enterprise scenarios there are a vast array of industries that deploy immersive technologies to expedite workflows, however, enterprise technology has historically focused on productivity at the expense of experience and has never before needed to factor in the complexity of the environment of the experience to this extent. The audio team developing Microsoft Dynamics 365 Mixed Reality applications has spent 2+ years refining a framework for audio productivity in MR, and will discuss a number of approaches to designing and implementing audio features for enterprise and consumer applications where productivity of the user is the goal.
| This session is presented in association with the AES Technical Committee on Audio for Games |
Thursday, October 17, 2:15 pm — 3:15 pm (1E13)
Game Audio & XR: GA10 - Lessons Learned in Game Audio
Presenters:Alex Wilmer, Wilmer Sound - San Francisco, CA, USA
Sae Wilmer, Co-Founder & COO, Wilmer Sound
Alex Wilmer will present his guiding principles, learned during his career in Film, Console, Mobile, VR, and AR audio. He will share anecdotes to illustrate why these principles are crucial for success in the future of game audio. He will discuss the successes and the failures in the hopes that people can learn how to succeed and not make the same mistakes that he did!
Thursday, October 17, 3:30 pm — 4:30 pm (1E13)
Game Audio & XR: GA11 - Canceled
| This session is presented in association with the AES Technical Committee on Audio for Games |
Thursday, October 17, 4:00 pm — 5:30 pm (1E21)
Recording & Production: RP11 - Pitch Shift FXpertise – Record and Mix Strategies for the Broad Range Pitch Effect Possibilities
Presenter:Alex Case, University of Massachusetts Lowell - Lowell, MA, USA
Pitch shift as an effect has been a part of sound recording from the very beginning of audio and continues to evolve briskly today. Recording at one speed but playing back at another has been the pitch shift modus operandi across all analog formats—cylinders, disks, and tape. Digital audio continued these time-domain techniques, exploiting sample rate differences between record and playback. The digital luxury of frequency-domain pitch shift, offering new effects possibilities, became an important part of the pop engineer’s tool kit in the late 90s.
From this are born a seemingly endless set of pitch-based outcomes. But what are we to make of the aesthetic possibilities? How do we organize its creative potential? Alex U. Case auditions a curated set iconic examples of pitch shifting effects in pop music and analyses them through multiple lenses—music, signal processing and psychoacoustics—to define the full range of the effect, giving it structure, and empowering you to develop your own strategies for the use of Pitch FX in your next project.
Thursday, October 17, 4:30 pm — 5:30 pm (1E08)
Immersive & Spatial Audio: IS04 - 3D Audio Philosophies & Techniques for Commercial Music
Presenter:Bt Gibbs, Skyline Entertainment and Publishing - Morgan Hill, CA, USA; Tool Shed Studios - Morgan Hill, CA, USA
3D Audio (360 Spatial) for immersive content has made massive strides forward in just the first five months of 2019. However, the majority of content remains in the animated VR world. Commercial audio (in all genres) continues to be delivered across streaming and download platforms in L+R stereo audio. With the binaural (headphone) delivery options for spatial audio as a topic of discussion for many major hi-res audio delivery platforms, commercial music delivery options are coming very soon. The ability for commercial artists to deliver studio quality audio (if not MQA) to consumers with an "in-the-studio" experience soon will be delivered in ambisonic formats.
This presentation will demonstrate studio sessions delivered in 360 video and stereo mixes translated to static (non-HRTF) 360 audio, which was originally captured for standard stereo delivery through traditional streaming and download sites. All of this audio is prepared to be delivered in a simultaneous (and rapid) turn around from pre-production to final masters delivered on both 360 and stereo platforms. To do so, requires planning in even in the earliest of (pre-production) stages prior to actual recording.
Friday, October 18, 9:00 am — 10:00 am (1E08)
Immersive & Spatial Audio: IS05 - Building Listening Tests in VR
Presenters:Gavin Kearney, University of York - York, UK
Tomasz Rudzki, University of York - York, UK
Benjamin Tsui, University of York - York, UK
In this workshop we will demonstrate how to prepare and conduct various listening tests in VR easily with the tools we created. Our open source tool-box consists of Unity objects, VST plugins, and MATLAB based data analysis app. It provides an end-to-end workflow from creating the test to visualizing the results. Researchers can create their own listening tests which can be run on different VR headsets, e.g., Oculus Rift, HTC Vive. We will dive into some of the actual use-cases to show the practicality and robustness of using audio-visual VR presentation for perceptual tests. We would like to encourage researchers to use the toolbox, express feedback, and contribute to the project development.
| This session is presented in association with the AES Technical Committee on Audio for Games |
Friday, October 18, 9:00 am — 10:30 am (1E13)
Game Audio & XR: GA12 - Not Playing Games With Your Business
Chair:Alex Wilmer, Wilmer Sound - San Francisco, CA, USA
Panelists:
Jason Kanter, Audio Director, Avalanche Studios - New York, NY, USA
Adam Levenson, Sr. Dir. of Business Development & Marketing, Waves Audio - San Francisco, CA, USA
Michael Sinterniklaas, President, NYAV Post
The business of game audio is rapidly changing. This panel will include some of the most qualified professionals in the industry to share their views on the opportunities available to artists, producers, directors, and entrepreneurs within game audio. Whether you're a veteran or just starting out, this panel will cover all angles of the business of game audio to help you achieve your goals.
Friday, October 18, 10:30 am — 12:00 pm (1E06)
Game Audio & XR: GA13 - Borderlands 3 - The Wild West of Atmos
Presenters:Brian Fieser, Gearbox Software
Julian Kwasneski, Bay Area Sound
Mark Petty, Gearbox Software
William Storkson, Bay Area Sound
This event will cover: Emitter / object based in game design vs rendered 7.1.4 assets; How do these two approaches differ when it comes to spatialization from the player perspective; Understanding the end user environment and mixing for Atmos virtualization; Atmos for headphones; Linear post / cinematic design for Atmos; Mix perspectives / how aggressive should we be with height information— Atmos is the wild west; Using 7.1.4 discreet as a tool to better understand inconsistencies in game / run time spatial information.
| This session is presented in association with the AES Technical Committee on Audio for Games |
Friday, October 18, 1:30 pm — 2:30 pm (1E17)
Game Audio & XR: GA14 - Just Cause 4 OST: Creative Collaboration
Presenter:Zach Abramson, YouTooCanWoo
Just Cause 4’s massive soundtrack incorporates many different styles, sounds, and strategies crafted by composer Zach Abramson and a team of friends at their small studio in Brooklyn, NY. A collaborative approach was developed between Abramson, his team in Brooklyn and the audio department at Avalanche Studios across the East River in Manhattan, which led to stronger ideas and results that would have been difficult to achieve if working alone.
This presentation will provide an in-depth analysis of ways composers and audio teams can work together to better interpret creative briefs, design pillars, and game mechanics informing a soundtrack’s aesthetic as well as its technical design. From there, the discussion will continue into the various approaches used to achieve these results, ranging from broad topics like creative decision-making down to specific production techniques. These approaches directly relate to how composers can rely on a team to help mitigate the struggles of working in difficult environments like New York City, where workspaces and time often come at a premium.
Attendees will learn different composition techniques and how these various approaches relate to big concepts in video game scores, as well as real-world tips for how to collaborate effectively in a fast-paced creative environment. This talk is intended for composers, game audio professionals, music supervisors and anyone who is interested in learning more about video game music creation.
| This session is presented in association with the AES Technical Committee on Audio for Games |
Friday, October 18, 2:15 pm — 3:15 pm (1E08)
Immersive & Spatial Audio: IS06 - Capturing Reality with the Use of Spatial Sound and High Order Ambisonics – Ethnographic and Six Degrees of Freedom (6DoF) Case Studies
Moderator:Tomasz Zernicki, Zylia sp. z o.o. - Poznan, Poland
Panelists:
Florian Grond, McGill University - Montreal, Canada
Eduardo Patricio, Zylia Sp. z o.o. - Poznan, Poland
Zack Settel, University of Montreal - Montreal, Quebec, Canada
This workshop will present spatial sound works from a practical perspective. Professional audio engineers and musicians will discuss their 360°, 3D, and ambient productions combining sound, image, and written text. The speakers will address the use of spatial audio and ambisonics for creating immersive representations of reality, including six-degrees-of-freedom live recorded sound. The need for new thinking and specific tools will be discussed and demonstrations of examples created via experimental 6DoF end-to-end workflows will be presented. The workshop will focus especially on the usage of spherical microphone arrays that enables the recording of entire 3D sound scenes as well as six degrees of freedom experiences (6DoF VR). Additionally, the workshop will address Ambisonics and the separation of individual sound sources in post-production, which give creators access to unique sonic possibilities.
Saturday, October 19, 9:00 am — 12:00 pm (1E13)
Immersive & Spatial Audio: IS08 - Ambisonics Tools for Immersive Audio Capture and Post-Production
Presenters:Ianina Canalis, National University of Lanús - Buenos Aires, Argentina
Brian Glasscock, Sennheiser
Andres A. Mayo, 360 Music Lab - Buenos Aires, Argentina
Martin Muscatello, 360 Music Lab
Over the past 3 years, immersive audio production tools evolved considerably and allowed producers to combine them in many different ways. In this workshop we will provide an in-depth explanation of how Ambisonics works and how it can be a central piece of an immersive audio production workflow. Attendees will be able to experiment with dedicated hardware and software tools during the entire workshop. Bring your own laptop and headphones for a unique learning session!
Preregistration is required for this event. Tickets are $75 (member) and $125 (non-member) and can be purchased on-line when you register for the convention. Seating is limited. For more information and to register click here.
Saturday, October 19, 9:00 am — 10:30 am (1E08)
Immersive & Spatial Audio: IS09 - Producing High-Quality 360/3D VR Concert Videos with 3D Immersive Audio
Presenters:Ming-Lun Lee, University of Rochester - Rochester, NY, USA
Steve Philbert, University of Rochester - Rochester, NY, USA
Our 3D Audio Research Laboratory at the University of Rochester has recorded over 40 concerts at the Eastman School of Music since Fall 2017. We have used an Orah 4i 4K 360 VR Camera and a Kandao Obsidian R 8K 3D 360 VR Camera to make 360/3D video recordings, as well as two Neumann KU100 Binaural Microphones, a Sennheiser Ambeo Smart Headset, a 32-element mh acoustics em32 Eigenmike microphone array, a Sennheiser Ambeo VR Microphone, a Zoom H3-VR Handy Recorder, a Core Sound TetraMic, and a Core Sound OctoMic to make 3D immersive audio recordings. With Adobe Premiere, we have been able to edit and render high-quality 8K 360/3D concerts videos mixed with binaural recordings for head-locked binaural audio or Ambisonic recordings for head-tracking binaural audio.
This workshop aims to show our optimized workflows for making high-quality VR concert videos from recording, editing, rendering, and finally publishing on YouTube and Facebook. We plan to demonstrate some essential recording and editing techniques with practical examples for the attendants to hear binaural audio with headphones. Making long concert VR videos is much more challenging than making short VR music videos. We have encountered and investigated so many technical issues, including stitching, video/audio drifting, synchronization, and equalization. Therefore, we also want to share our experiences in resolving some critical A/V issues and improving the audio quality. Our session also welcomes the audience to join discussions and share their experiences.
Saturday, October 19, 1:30 pm — 3:00 pm (1E08)
Game Audio & XR: GA15 - Binaural Audio - Just a 360/VR Geek Thing or Future Audio Entertainment?
Chair:Tom Ammermann, New Audio Technology GmbH - Hamburg, Germany
Panelists:
David Miles Huber, Seattle, WA, USA
Andres A. Mayo, 360 Music Lab - Buenos Aires, Argentina
Agnieszka Roginska, New York University - New York, NY, USA
Thilo Schaller, SUNY Buffalo State - Buffalo, NY, USA
Is it just a fashion that over 85% of the people listen to their music on headphones? Is 360/VR/AR the only application where binaural audio is applicable? Could the mobile entertainment gain the need for new binaural audio productions in music, film, interactive, and game? And how to create and deliver binaural audio to the market? This and a lot more questions will be discussed by a panel of experienced binaural audio persons.