Using games to study the behavioural and neural correlates of active auditory learning

Overview

Project ID: S3 8

As we listen to the world around us, we automatically learn the structure of the sounds we hear and form predictions about what we will hear next. For example, we can recognize when a word is out of place in a sentence, and we can tell when a musical phrase has ended. While many studies have looked at auditory perception through the lens of statistical inference (Clark, 2013), most have used paradigms involving passive exposure. Because of this, we know little about how active, goal-oriented engagement with speech, music, and other structured sounds allows us to learn them in the real world.

Games provide an active, engaging, goal-oriented way of creating learning paradigms to study sound sequence learning. Through gameplay, players can learn to associate sensory events with game-based actions that are required to advance (Schell, 2019). In this way, we can use games to study how engagement and meaning affect learning, and we can identify how the brain’s responses to the sound’s structure change through active experience.

Recent work in cognitive neuroscience has also demonstrated that the brain continuously tracks the structure of speech and music (Brodbeck et al, 2018; Di Liberto et al, 2020). Using state-of-the-art quantitative methods, we can map EEG fluctuations to changes in the expectedness of each sound in the sequence, allowing us to measure how well the brain encodes these expectations. However, it is still not clear how the brain forms these expectations with experience.

In this project, we will use games designed specifically to teach participants sound sequences and to recognize unexpected transitions between sounds. We will look at how gameplay affects participants’ ability to recognize familiar sequences and detect unfamiliar ones. We will then use EEG and state-of-the-art quantitative methods, based on machine learning techniques, to understand how the brain changes when learning through gameplay.

Students will gain experience with game design for neuroscience, EEG data collection, behavioural experiment design, neural data analysis, and computational neuroscience.

Supervisory Team:

Director of Studies: Dr Nathaniel Zuk, Senior Lecturer of Psychology and Hearing, NTU Psychology.

2nd Supervisor: Dr Chris Sumner, Associate Professor of Auditory Neuroscience, NTU Psychology.

3rd Supervisor: Prof Gary Jones, Professor of Psychology, NTU Psychology.

References:

Clark (2013), Behav Brain Sci, 36:181-204. doi: 10.1017/S0140525X12000477

Brodbeck et al (2018), Current Biology, 28:3976-3983.e5. doi: 10.1016/j.cub.2018.10.042

Di Liberto et al (2020), eLife, 9:e51784. doi: 10.7554/eLife.51784

Schell (2019), The Art of Game Design: A Book of Lenses, 3^rd Edition