How AI Learns to Hear: Understanding Machine Listening Technology

What is machine listening technology?

Machine listening technology is a research project focused on building computer systems that possess superhuman hearing capabilities. According to an MIT press release, this technology is intended for practical applications including speech recognition, music understanding, and environmental monitoring. Please note that this announcement is a preprint or press release from MIT that has not been peer-reviewed.

How do machines hear and understand sound?

Professor Paris Smaragdis leads this research at MIT, where he works on creating machines that can hear and understand sound better than humans. Raised in Greece, he went to Berklee College of Music, where he focused on bass and learned to utilize computers and synthesizers to manipulate and shape sound.

During his time at Berklee, Smaragdis collaborated with his former mentor, the late Professor Barry Vercoe, to create software synthesizers. Smaragdis completed his master’s degree, PhD, and a postdoc in Vercoe’s group at the MIT Media Lab. After a 15-year tenure at the University of Illinois, Smaragdis joined the MIT faculty in 2025.

Who is leading the MIT research on machine listening?

Smaragdis chose to use machine learning to teach computers how to listen, as he found it offered advantages over traditional signal processing techniques. Smaragdis chose to use machine learning to teach computers how to listen because he found it offered distinct advantages over these traditional techniques.

The research team is currently exploring signal processing algorithms that do not require fixed sampling rates, frame sizes, or context-windows. By moving away from these fixed constraints, Smaragdis believes that rethinking basic signal processing can result in more powerful and flexible systems.

The group is investigating how to incorporate techniques like large language models into audio enhancement and speech recognition.

Where has this audio technology been used before?

At Mitsubishi Electric Research Labs, he worked on AI systems that could scan video soundtracks to identify specific events, such as highlights in a soccer game. Later, he created an audio research team at Adobe that developed user-friendly tools for artists, musicians, and producers to manipulate sound.

This sound separation technology was used in the 2021 documentary mini-series "The Beatles: Get Back". Smaragdis noted that cleaning these archival recordings was unique because the process lacked a fixed formula to optimize, which differed from his typical academic work. Smaragdis and his team received an Emmy Award in 2022 for sound editing on a reality or nonfiction program.

Currently, his MIT research group persists in investigating machine learning for sound source separation, developing smart devices to enhance human audio environments, and building tools for the understanding, manipulation, analysis, and creation of music.

Will AI replace musicians?

The goal of this research is not to replace musicians but to provide them with new capabilities. Smaragdis stated that the aim is to enhance the toolbox a musician has rather than using computers to replace them.

His research group at MIT develops systems that enable computers to listen to music to provide constructive feedback to performers. In practice, the group creates fundamental tools for artists and scientists to process, analyze, and manipulate audio.

What is the next step for audio AI?

While AI is an immensely powerful tool, Smaragdis believes that researchers have not yet found the ideal way to apply it to sound and music.

How are students learning to build machine listening systems?

At MIT, Smaragdis uses hands-on lessons to teach students how to turn equations into tangible audio experiences. This approach allows them to turn mathematical equations into tangible audio experiences.