Making machines that can listen

Paris Smaragdis, a professor who joined MIT in 2025, is building machines that can hear and interpret sound at a level he describes as potentially better than human listeners. He holds a position in the Music and Theatre Arts program of the School of Humanities, Arts, and Social Sciences, shares an appointment with the Department of Electrical Engineering and Computer Science through the MIT Schwarzman College of Computing, and is affiliated with the Music Technology program and the Research Laboratory of Electronics. His group's stated aims include machine learning for sound source separation, smart devices that augment a person's audio surroundings, and tools for understanding, manipulating, analyzing and creating music.
The motivating scenarios are concrete: a system that could locate a traffic accident by recognizing squealing tires and crunching metal, or flag an industrial drill at risk of breaking while boring through dense bedrock for a geothermal well. Smaragdis frames the field by analogy with computer vision — the same perceptual problem, shifted from sight to hearing.
His path into the work began with classical guitar in Greece and a small synthesizer his father brought home; a function on it for designing original sounds hooked him. Wanting to study music made with synthesizers, a path traditional European conservatories did not then offer, he enrolled at Berklee College of Music in Boston, majored in bass to play more jazz ensembles, and with mentor Richard Boulanger built tools for writing, producing and enhancing music on computers. A Berklee collaboration with Barry Vercoe (who died in 2025) at the MIT Media Lab led to software synthesizers, then to master's and PhD study and a postdoc in Vercoe's group.
At the Media Lab, most peers applied classical signal processing to sound; Smaragdis took the then-less-traveled route of machine learning. His argument is that listening cannot be fully specified by hand: humans lack even a vocabulary for distinguishing, say, a cat's meow from a violin, so the skill has to be learned from examples. That framing anticipates how modern audio systems are trained rather than programmed — a point of context rather than a claim about any particular model.
At Mitsubishi Electric Research Labs he worked on AI that scanned video soundtracks to infer what was happening on screen. The motivating example: finding highlights in a full soccer match is hard visually, but a roaring crowd is an obvious audio cue. The system was incorporated into electronic devices, including a personal video recorder Mitsubishi was then developing, plus traffic monitoring systems and cell phones — an early instance of audio understanding shipping in consumer hardware. He later founded Adobe's audio research team, building tools that let artists, musicians and producers manipulate sound without specialist expertise.
He then spent 15 years on the faculty at the University of Illinois at Urbana-Champaign, where his group brought contemporary machine learning into audio processing. One result used neural networks to separate individual sounds from a mixture. That work reached the team behind the 2021 documentary mini-series "The Beatles: Get Back," which enlisted him to clean up archival Beatles recordings. The constraints were unusual: output had to sound right rather than optimize a clean formula, and each recording posed a different problem — one clip had John Lennon playing while Paul McCartney read newspaper headlines and teased George Harrison. Rather than solving cases one at a time, the aim was a general technological solution. His contributions helped the team win a 2022 Emmy Award for sound editing for a nonfiction or reality program.
At MIT his group pursues two strands. The first makes computers better tools for musicians — for instance, systems that listen to music well enough to offer constructive feedback, framed explicitly as extending a musician's toolbox rather than replacing the musician. The second builds fundamental tools for processing, analyzing and manipulating audio. A current direction is signal processing in which standard frames of reference — sampling rates, frame sizes, context windows — are not fixed or even defined. That implies unfamiliar computational structures; the payoff Smaragdis cites is more efficient processing and more flexible, more powerful systems, at the cost of rebuilding assumptions that most audio pipelines treat as given. The group also examines folding techniques such as large language models into speech recognition and audio enhancement, with Smaragdis noting that the ideal way to apply AI to sound and music has not yet been found.
For developers, source separation is the most familiar thread: isolating a voice, an instrument or an environmental sound from a mixed recording underpins noise suppression, transcription in noisy rooms, remastering and archival restoration. The trajectory described — from hand-designed signal processing toward learned models, and now toward models that borrow from language-model architectures — mirrors the broader shift in audio tooling. The unresolved part is evaluation. Restoration work judged on how it sounds rather than on an optimizable metric is awkward to benchmark, and this is my own reading: teams shipping audio features will likely need listening tests and task-specific measures alongside standard error metrics. Redefining the frame itself is a deeper bet; if it pans out, it could change how audio pipelines are structured at the lowest level, though no results or timelines are given here.
Why it matters: product teams building voice interfaces, call analytics, media tools or industrial monitoring depend on machines that can pick apart and interpret real-world sound, and this work sits at the methods level that such products draw on. The clearest near-term signal is direction of travel — learned separation plus large-model techniques — rather than a shippable release, and the MIT appointment suggests the research will be pursued in an interdisciplinary setting. My inference is that practical gains will arrive through better separation and enhancement quality rather than through any single new framework.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.