'Audeo' teaches artificial intelligence to play the piano

Music is not just notes on pages, they are something more than that occurring between the performers and instruments.

Some of the researchers of the University of Washington come to a thought that whether artificial intelligence could bring more to the musical experience. And, thus with the help of machine learning, they created a system called Audeo. From silent piano performances, the system created audio, and the music created by the Audeo was tested on SoundHound and it correctly identified the music played by it.

Eli Shlizerman, assistant professor of Electrical & Computer Engineering and Applied Mathematics said that "it was believed to be impossible to create music as played by Audeo. To check out the cues in the video frames that help to generate music, algorithms are essential. And also the system must imagine and check out the sound between the video frames. Hence a precise and imaginative system is necessary. The music achieved through Audeo was a surprise."

'Audeo' teaches artificial intelligence to play the piano

How Audeo Works

Audeo consists of a detector, translator, and synthesizer for its operation.

  • Detector: For each video frame, the detector detects the keys pressed to create the diagram over time. Music from this stage is of low quality.
  • Translator: It translates the diagram.
  • Synthesizer: Accepts the input from the translator and recognize the sound. Better quality music is achieved here.


Output Testing

Audeo output was tested using the Youtube videos of Parl Barton. Using 19,000 video frames they tested Audeo.

Type of Synthesizer to be Used

It was found that every synthesizer gives out music in a little different way. So, researchers made use of two different synthesizers. Fluidsynth gives out familiar music sound. PerfNet AI synthesizer was also used which give out better music. But it generated some noise too.


Audeo could be made into a virtual piano along with a camera that records the persons' hands. Also, Audeo could help in better ways of teaching piano by placing a camera on the top of the real piano.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.