Researchers at the University of Southern California Viterbi School of Engineering have developed a robotic hand that can hear a melody and reproduce it on a keyboard after a short period of self-guided practice. The system, called the Musician Hand, uses four tendon-driven fingers controlled by small electric motors designed to mimic aspects of human hand mechanics. Neural networks analyze the sound of a melody and translate it into motor commands that allow the robotic hand to play the notes back on a keyboard.
According to the research team, the hand learned through a process described as “motor babbling,” in which it randomly pressed piano keys for two minutes while recording the sounds produced and the movements associated with them. After that training period, the system was able to hear and reproduce a previously unknown melody of about 30 notes without using sheet music or pre-programmed scores.
The project was led by Hesam Azadjou, a doctoral candidate at USC Viterbi and the USC Alfred E. Mann School of Pharmacy and Pharmaceutical Sciences, under the direction of Francisco Valero-Cuevas, professor of biomedical engineering, aerospace and mechanical engineering at USC Viterbi. The findings were published in the Journal of the Royal Society Interface.
“The Achilles’ heel of traditional robotics is the assumption that perfect information is necessary to act well,” Valero-Cuevas said. “Animals don’t work that way. They perceive; they guess, usually correctly; and they adapt. We wanted to show a robot could do the same.”
The researchers describe the work as a proof of concept for perceptual robotics, an approach in which machines learn through perception, experimentation and adaptation rather than relying primarily on extensive programming or large training datasets.
The team said the approach could have applications beyond music, including rehabilitation, prosthetics, physical therapy and assistive technologies for people with movement disorders such as Parkinson’s disease. Azadjou also pointed to potential uses in home-based therapy systems that could learn from clinicians and adjust to individual patients’ movements.
The Musician Hand remains a prototype. The research was supported by the National Science Foundation and the Defense Advanced Research Projects Agency.
Photo credit USC Viterbi School of Engineering
