Piano playing is a complex sensorimotor task involving vision, audio, and touch. Most previous attempts require a specific design or manual ordering. A recent paper on arXiv.org suggests a reinforcement learning algorithm so that an agent could learn directly from a machine-readable music score to play the piano.
The method exploits tactile sensors for control and uses the corresponding notes generated as a reward. The piano-playing task is formulated as a Markov decision process. A multi-finger allegro hand equipped with tactile sensors and a small piano keyboard are created to perform the task.
The experimental results show that the method can train a robot hand to play the piano with correct notes, velocity, and fingering. The study also shows the possibility of vision-based tactile sensors to improve piano playing, especially on the fingering indication.
The virtuoso plays the piano with passion, poetry and extraordinary technical ability. As Liszt said (a virtuoso)must call up scent and blossom, and breathe the breath of life. The strongest robots that can play a piano are based on a combination of specialized robot hands/piano and hardcoded planning algorithms. In contrast to that, in this paper, we demonstrate how an agent can learn directly from machine-readable music score to play the piano with dexterous hands on a simulated piano using reinforcement learning (RL) from scratch. We demonstrate the RL agents can not only find the correct key position but also deal with various rhythmic, volume and fingering, requirements. We achieve this by using a touch-augmented reward and a novel curriculum of tasks. We conclude by carefully studying the important aspects to enable such learning algorithms and that can potentially shed light on future research in this direction.