Remote Piano Learning
There is a version of Skype built for specifically for HoloLens. We came across it often in the early days of this project while setting up the infrastructure.
This got us to thinking about how to expand the music instruction concept. Could we open up our model to give piano instructors the agency and interface to work with their students remotely?
In such a setting, a teacher with a MIDI keyboard in their home or studio would be able to instruct their students and leverage the tools currently being built for the Music Everywhere system (e.g. keyboard overlays, 3D animated hand, musical score, etc). The instructor’s interactions would be transmitted via MIDI over the Internet and the student would see these actions through the HoloLens. The instructor might even be able to interface with the 3D hand using a Leap Motion device on their side of the “call.”
This idea would require the ability to send video (for face-to-face communication as well as for the teacher to watch the student’s hands on the keys), audio (for verbal instruction) and MIDI data (for what is played on either individual’s piano keyboard). We began drawing up a model to indicate what aspects would need to transmit in which direction.
From there, the mission was to determine a viable, technical solution. Where Skype had been our inspiration for the idea, there was a problem. Sending MIDI data would be outside the scope and capabilities of the current Skype build—even the one designed for HoloLens. Adding a “plug-in” to Skype wouldn’t be possible.
After some discussion and further investigation, we came across a separate Microsoft-supported, open source project called, WebRTC. WebRTC would give us the ability to customize the peer-to-peer connection to include all the data we’d need to pass from one side of the conversation to the other.
At first we struggled with the implementation. There was documentation on the site, but still we weren’t able to make the connection. Thankfully, a generous member of the WebRTC development team responded to our inquiries and provided sample code for us to learn the correct procedure.