UI Development – Keyboard Overlay Iteration

UI Development – Keyboard Overlay Iteration

We continue to experiment with variations of AR indicators in the HoloLens to demonstrate particular notes for the guest to play.

As previously mentioned, our primary design challenge is working within the limited FOV (field of view). Placing a piano keyboard overlay for the entire length of the piano key often produces cut-offs from the guest’s perspective. Our first iteration of the UI was to create a second piano key overlay, this time occupying 1/3 of the keys length, placed starting at the back of the key bed.  Shantanu programmed the application to toggle back and forth between large and small key display via a remote controller to demonstrate and test with guests.

Additionally, we experimented with other types of indicators. As demonstrated in the previous UI Dev post, we began with a series of orbs that hovered over the keys. However, the orbs themselves produced approximately 60% accuracy in users identifying specific keys to press. We added a dripping animation to attempt to be more specific.

Then, we moved on to a pyramid shaped indicator that drops from above and falls onto the intended key. This enabled more accuracy but the placement was at first too far forward and producing the cut-off challenges of FOV. We pushed the animated elements to the back of the piano keyboard.

From there we were inspired to try a larger arrow-shaped icon which gave us the real estate to incorporate finger numbers to instruct guests not only which notes to play, but with which fingers to play them.

[UI photos coming soon]

A quick test from within the notation software Sibelius proved successful. We will be able to export fingering numbers from notation via musicXML, and import into our score editor via Phan’s XML parser.

Finally, we will begin working on ideas for connecting the notes on the piano keyboard to how they’re represented on the staff. Below is an early concept from Yen.

Animated Overlay UI Concept
Animated overlay UI concept transitioning from score to piano keyboard (albeit at the wrong key)


Comments are closed.