Designing the Bands
Having a guest play along with animated musicians required several components to work together in sync.
First, the music. A band has individual virtual musicians, many if not all playing at the same time. The programmers on the team built the band system to handle the synchronization of various players’ sound files.
NOTE: When working in development environments with sound clips which require looping or tempo-based accuracy, consider using WAV or other uncompressed audio formats. MP3s as it turns out have a difficult time keeping the correct length, causing timing issues.
Add on to that, we wanted to provide visual guidance for the pianist who’s learning, to play along. That overlay system needed to be kept in sync with the audio files, and it required two types of functionality: Timed and Improvisation overlays. The former is driven by scores created in the notation software, Sibelius, saved in XML format. The music everywhere system was designed to parse, read and translate the XML into a Unity asset, retaining the tempo and sync with the audio files. The latter overlays are derived from the band module that contains the audio files, and display overlays based on assigned MIDI note numbers.
And finally, the art. Our UI designer created virtual characters to play in the Blues, Rock, Jazz and Classical ensembles.
Each character was designed based on a real life musician in the specific genre. And a low-poly implementation of these models was required to preserve performance. Characters were assigned animation scores, which dictated the particular motions the characters would make, coordinated with the music they were playing.