Week 11

PROGRAMMING

This week we created a demo which demonstrates our entire machine learning pipeline as well as user experience flow. Here’s how it works

1. In the beginning, the user is given prompts to perform movements and record them as videos.
2. The user must then upload the videos to a specific Google Drive location.
3. We then run our machine learning pipeline which fetches the videos from the Google Drive location and the machine learning computations are performed on a Google Cloud Virtual Machine. After the processing is complete, we get output as animation data. This animation data is stored at a specific place in the Google Cloud storage.
4. After this we run our standalone application built on the Unreal engine. In the application, the users can click a button to download the latest animation data. The users can then play the animations which are created at run-time using the animation data.

This demo showcases our core game loop, but, is merely a demonstration of our technical achievements so far. We need to fit this into our storytelling experience in the next 3 weeks leading up to soft opening. Currently, the quality of animations is also very poor for the cinematic storytelling experience we are planning to build. In the next week, we will try to implement techniques which improve the animation quality. Apart from this, the speed at which resulting animations are playing is much faster than the original movements in the videos.

Here’s a look at the demo. The first video is the original video of movement and the second video is the resulting animation applied on our cat character.

We presented this demo to our client and got some feedback on the demo and the design of our storytelling experience as well. The feedback can be summarized as follows:

1. Focus on the core loop (Recording & uploading videos, and generating resulting animations in Unreal).
2. The current character has very short limbs and limits the range of motion. Try out different options to find out optimal type of characters.

DESIGN

We used our client feedback, to tweak the design of our experience. This keeps the context of the previous design but places emphasis on the core loop. In the next week we will implement this cinematic experience. We have worked on tightening this game loop and will discuss the design in next week’s blog. We are also working on refining the documentation to include all our new observations and learning.

Next week

In the next week we will focus on improving the quality of animations and adjusting the playback speed to match the original video’s speed. We will also implement the cinematic storytelling experience with a strong focus on the core game loop.