Overview
Our project started life as a semester-long effort to understand if and how visualization could be used to improve the story development process in filmmaking.
Our project started with each of us having slightly different opinions and understanding of what our project goal was but eventually, after team meetings and consultations with faculty and industry professionals, we decided our project goal was to use visualization as a tool to give filmmakers a rough version of their film in pre-production so that they could make better/more informed decisions in production.
To enable this, we developed 2 in-house tools – a virtual camera system and a motion capture system to animate characters.
WHAT WE’VE DEVELOPED
Our workflow is to start with a script and break down what are the story elements (assets to indicate the environment or pivotal for the story – eg: a cactus in a desert or a gun in a murder investigation) and the detail elements (assets that help distinguish characters from each other – eg: a police badge). This helps us break down our visualization to its most essential elements and help the viewer focus on the story. Then we create the 3D assets for each of the story and detail elements. Then comes the part of getting our “actors” to interact in the world we create.
Character animation is an essential part of traditional previz. However, it can take up a significant amount of time to keyframe animations for shots. In this project, we wanted to be able to quickly block in character motion with upcoming technologies like virtual reality. The idea was to get a rough outline of the motion, rather than realistically capturing performances.
For the camera system, we developed features that represent the real camera that filmmakers are familiar with – placing camera to get static shots, changing focal length, tilting and panning, dollying, adjusting depth of field (DOF) and racking focus. The camera view is embedded in a menu that users can tap on the touchpad to go through all the features.
WHAT WENT WELL
In this semester, we mainly finished 3 case studies based on existing scripts – the opening scene from Butch Cassidy and Sundance Kid , a scene from Moonrise Kingdom, and the ending scene from Se7en . While trying new scripts, we kept the tool development process ongoing parallelly. Except for Moonrise Kingdom , we visualize the sequence 100% based on the scripts to go through a real previsualization process. Based on our feedback, we successfully told understandable stories, and also managed to produce a large amount of shots with decent efficiency.
Exploring shots and blocking is simple and efficient in the VR environment, since the immersive environment is inspirational and our virtual camera allows simple and quick iterations for the user. For example, in our case study of Se7en, a lot of shots in the final clip (around 50-60%) are different from the designed ones simply because we found better solutions while exploring in the systems (like dutch angle shots and high or low angle shots). Second, the dollying camera movement helps a lot in building tension and a dynamic sense of space in the image. Most significantly, the DOF gives the image a sense of depth and clearly tells the viewers where to focus.
Additionally, in order to better organize the interaction caused by the huge camera features implementation, we explored multiple ways to improve the interaction and they indeed help the interaction much less annoying and frustrating. The menu we designed and the well-organized interactions ensure friendly user experience in our camera system.
Our simple visual style of the previz clips helps fit various types of scripts and with less details on human figures that users won’t find distracting. With our motion capture system, we are able to quickly capture motion using VR equipment, and record complicated scenes where we have multiple characters moving in our 3D sets and interacting. This has helped us save time and put a lot of shots together.
WHAT COULD’VE BEEN BETTER
During this semester, we were always trying to seek a balance between automated process and manual process in our workflow – since previz efficiency is one of our main focus, but meanwhile we only have limited tech people in this team. So in our developed workflow, motion capture is still an involved process – individual animations are still put together sequentially by hand. Automating the entire process would require a larger team and time. Also, not being able to play or pause animations in the VR camera system brings difficulties to control the timing. Another problem is the camera shake in our shots that, albeit for artistic purpose, is still at some level distracting for the viewers.
As we were discovering solutions to refine the interaction in our tool, we found a time conflict with camera features implementation. We were all learning about which tasks were in most priority, what was the correct order to work as we have a single programmer working on the camera system. We chose to add camera features ahead of UX improvement because making a tool that has enough functional features for shots taking is our goal above all.
A question we received throughout the semester was “Who is your end user?” At first we agreed that our project is more about discovering ways to produce quick and cheap previz, so a lot of features were developed based more on our assumption of potential users. However, lack of clarity of specific user cases sometimes cause vagueness in the project direction and our understanding of how the tools would be useful in real cases. Although our project is more like a combination of product development and animation production, and it requires different production methodologies and the balance between them, we should always have a clear idea of user cases (which could also be pivoted during development) to better fit into a real-world context.
LESSON LEARNED AND CONCLUSION
From a previz perspective, what we learned from shooting is – First, DOF is one of the essential elements for visual storytelling that would be useful to visualize. Second, a VR environment is more helpful than visualizing in a 2D tool, since it gives the artists an immersive sense of the world they build and freedom to explore in it.
From the perspective of developing tools, we learned to be more specific about the user cases of our tools to gain a clearer idea of what features are needed and what are not.
This project taught us a lot about the film industry, the workflows that guide it and we saw the benefit of how visualization could help the filmmaking process by adding a step early on in the process that reduces the complexity later on.