This semester, our team, FeelsBot, was given the unique opportunity to create an emotional virtual reality experience in conjunction with Oculus Story Studio. Ultimately, we have created an audio-narrative exploration game, titled “Left Unsaid,” in which the player explores the past with the goal of reconciling two family members. Early on, the decided to test whether an audio-narrative exploration would be engaging in VR by utilizing a mechanic in which audio was played through items, rather than ambiently as is seen in typical non-VR experiences such as Gone Home. This meant that, as a team, we attempted to tackle a new mechanic for this genre. This required: careful and purposeful writing which could elicit meaningful emotion and inform the player, constant design iteration to craft an engaging experience which runs at a significant fifteen minutes, all while wrestling an unfamiliar game engine, Unreal Engine.
Left Unsaid development was not without its many challenges and setbacks. Early into creating the experience, it became apparent we would need to rely on outside sources to allow us to fulfill our vision. Specifically, we would need to hire outside actors and a composer to make sure we had professionals assist in bringing our characters and world to life. Also, we would need to rely on Oculus Story Studio to provide their Touch Controllers, as we knew from very early on we wanted Left Unsaid to be a fully interactive game. The team soon learned, however, that bringing in any third party or requesting hardware would be met with weeks of legal setbacks. In retrospect, we were very lucky that we pursued these outside resources earlier than required, otherwise our deadlines would not have been met, resulting in a sub-par experience.
Conquering Unreal Engine was also a major hurdle for the team as we all learned the do’s and don’ts of working with such a powerful and complex piece of software. We had to dedicate several weeks of time for our programmer, Justin to become proficient enough to begin technical development. Then, we spent more time constructing an effective pipeline for art, as many models would not import or render properly. Ross, with a background mainly in 2D art and game design, had to quickly learn how to make the dozens and dozens of models necessary to populate the many time periods of our game. Needless to say, there were many long and frustrating nights filled with trial and error.
DESIGN
Player Identity
A number of game design related issues also presented themselves over the course of development. “Left Unsaid” is primarily about uncovering a story that revolves around two NPC’s. Early on, we decided that the protagonist of the story would essentially be the player rather than a character. This meant that their character would have no backstory, no character development, and no voice. Their sole role would be to uncover the story of Candace and Edward. This decision was not made without precedent; many games (The Legend of Zelda, Half-Life, and Elder Scrolls, among others) have employed silent protagonists to give players more ownership over the emotions that they are experiencing throughout the game.
For a significant portion of development, the only direct mention of the player-character came during a phone message from Candace at the beginning of the game when she leaves a phone message for the player and briefly makes mention that the player-character is moving into her father’s old house. After extensive playtesting we found that this lack of character definition confused or frustrated players. Many players could not recall the exact phrasing of Candace’s phone message (we theorize that this is due to the overwhelming cognitive load of VR), and spent the remainder of the experience attempting to use context clues to determine their identity. At best this search for player identity acted as a distraction from the story that we wanted to convey, and at worst it led to incorrect assumptions that resulted in interpretations of the experience that differed drastically from the what we intended (many testers assumed that they were Edward since the game takes place in his office. A smaller group assumed that they were Candace.)
We solved this problem by giving the player-character a small amount of backstory (they are a close friend of Candace’s). We used a multi-pronged approach to conveying this information throughout the game’s prologue. Candace clearly states all relevant information on the phone and we also include a few notes and documents scattered around the office that reiterate the important bits of information for understanding the basic traits of the three characters in the story. These changes have largely cleared up the player identity issues that plagued us for weeks. The main takeaway from this experience seems to be that audiences’ attention spans and comprehension abilities instantly drop when wearing the HMD. Being immersed in a new world is an overwhelming experience, especially for the first few minutes. Designers should look for ways to ease players into the game before conveying important plot information and should attempt to reiterate that information in as many ways as possible.
World Building vs. Game Mechanics
The struggle between game mechanics and cohesive world-building was also a constant tug of war for our team. In our earliest design docs and story treatments, Edward’s machine had one function (transporting users to a “spirit realm” where they could listen to memories associated with various objects around the room). As we developed our story we quickly realized that we needed a way to pace the experience. It was impossible to create a coherent narrative if players could experience any part of the story in any order. This was the equivalent of giving players an out-of-order DVD scene selection menu and telling them to try to assemble a coherent narrative. To solve this problem we introduced the concept of time travel into our experience. The player travels from period to period in a predetermined, linear fashion. The player would have the opportunity to travel to Edward’s office in four distinct eras; 1985 (both at the beginning and end of the game), 1956, 1970, and 1977. This allowed us to gate the experience so that we could be sure that certain plot points would be discovered before others. We now could create a story that fell roughly into the traditional five act story structure and featured tidy character arcs for both Edward and Candace. We also determined that the mechanic for activating time travel would be to look at a retina scanner that emerges from the machine. This decision was made to guarantee that players would have to look away from the main office scene for a moment, allowing the level loading transition to occur. This also ensured that players would get to swivel around from the eye scanner and be surprised by the new scene in front of them. Around this time we also decided that a powerful climax to the narrative would be allowing Edward (who is deceased in the present of the story) and Candace to have one final conversations across the dimensional boundary of the afterlife and the corporeal world.
These decisions made sense from both a game design and plot perspective, but they diluted the world-building of “Left Unsaid.” Now the machine had three distinctive functions: transporting players to the spirit world, time travel through the spirit world, and connecting the dimensions of the afterlife and the living. Making our world more complex in the name of game design certainly had its benefits, but it made also made our world more convoluted as a result. While the plot and the game mechanics and flow were made more lucid, the rules of our fiction became more obtuse. There is probably a way that we could have struck a better balance between these poles, but it would have required a significant retooling of the experience.
Other challenges emerged from our process of converting an established genre (audio narrative exploration games/ walking simulators) into the medium of VR. We looked to games like Gone Home, Firewatch, and Everybody’s Gone to Rapture as our primary points of reference. In these games, players uncover a story by reading documents that they find around their environment and listening to audiologues. These games live and die by the quality of their environmental storytelling, as well as the quality of their world and exploration mechanics. In each of these games the player walks around a spacious environment and uncovers a mystery. They usually need to find keys, tools, or other elements, to reach areas of the environment that were inaccessible at the beginning of the game. We made an early decision that “Left Unsaid” would take place within one confined space. This was based on a few factors: hardware limitations, simulated movement in VR can result in nausea, and creating a room scale experience was not an option since our primary platform was the Oculus Rift. Our solution to this problem was the aforementioned time travel mechanic. Rather than traveling through a large environment, we chose to change the setting around the player as they progress through the experience. This solution worked well for this particular problem. It gives players a sense of discovery while playing a stationary, seated experience.
The Throwing Problem
We also learned through testing that players enjoyed throwing our objects around the room without giving much thought to their narrative implications. For a while we wrote this problem off as being insignificant. We decided that throwing things around without real-world consequence was fun in VR and there was nothing we could do to stop it. After some thought, we determined that this behavior was symptomatic of a more serious, underlying issue with our game. As designers, we put each object in the environment into one of three classes: Critical Path Objects, Backstory Objects, and Set Dressing. In our world, Critical Path Objects emit the audiologues that contain critical plot information, These objects also act as fuel for the spirit machine, thus they are also key to advancing to the next time period and ultimately completing the experience. Backstory Objects do not emit sound and are not necessary for completing the game, however they contain visual information that contributes to world-building and the characters’ backstories. These exist to provide more information to players who are engaged with the story. These include objects like notes, trophies, and knick-knacks which give the observant player a new layer of depth to the story. The rest of the objects are used for set dressing. These include things like pens, staplers, and mugs. These exist to make Edward’s office feel more like a lived-in space, but do not contribute directly to the environmental storytelling. While the player’s primary goal is to find the Critical Path Objects, we wanted the environments to feel as immersive and engaging as possible.
After several weeks of watching players picking up objects in rapid succession and throwing them across the room, we came to realization that, in the player’s minds, there was no rhyme or reason to which objects were critical and which were not. For example, why was Edward’s model rocket a Critical Path Object, but the family photo on his desk was relegated to a Backstory Object? To solve this problem we built simple puzzles into the experience. We made the choice to hide at least at least one of the Critical Path objects in each of the time periods. We included clues in the Backstory Objects for how to reach these objects. To give an example for how one of these puzzle sequences works, I will explain the order of events in 1970, the third time period that the player visits. When the player reaches this time period and looks around, they will notice a stack of film reels on the cabinet to the left of them. The film reel on the top of the stack is a critical object that provides exposition for this era. Underneath the stack of film reels is a key to Edward’s desk drawer, inside which, the player finds the second Critical Path Object, Delilah’s birthday party invitation. Underneath the invitation, the player finds a detention note from one of Candace’s teachers that implores Edward to find and throw his daughter’s cigarettes in the trash. This is a clue to the location of the third Critical Path Object. If the player digs through the trashcan to their right, they find Candace’s cigarettes, which contain the final, critical piece of information for this time period. These puzzles are not meant to stymie progress, rather they are meant to encourage players to carefully consider the importance of the objects that they choose to interact with. From our playtests, it is clear that these puzzles slow down the experience and make it more contemplative, encouraging critical thought and deliberation from the players, which almost entirely eliminated our “throwing problem” and significantly improved player’s retention of the story.
VR Affordances
The concept of VR affordances also had a significant impact on our design process. In games that use a controller or mouse/ keyboard as the primary input device, a layer of abstraction exists between the player and game. With this control setup, context sensitive button presses to accomplish complex actions are the norm. For example, in most games, you would simply tap one button to open a closed door or drawer. This is not the case in VR experiences that use hand controllers. The level of abstraction between the player’s physical body and the game has been lessened. Because the player is actually performing gestures with the hand controllers that mimic real world actions, designers and artists must pay careful attention to the visual designs of interactable objects in their games. The idea of affordances is a useful way to frame this conversation. Affordances are an industrial concept that emerged several decades ago which centers around the idea of physical objects communicating their actionable properties to users through their visual design. For example, it is usually easy to discern whether a door should be pulled or pushed based on the aesthetics of the door handle. If a door looks like it should be pulled, but it actually should be pushed, it creates an unpleasant experience for users. We tried to take these kinds of design considerations into account when creating our environment and the interactable objects therein.
These design considerations were perhaps the most pertinent in the context of the creation of Edward’s transdimensional travel machine. Obviously this is a fantasy creation with no real-world amalgamation. We needed the machine’s functionality to be clear to players almost instantaneously. One of the machine’s primary functions is to siphon “memory fuel” from the Critical Path objects in the experience. We put a “microwave” on the front of the machine with a door that lifts upward (as designated by the handle placed on the bottom of the door). It was immediately clear to players based on their preexisting knowledge of affordances how to interact with this device. We also included an analogue meter with a moving needle above the microwave to communicate to players how much fuel the machine still needed in order to give them an indicator of their progress. The machine also employs a retina scanner to travel from time period to time period. The scanner resembles a camera and features a blue, glowing lens. We also observed this interaction as being fairly natural for testers (despite a speed bump early in development when the camera was not providing enough feedback. Now the scanner grows brighter the longer the player looks at it, and makes a noise if the player diverts their gaze.)
The machine also has a monitor located to the left of the microwave. We used this monitor as an immersive way to communicate UI information to the player that might typically be seen as part of a HUD in a non-VR game. This screen was also used to provide the player prompts if they ever got lost or confused. At time in the experience after the prologue in 1985, the player can look at the screen for direction.
ART
Next, we will pivot from design to discuss the process of defining the art direction and creating the numerous assets that player interact with in “Left Unsaid.” Early on, we decided to pursue a low-poly art style. This decision was made largely due to the fact that we needed to make a significant amount of art assets, but our team only had one dedicated 3D artist and one 2D artist. We drew inspiration from other games with low-poly, stylized art for inspiration like Virginia and Firewatch. Because “Left Unsaid” is a period piece focused on environmental storytelling, it was important that each asset looked as if it could have existed in the appropriate time period. We researched architecture, toys, furniture, and other objects to make sure that they were evocative of the mid-century aesthetic that we wanted to capture.
It was also important for our team that objects had character. Because you never actually see a human character throughout the experience, we wanted the important objects in our environment (things like Edward’s machine, the toy, bunny, the desk, etc.) to be memorable from a visual standpoint. We accomplished this by paying careful attention to the silhouettes and color choices.
TECHNICAL
Unreal vs Unity
In deciding between the game engines, we considered the advantages and disadvantages that each may bring to the team and the experience. If we had chosen Unity, we would have the advantage of working with the engine in the past. We would be able to begin prototyping during pre-production to explore the feasibility of more ideas. However, we also understood the limitations of Unity, particularly in the area of lighting. Without an efficient global illumination system, the Unity engine requires many different lighting optimizations in the form of baked lights, shaders and textures to improve the graphics in a scene. With our team of one programmer and two artists proficient in 2D, we could not dedicate the resources to learn these lighting optimization techniques.
On the other hand, Unreal was a new engine that we were not familiar with the exception of Brian, who had experience working with Unreal’s blueprint system. Unreal engine has a great lighting system that is able to illuminate the scene beautifully. It is inbuilt in the engine and required no extra work on our end. However, it also came with a level of complexity that required time to learn. This meant that we would not be able to test out prototypes in VR since time allocated to that would be spent learning the engine instead.
Considering the team’s vision type of experience, we knew that the mood and tone in the whole experience was essential to effectively convey emotions. Since our experience is an emotional story in VR, this meant that the overall visual effects took priority over the interactive mechanics of the experience. The lighting system in Unreal engine was able to satisfy our needs better than Unity, hence we decided to choose Unreal instead. The downside was time spent learning about the engine. Since Justin was the only programmer on the team, he would be the only one focusing on the engine while the rest of the team focused on other parts of the experience such as the story and art. We also set a six weeks time period for Justin to learn and experiment with the engine to ensure that we would have sufficient time to work on the experience. Apart from that, we avoided any complex game mechanics that would take a time to implement otherwise.
Level building with Unreal objects
Blueprints in Unreal allow a programmer to create objects with customizable properties and functionalities. By doing so, the programmer can create objects for level designers to place objects in the scene. This was crucial for the project since we had a tight schedule. Apart from that, since production of art assets started before engineering, there was a backlog of art assets that were ready yet not added. Hence, Brian was able to tweak properties of readymade objects and added them to the scene while Justin built more objects to be used. This concurrent workflow allowed us to speed up our production schedule.
Optimizations to achieve 90 FPS
Maintaining a high and consistent framerate is essential in VR. However, we were facing severe framerate drops as we began adding more objects into the scene. We were able to solve this problem after attempting a number of different solutions.
The profiler provided by Unreal was extremely helpful in helping us to identify the crux of the problem. We used the stats from the profiler to determine that most of the computation time was spent on lighting up the scene and calculating the shadows required for each object. Knowing that, we immediately tried to reduce shadow draw calls on various items in the scene. We also began learning about baked lighting to achieve similar lighting results. However, we were confronted with the problem of lightmaps and lighting. We were unable to achieve a similar lighting result despite numerous at using custom lightmaps on static objects and shifting lights around the scene.
Failing to bake lightmaps correctly, we had to fallback to using Unreal’s solution of stationary lighting. To solve our problem, we attempted to cut down the number of lights in the scene and ensured that only one light in the scene could cast dynamic shadows. This led to a more consistent framerate while maintaining a visual aesthetic that we thought was good. Despite that, lighting for static objects in the scene were still calculated at runtime since we were unable to bake lights correctly. Learning how to do so would have certainly allowed us to push the system further to create better visual effects.
Conclusion
This semester, FeelsBot has created an audio-narrative exploration game, “Left Unsaid” in only a handful of weeks. The team has worked constantly to make sure the game is as fun, engaging and emotionally impactful as possible for the player. This project has allowed an immense amount of learning in terms of virtual reality design, game design and rapid prototyping. While the team definitely had mistakes and missteps along the way, we are all immensely proud of the experience we have created and look forward to showing the world what we have created.