We are going to work on the Practical AR project for this semester! For the past three semesters ETCers have been exploring augmented and mixed reality experiences in Silicon Valley using Epson Moverio AR glasses and Google Project Tango tablets. In this next step of moving towards an augmented world, we are asked to develop an augmented or mixed reality application that has real value for users today. The base device will be either an iPad or an iPhone with a Structure Sensor by Occipital mounted on it, and may be hand-held or head-mounted. The sensor enables 3D scanning for volumetric reconstruction, real world occlusion and unbounded positional tracking, all supported by Occipital’s Structure SDK.
Everyone was super excited even though the first week was mainly about set up in the new work space and some administrative work. We settled on a schedule for core hours and weekly meetings with our client and advisers. Our kick-off meeting with our advisers Carl Rosendahl and Heather Kelley was on Tuesday. They talked to us about the ETC project process at SV campus and discussed our individual roles, and went over the project goals, expectations and metrics. We also discussed about our questions and concerns. The final deliverable will be a distributable app that solves a current real world problem for its users by the end of the semester.
We had an internal discussion after the kick-off meeting. First, we talked about everyone’s ideal role of the project. Our skillsets are very balances, we have two programmers (Jack and Atit), two artists (Qing and Sophia) and one designer (Wei). Atit will also be the producer of the team. Second, we came up with a rough schedule of this semester. Considered the long review process of App store, spring break and GDC, we have to be very efficient to deliver a polished App on App store. Our plan was to start prototyping from week 3 and ideally submit the first prototype before week 8 to ensure the successful delivery of the project. Thirdly, we named our team as ARchitect to indicate that we are going to use AR technology as innovative creators. Last, we decided to do our individual research and think about 3 possible ideas before the next meeting.
We not only looked into traditional AR games and Apps, but also downloaded all the Apps that supported by Occipital Structure Sensor to experience. All the Occipital Apps are based on 4 main features of the structure sensor:
- 3D scanning & texturing
- Indoor Mapping
- Positional Tracking
- Augmented Reality
Based on the research and brainstorm, we came up with 5 ideas:
- Shooting Genre: Players can use virtual bullets to shoot virtual targets in the real environment. Because of the positional tracking feature, it’s easy to know where is the player and how the player moves. With the Indoor mapping technology, we can scan the real room and generate a virtual mesh based on the geometry. So it’s also possible to make a real-world version Angry Birds which allows players to smash their “room” with virtual “birds”.
- Sandbox: Take advantage of the Indoor mapping technology, we can generate the virtual model of their real rooms. Players can edit their own levels in the room and upload it to share with friends. Other players can get access to the virtual rooms and play inside without having the sensor. The concept is similar to King of Thieves. There are many possible gameplays, for instance:player A hides something and set up obstacles in the room; player B plays a treasure hunting game inside.
- “Fish Tank”: We wanted players to consider the tablet/phone as a magic window to observe an alternate world. Players can pre-scan the room and a parallelled living world will be generated based on the geometry of the real world. Players can look through the tablet/phone to observe or interact with the virtual world.
- Plastic Surgery Preview: The 3D scanning & texturing feature allows us to scan a high poly detailed model in a very short time. We wanted to allow users to scan their faces and customize the facial features or hair styles. It could be valuable for people who want to have a plastic surgery or hospitals.
- Escape Room: Players can pre scan the room and we will generate puzzles based on the mesh information. We wanted players to hold the tablet and actually walk around in the room to investigate. A possible gameplay could be: we hide a clue inside the wall, players can throw virtual balls to find out the empty block based on the sound differences.