FINAL PRESENTATION.

The semester (finally) went to the end. Many faculties and guests mentioned the project was a huge success and there was a lot of potential.

One of the things, the team didn’t mention during the final presentation, was how others helped them went through the whole semester and overcame different kind of unexpected.

– Thanks to the client, communication over 13-hour difference was not a problem.
– Thanks to the instructors for always encouraging the team especially during the meta development phrase, it was hard for anyone to evaluate what we have done.
– Thanks to the IT team for the devices, especially for HoloLens and the micro-USB charger during ETC showcase
– Thanks to all the guests and faculties’ feedback, especially especially David Culyba for always asking us insightful questions and John Dessler for discover the hidden efforts behind the fun demo and all the critiques.

Without any of you, the team could not have been this far.

There were many things to be proud of,  but one of them is very unique and important.

Thanks to the hardworking programmers, the team eventually became the first team ever in the whole HoloLens community to able to send file to HoloLens, and they decided to make the application open source.

Thanks.

This week the client Erle Lim visited the team again and they have lots of discussions about the project.

Tuesday, a group of visitors from Merck, one of the largest pharmaceutical companies in the world, visited the project room and they are amazed by the project a lot, which turned out to be another 2 last-minute visit from Merck.

Wednesday, lots of guest for Entertainment Technology Center Spring Showcase visited the project and were excited about the project. However, HoloLens ran out of battery several times and only half of the guests were able to test the demo by themselves while other guests ended up with watching the live stream of HoloLens and deeper discussion about the project team members.

The team was still migrating their work to HoloLens and the client provided several new scenarios about disease – symptoms.

They team also discussed about the lessons they’ve learned from doing the project and there were 2 parts should be done better.

• playtest

As the project was very specific to its demographics : medical teachers and medical students, it was very hard for the team to arrange playtests.

The team had the common misconception that it was important to test with people of certain demographics to get useful results only to find out that the reality was that they can and should test their solution with almost anyone, at least for usability and intuitiveness. There was a saying in big data industry, more data beats better algorithm.

Though at the middle of the semester, they team finally connected to several medical students from UPMC and had them did regular playtests with positive feedback. The team should have utilize events like ETC playtest day.

• configuration tool

The team didn’t know the different between HoloLens and the former device that files sent to HoloLens should be deployed to the internet first, which might be not trivial to do for non-
technical people and there were not enough time for them to design an intuitive system for teachers with little technical background.

But the team made clear documentation and the configuration part of the tool was fully tested.

Soft opening.

The team got a lot of useful insights during the walking around and had noticed several aspects of the application needed to be urgently addressed. Although the application was not intended for ‘naive’ guests, there are some common things the team need to be aware.

The team also discussed new ways of interaction. Currently they used Google Voice API to enable voice control in several checking procedure, which is convenient and precise. But some playtesters mentioned that voice was a new input but it is not fully connected with all checking procedures. Some procedures were better connected with voice as a doctor would need to ask their patient to do things when doing examination but other procedures were less connected with voice as a doctor would do physical operations in clinical.

The decision is enable both way of interaction and decide whether disable one of the two ways based on play test.

Given the time constraint, the team noticed that they would not be able do fully optimization specifically for HoloLens and the device was not magic. There was frame rate issues and brightness, color issues need to addressed.

Overall, the team was confident about the project and excited about the visit of their client from Singapore.

The team discussed some design challenge with instructors.

The team used a ‘cursor’ to indicate where the user is looking at as one of the ways to precisely interact with virtual elements. It actually helped the user about control but it raised another problem. The user need to focus back and forth since the ‘cursor’ is in 3d space and in between the user’s eye and the virtual object being looked at. Some of the playtesters mentioned the experience is not very natural and pleasant.

The team also came up with some physical set up of the system to take more advantage of lighting given the limit of the AR device they have, e.g. the field of view is very narrow, the brightness is low.

HOLOLENS

Microsoft finally shipped HoloLens and the team was extremely excited about the possibilities come with HoloLens. After played with the device, some of the concerns they team used to have were no longer problems and team had to adjusted current application as they reinvented some wheels for HoloLens. The way of interactions need huge changes.

This week the 3d artist made several textures for the symptoms like “spider” and “meduza”.

The designer made the html template for the report after each session for users to get feedback and keep track of their performance.

The graphic programmer made a new shader such that it can render multiple textures in different layers on the object which can reduce the workload of GPU and the “true” customization of symptoms.

PLAYTEST

On Saturday, 4 medical students from UPMC and 1 guest came for playtesting. The guest is actually one of the investors of meta AR glass and she mentioned that with the 2nd edition of meta, things get better a lot.

The team recorded the whole playtest session and the medical students all pointed out that this application could have made their lives so much easier and the visual was so real. As the only practice source is the TAs of the course and there are more than 200 students, many of them didn’t even have the opportunity to practice with their TAs. More importantly, the “actor” can perform facial expression but they cannot perform abnormal organs so the students cannot have the feeling until they face a real patient.

The team arranged another play test session with some new medical students.

This week the 3d artist made several colliders for the programmer to determine whether the user is doing a touch operation and finished different textures of the body he made. He also rigged the character and started to working on textures of symptoms.

Programmers made several interactive prototypes which also lead to a discussion with the instructors on what should the team focused on doing in the rest of the semester, especially before softs opening where all the faculties will visit and try the app the team had been working on for a whole semester.

One of the programmer worked close with the UI designer on iterating the UI again.

The team also arranged a play test session with medical student from University of Pittsburgh to accommodate their inputs and thoughts of the application, what kind of information they want to have access to when the practice palpation and what are the common steps are likely to be missed when the follow the examination procedure.

This week is 1/2 Presentation.

The team did a lot of practice on weekends.

The team also participated all the sessions of presentations by peers.

There was a meeting on Friday in which the team discussed the current process and re-distributed the work load based on current knowledge and schedule.

The team also received a lot of great feedback from faculties about their presentation and product. They believed that they should have talked to faculties beforehand to make sure all the bells and whistles of the project is well understood and no confusion will be caused.

Carrie and Richard left for GDC week, the rest of the team stayed.

GDC 2016

This week the team mainly worked on preparing 1/2 presentation material and compile them to a informative presentation.

The team also told that Microsoft is going to start shipping HoloLens from March 30th, 2016. More info can be find on their website.

Want to start creating? If you haven’t already done so, visitwww.hololens.com/developmentedition to apply now for the Microsoft HoloLens Development Edition. Devices start shipping in the US and Canada on March 30^that a price of $3,000.

On Wednesday, the team introduced the project with 1 guest and 1 potential ETC student.

On Friday, the team did a rehearsal with the instructors and changed the slides accordingly.

On weekends, the team did some extra rehearsal in both project room and the RPIS.

Spring break.

The programmers finished the synchronization between physical organs and virtual organs which is also a sign for the finish of practice mode. The student would be able to see the movement and the organs and feel them at the same time, which should help them make the connection.

The artists had a discussion on how to improve the user interface. They made an agreement that with narrow field of view, more visual elements should be used while less text should be used.

Some discuss are documented as follows:

This week, the ETC has several talks and the team attended them.

The programmers are working on refining current prototypes while the artists are working on drawing storyboard and customer-journey-map, which is a diagram that illustrates the steps the customer(s)/user(s) go through in engaging with the application and the installation.

On Wednesday, the team received process grade for quarters and it was (a little bit) beyond the expectation for ETC student. 2nd years went for the OPT sessions after the 1 on 1 meeting with instructors.

The team got the 3d model used to create Abe but it is all shells after converting bi-rep to 3d print files and the object names are not organized. And there are a lot of holes in the 3d models.

On Friday, the 3d artist (almost) finished clean up process for the models created Abe and the programmers will start making the config tools and try to synchronize the organ movement.