Post Mortem

Post Mortem

Introduction

The team consisted of the following ETC Students: 

  • Roderick Luke “Derek” Chan, 2nd Year: Producer
  • Jiewen “Angela” Chen, 2nd Year: Front-end Developer
  • Euyne “Grace” Kang, 2nd Year: Visual Designer
  • Kathryn-Mae Eiland, 2nd Year: Assistant Producer, Designer
  • Ningshan Ouyang, 2nd Year: Back-end Developer

This semester, the project (along with all ETC projects) was completed remotely in response to the 2020 COVID-19 pandemic. The following is a post mortem analysis of what went well, what went not so well, what we learned, and the life of our work after the end of the semester.

Anyone reading this post mortem should feel free to skip around to relevant sections as opposed to read each section in order. This document’s primary purpose is to be useful and help future developers see what worked and what didn’t.

Our client Ana Rodriguez-Castillo expressed a desire to pursue funding for potential future development of the concept (thus the documentation package). Ideally, a future team should be able to read the documentation and continue development on its design themes. Additionally, the partners are seeking to hire one or two developers such as ETC students to continue updating the tool and performing bug fixes for any issues that arise from long-term use starting in January.

What went well 

Productive and invested client relationship 

The team enjoyed a productive and invested relationship with our client, Ana Rodriguez-Castillo. She was directed towards our ultimate goal of creating a tool that met the needs of our unique 5 partners and met with us often to be supportive of the team’s progress (even during more ETC requirement heavy weeks and while creating final deliverables).

Logistically, the client had an intimate understanding of our institution partners’ needs and comfort level with electronics.

She also worked alongside the ETC team to ensure we made the most impactful and long-lasting implementations to the VTC. We recognize the gift we were given to have a project coordinator solely dedicated to making this grant a success. She was involved and eager to help us problem-solve and it’s apparent that much of how the final product turned out is due to the diligent work she and the summer pre-production team did.

Thorough and systematic Playtesting

With playtesting being one of our top three most important metrics, we prioritized it. Our playtests towards the beginning of the semester were somewhat unfocussed and tried to test too many things at once. But we took a step back and reevaluated and made the playtests much more targeted and systematic after halves and the results showed clearer because of it. 

We were able to stay on track with builds being locked each Thursday and the following week we would playtest the previous week’s build before the next Thursday, while our developers worked on bug fixes or other features. We continued in this cycle till the VTC was locked and handed off to our partners.As a result of pre-production happening over the summer, and the level of fidelity we had to provide in the VTC prototype, we didn’t have time to add additional experience design before beginning development. Due to this, most of the time we had to skip the paper prototyping and Figma prototyping stage of playtesting and get features built to playtest them. We did playtest sooner than that whenever possible, but due to many of our playtesters also being museum partners, we didn’t have a naive guest pool of playtesters to draw from, and we had to ensure client confidence in the process so oftentimes we were unable to show prototypes for fear of them being perceived as unpolished.

Final product is received well by staff and administrators

The outcome that we look to the most as making this a success is the response from this tool’s users. We shared in our final presentation and in our team trailer some of the testimonials from them, including how pleased they are with the VTC’s functionality, the fact that we listened to their needs, and how needed this tool is for their work right now.

This can directly be credited to our client requirements metric as well as our playtesting metric. Keeping these goals at the forefront our decision-making process helped us have a guiding direction.

Clear and pressing problem meriting a solution

It is fortunate that we had such a compelling reason for our project. It wasn’t an experiment being made on a whim, not that such a project is negative in any way. But the merit of our project being necessary for 5 different entities in Pittsburgh made it a worthwhile project to pursue and provided some of the most pride about this project. Meeting that need felt worthwhile.

Preproduction package

The pre-production package the summer team provided us with was a helpful addition to get us started. Even though we didn’t get 100% familiar with all the materials they left us, we were able to get a great introduction to Ana and the 5 partners and start off in a good direction thanks to the preproduction item provided.

Comparable products were clearly understood and experienced

Design innovations that we later better able to articulate are: 

Easy reproducibility 

Due to the nature of stops in the VTC, it gives our partners a way to easily make tour templates and reuse stops with their branding, advertisements, and other information often used. This is a feature that is not so easily accomplished in PowerPoint without cumbersome screen switching.

Separated roles for better teamwork

Another feature that our clients appreciated that is unlike PowerPoint, is the ability to give hosts access to present a tour and at the same time, being able to restrict that host’s access to edit the tour itself. This benefits both parties, making them feel more comfortable and confident in the tool they’re using.

Simplified front-end design to only show what’s needed

Our developers spent a lot of time creating and polishing the front-end of the VTC to be as simple and easy to use as possible. One common criticism we got while making the tool was that people couldn’t understand what we were putting out time into. For our client Ana who saw the progress each week and was a part of the decision making process, it likely made sense. But to others, the minimalist look of the UI made it seem easy. 

In reality, there were two very big challenges we were spending a lot of time working through: 

  • Changing the WordPress Architecture: We attempted to circumvent hard coded aspects of WordPresses framework. And we were successful at first, but it took a long time and the farther we got into it, the more removing these things had unintended consequences and the more we found certain buttons we weren’t able to remove. We believe this could be done with more time devoted to it, but we got to the point where we had to remind ourselves why we chose the WordPress engine and if taking it apart so drastically was really the best use of our time.
  • Creating the Front-end of the VTC, going through iterations, and not as Web Developers: The other half of our time was spent developing a WordPress theme from scratch that worked optimally for our partners’ needs and constantly adding in new features, such as tour permission labels, new overlays, and making it responsive for different computer screen sizes. We also were going through iterations on the front-end design of this with each playtest, which takes time. Design and programming worked closely to develop ever improving forms of communication to get to the best iteration as soon as possible.

In the end, we are happy with the way we were able to explain in depth some of these iterations we made to better communicate the breath of what it took to simplify the VTC.

Strong development tools

Pantheon

Pantheon was a strong development tool for us primarily because it allowed us to develop on a free hosting trial for as long as we needed and it made versioning out new builds for playtests easy enough for non-developers to create them. This helped streamline our productivity.

Figma/XD

Generally we would have used prototyping software such as Adobe XD, but due to us being remote and XD not helping with that issue, we also sought new software. We settled on using Figma, as the summer team before us had begun using it and members felt comfortable picking it up. This proved to be useful considering it communicated some of the necessary CSS for our front-end developer to not have to search for the needed values, it has been a much requested proficiency in job roles, and it allowed the team to communicate effectively even while remote.

Remote work was mostly effective

The team was able to cope with working from home fairly well. It allowed some of our team members to be in places other than Pittsburgh to be closer to family, and it helped us to understand our partners’ desire to provide better virtual experiences. We made use of digital means of tracking things (e.g. Scrum boards, wireframes, meeting notes), and we had a regular meeting time every weekday. Dependencies were communicated through instant message on Slack, tasks were tracked on Google Sheets and Trello, and assets were transferred using Randon and Google Drive.

One challenge to remote work was that it took us a few weeks to adjust when a teammate’s schedule changed due to timezone. Once we were adjusted though, we found ways to use this time difference to our advantage. It gave us more time between cycles. On the whole, the team was typically able to maintain forward progress and meet deadlines during remote work.

What didn’t go well

5 client relationships

From the beginning of this project, a lot of time was spent in meetings with our client and at times with the partners. This, while useful to help everyone get to know each other, was detrimental to our progress on development. There were weeks where we spent more than half our time in a week in meetings, and that left us hardly any time to develop designs or builds. It was also taking up all of our time throughout the day leaving us to find time at night or 15 minutes here or there to work. It wasn’t productive.

It wasn’t until halfway through the semester that we were able to separate out responsibilities and people required at client meetings so it was easier for teammates to get work done. We struggled with wanting to please the different partners, especially in the first 6 weeks. Our team was trying to become web developers, make feature estimates, try to gauge which museums would benefit from which features, and all the while figuring out what made a virtual tour successful, and amidst all the meetings, little progress was being made.

Unfamiliarity/relative inexperience web development 

This proved to be a significant hurdle, as our team programmers were not web developers. They have some overlapping skills, but really they are different languages. and proved a great hurdle to remote development: only specific pre-registered devices can run the experience using testflight. While iPads are the premier tablet on the market, having to build the project on iOS increased build time and involved many iOS-specific steps before pushing the build to the iPad itself. Committing to iOS also meant that the game experienced weird glitches when switching platforms to Android.

Initial conflict in visions for the project

ETC Faculty 

The ETC faculty were often confused as to why this was an ETC project in the first place. This was a tough circumstance to be in because it often fuelled discontent on the team and kept the idea of making this project what the client didn’t want or need alive.

It was a fair ask, but we felt that our answer (that was the differences between it, PowerPoint, and Google Slides) was accurate, and constantly having to defend the point when the team wasn’t bought in was constantly opening old wounds. That being said, we hope the interviews at the end of the project have finally added some closure to that point.

ETC Students

Perhaps as a result of the 8 week pre-production, in which we didn’t get a say in the design of the VTC, the team lacked a feeling of ownership of the VTC from the beginning. There were also feelings of this not being what people had wanted to get done this semester for their portfolio, which is hard to come to terms with. 

We also lacked systematic and exact design documentation that had come to be expected, and that caused a rift early on, leading to mistrust for the process. Team members elected to be present for creative meetings rather than be notified later of the final design. And this was helpful and harmful in some ways. It was helpful because everyone felt like they had a say, but it was moreso harmful because decisions took longer to make, everyone was present so no one felt the need to take detailed notes, and there were so many people present that not everyone spoke up which often caused us to go back to points we thought were finished. 

Client  

During the first half of the semester we had some difficulties regarding meeting ETC specific deadlines. We believe this was partially due to imbalance expectations regarding the difference between the work-for-hire the summer team acted under and the ETC project role we were acting under. Our client didn’t always know what to expect from 16 weeks of work from an interdisciplinary team of graduate students, and sometimes that miscommunication was stressful. But as we got to understand each other better and learn more effective ways to communicate, this stress and uncertain feeling was alleviated and we ended up finding great ways to have the ETC requirements, like the team trailer, act as useful artifacts for our client as well.

Design Innovation vs. Efficacy: “How is this an ETC project?” 

While the team successfully understood the need for museum’s to have help presenting better virtual tours, we struggled in the beginning with turning that into a cool, fun and creative ETC project. It took us some time to understand the growth opportunities this project offered.

Luckily, our design pillars were well tailored to the needs of that project, but that mostly served as confusion during the first half of the semester when we were still figuring out what the long-term professional benefits were that we were gaining. As we came to understand our clients better and why the summer team made the choices they did, we began to use the design pillars more intentionally, halfway into the semester.

Lessons Learned 

Overdesigning wastes valuable iteration time. 

We had wanted to playtest more often, but we were concerned about the item we were playtesting coming across as “unpolished”. Due to this fear we often built tools that we thought would be helpful, but often-times they didn’t help the client because our client needed things more simplified.

Our lesson here was to playtest sooner anyway. An area where we did well here was finding playtesters similar to our partners that weren’t our partners. We tested with Rebecca since she also hosts tours of the TC, and that offered strong results. We would like to have done that more when designing solutions so we didn’t overdesign them. Alternatively, we would have our partners playtest unfinished designs to make sure we weren’t overdesigning the solution.. 

Pre-production work is valuable and should be utilized when tough decisions need to be made.

The summer team did survey work that we didn’t utilize, so we didn’t know our audience well enough. Our lesson from this is that pre-production surveys are valuable and should be utilized to make tough decisions.

A common issue during development was that we often didn’t feel like we had agency over the project or could change design decisions the summer team had made. If we were more familiar with the reasoning behind the summer team’s design decisions we wouldn’t have felt this way and we would have produced iterations quicker. There were plenty of excuses as to why we didn’t take the time to understand the research done sooner, but at the end of the day there are just a lot of excuses. 

The balance of using enough text to communicate the point, and breaking it up so as not to overwhelm is a hard one. We’re sure the summer team struggled with this, and we certainly did as well. We knew it would need to be very precise to be effective, so we began outlining early on, in week 5, and came back to it periodically. We didn’t start really writing from that outline until 3 weeks prior to delivery though. 

Writing could also be the entire job of one team member. Our team tried to disperse the responsibility of documentation, and that was helpful, but ultimately found most teammates focusing on their primary responsibilities before getting to documentation. We will be making ourselves available as much as feasible after this semester in case there is confusion around the documentation, but we’re sure we’ll get a lot better in our next documentation endeavor.

Our lesson learned here is to find a way to get through more of what the pre-production team came up with because they spent a lot of time on it and it helps us not do double work.

It is vital to take into account the target user’s technical fluency when designing a tool for them to use.

    We were intimidated by the user’s technical fluency so avoided really quantifying it. We needed to find the minimum so we could take that into account and build for it. Instead, we built for a role that we knew had a higher technical fluency and could teach the other role. 

It wasn’t a harmful decision, as this is no different than how these two roles interacted with each other before the VTC. But, it did waste a great opportunity to create something to empower the lower tech fluency role so they could use it on their own, without instruction, and free up the higher tech fluency role to accomplish even more with the time they’d save. Our lesson learned here is that if it is in scope, we should seek to design for the lowest technical fluency user first so that we are not barring any users from the tool, just because.

Last Steps and Wrap up for the Project

The remaining steps for Museum.Live are to conclude launching the final three out of five partner’s websites. We have included documentation, an updated educator tutorial video, and completed a handoff meeting with all of  the partners. The documentation includes common breaks and bugs, accessibility tips, instructions for the tool, instructions for future development, and recommendations for future development. The partners intended to begin giving virtual tours starting in January 2021.  

Our client and partners have expressed a desire to pursue funding for future development on the concept. Ideally, a future team should be able to read the documentation and continue development on its design plans. Additionally, the partners are seeking to hire one or two developers such as ETC students to continue updating the tool and performing bug fixes for any issues that arise from long-term use starting in January.