Week 8: Evaluate

This week, we evaluated the effectiveness of the experiences that we are building by performing a playtest and conducting user interviews. Because the results of user interviews have not yet been compiled, I will spend this blog post discussing the playtest.

The playtest was broken up into two portions: one for the app, and one for the inter-inning event.

  • App Playtest
    • Guests watched one inning of a pre-recorded baseball game, without the app.
    • Then, guests were given phones with the app and asked to make predictions about what would happen the following inning.
    • Finally, guests watched a second inning of the same game with the test phones in hand, and received notifications when their predictions came true.

Throughout the test, it became clear that guests were more engaged with the baseball game during the second inning, when they were watching the game with their predictions in mind and getting rewarded for making correct predictions. This means that the prediction mechanic is sound at its core, which is very encouraging. However, it does not mean there are no areas for improvement. We discovered a number of technical bugs during the test, which our programmers will be working to fix in the coming week. Additionally, we found that guests were confused by the layout of the prediction screen. Going forward, Eric and Richard will work together to mock up another version of the screen, which presents fewer options (so as to not overwhelm the user) and indicates visually which events are likely or not likely to occur.

  • Treasure Hunt Playtest
    • Chairs covered in sticky notes were spread throughout the RPIS to match the different colored markers on the guests’ treasure hunting screen.
    • One guest from each audience section volunteered to be their section’s “wanderer” – with the exception of an audience section that did not have enough people to spare a wanderer. (Ramya volunteered to be their wanderer. Thanks Ramya!)
    • The wanderers walked around the RPIS, and guests were asked to use their phones to indicate whether they were getting “warmer” or “colder” in relation to the place where they were supposed to “plant a flag.”

Like the app playtest, the treasure hunt playtest was successful in that guests were clearly engaged in the game. But also like the app playtest, the treasure hunt playtest showed that there are some major problems in the game’s current iteration. For example, some guests pushed the “warmer” and “colder” buttons on their phones a few times, but, after receiving no feedback as to whether or not the buttons were working, assumed the buttons were broken and stopped pushing them. Instead, they simply yelled directions to their wanderer. While we don’t want to discourage people from yelling, since yelling is a form of engagement, we also do not want people to abandon their phones entirely – so we are working on another iteration of the treasure hunting game that will hopefully prevent guests from giving up on the buttons. If all goes well, we hope to test the new iteration this week.