by Seann M. Dikkers
How do you translate a great model
of fun learning into an approved
research project for publication?
How do you translate a great model of fun learning into an approved research project for?
RTR by itself can stand alone as an engaging model of practice in learning environments. Researching interesting questions has long been an entry into learning not only about the topic at hand, but about the practice of research itself. At a third level, those that enjoy research and the process of discovery are often leaders in their respective fields. RTR is a fun entry into practice on this level and we hoped to capture these experiences in the book you are reading now - not only so you could enjoy the work itself, but the process, practice, and even your own use.
In fact it is our hope that you’d enjoy the work here so much that you would use RTR as a way to bring playful rigor to your learning environments, classes, and practice. Use RTR to test new ideas that may be worth further study, build concepts, methodologies, and research team cohesiveness.
To share the work of RTR, as a learning tool, it was important to pursue good standing with the Instructional Review Board (IRB) and those that review research for the university. Simply conducting an RTR session in a class or business environment wouldn’t require any of this, but to publish we needed IRB approval. IRB’s have been in place to protect the institution, but also to protect the researchers and the fields they represent.
Each IRB is unique to it’s institution so your work getting approval will essentially be local. Attached here is the language that we used at the University of Wisconsin - Madison after meetings, suggestions, and the help of the IRB panel. Below you’ll find the entire submission for your use and as a working point for any IRB work you may do. If you are trying RTR informally, without intent to use the data beyond course credit, you may want to skip to the second part of this chapter. If you want to be prepared for publication, then you are welcome to use it as a starting point for building your own IRB submission for research.
We found that making initial contact with the IRB provided the opportunity to share and connect on the vision with representatives ahead of time. Setting up a time to look at the submission provided invaluable insights into the importance of good review, how to structure our submission, and also gave us a strong communication line along the way. Your local IRB will have different levels of accessibility however and these initial meetings, though useful, are not essential.
When done, approved, and you are able to broaden RTR work for publication, this book always has room for a few more good projects.
Submitted to UW-Madison IRB:
Real–Time Research (RTR) is a conference workshop held at professional games (and learning) conferences – specifically, the Games, Learning & Society Conference (GLS, Madison WI), the Game Developers Conference (GDC, San Jose CA), and possibly the Digital Games Research Association Conference (DIGRA, London). The GENERAL PURPOSE of this project is (a) a learning opportunity for participants less experienced in successful interdisciplinary collaboration among academics, designers, and educators, (b) to provide a venue for piloting new research questions or replicating known ones, and (c) to provide a new and rich venue for a learning experience at these conferences. We accomplish this through a two-part workshop involving game researchers, game designers, and other professionals in game-related fields attending the host event. RTR attendees participate in a workshop at the beginning of the host conference to collaboratively design, on (typically 5-12) cross-disciplinary project teams of 5-8 individuals, separate research projects that are conducted over the course of the host conference itself. Groups identify a theory of learning and methodology to frame their project, generate a research question, and then gather the necessary data from fellow conference attendees during the host event. After data is collected, they reconvene in a second workshop at the end of the host conference to debrief on the feasibility of their methods and processes and to share their findings in the form of a 5 minutes public presentation on their project and a short “chapter” in an online RTR book to be published with ETC press. Our interest is in the project group work and the process of designing research itself. It is the final debriefs that we collect and analyze for RTR publication. The project reports in the second session become the target of study and a form of data used to write our reflections on the designs and processes employed – much like professors respond to class projects and write about lesson design referring to them. We are also interested in the refinement of the RTR process over time and how it evolves through much iteration.
STUDY DESIGN & METHODS
The inclusion criteria for participants in RTR research projects are purely voluntary. Adult professionals who are already attending the host event (GLS, GDC, or DIGRA), and furthermore select this session to attend, select themselves by taking the workshop. No underage minors are involved and no special groups are targeted in any way, nor is any personally identifiable or sensitive information kept. Because these conferences are conducted in English, all participants would have adequate English fluency. All participants have the option at any time to simply enjoy the rest of the conference without further participation.
Number of Participants
10-30 participants attend the RTR sessions and form projects. For their projects they have access to other conference attendees ranging from 300 (GLS) to 3000 (GDC).
Each project group varies in the number of cases it is willing and able to involve. This number varies as necessary depending on whether the project involves, for example, observation of participants using a specific game interface (15 minute protocol), a short interview (5 minute protocol), or a series of Likert scale questions (1 minute protocol). Longer protocols involve fewer participants given the nature of this workshop and the fact that data collection must only be done within the time constraints of the host conference. We estimate that, at most, 200-400 persons at each host event would participate in any form of the projects. This however is secondary to the core of the project, which is the smaller number of participants in the workshop and part of the RTR work.
Role of Participants
Each RTR interdisciplinary teams will participate in their projects as they see fit and these roles will vary. For the first session they are planning their projects, they carry them out during the conference, and at the second session each group of participants shares out on their project. These reports are the target of this IRB. We would ask them to write a complementary report on their projects and use this for our analysis and interaction with the data they collect.
So far involving conference attendees has included responding to short interview questions about game play preferences, answering Likert scale items about videogames and learning, briefly playing a game title at the host conference under observation, or agreeing to submit one’s online twitter streams for analysis (with identifying information removed). All of these interactions are studied in public settings at the host conference venue and are engaged for as briefly as possible so as to minimize disruption of their professional event while maximizing the number of cases that can be included. We provide a guide (can and can’t do list) to our participants, including a script that is attached to the IRB, that instructs them to state their name, project, how they selected the person, risk/benefit, voluntary nature of the work, and that no personal information will be kept.
We set up strict rules for the projects. No deception is involved, no identifying or sensitive information is collected (not even names), and no topics are raised that could in any way be embarrassing, diminishing, or deleterious in any way to participants (i.e. nothing transgressive, sexual, embarrassing, or unduly personal such as intimate feelings and relationships toward others or oneself). Participation needs to be entirely voluntary and, before any data is collected, oral consent is obtained and individuals are reminded that they can cease participation at any time. With consent, images, audio and video are, at times, recorded as part of data collection but only for record keeping and analysis with no such identifying data shared in any public venues either written or face-to-face.
The only compensation given for participation is a small sticker which reads “RTR – I subjected” for the individual to place wherever they like (e.g. their conference badge, notebook, or computer) or not. Consenting project teams can have their work be a case used and published as an RTR outcome.
The RTR workshop is held at three host events, all of which are professional games (and learning) conferences: (1) the Games, Learning & Society Conference (GLS) held annually in Madison WI, (2) the Game Developers Conference (GDC) held annually in San Jose CA, and (3) the Digital Games Research Association Conference (DIGRA) held this Fall in London (optional if international regulations would complicate the IRB process).
Does the study involve participants from places other than common public spaces?
The measurement procedures to be used in this study vary depending on the nature of research questions developed by each project group. Our observation of the groups at work in addition to the final group reports give the core information for reflection on the work, design of the study, and follow up questions. After we verbally share our thoughts about the projects, groups are invited to write up a summary and reflection piece about the experience with a template for consistency. These write ups along with our commentary make up the core of the research. Therefore, when the groups report back at the second session, we will record and keep records of the findings they present. We collect the slide shows they used and written reflections of the project along with our reflections and feedback on the projects.
Will any of the following be used as part of the study: questionnaires, measurement instruments, interview protocols, or a description of topics or an approximate script?
NOTE: Yes, but because the exact instruments will not be developed until the actual RTR workshop, we have no detailed measures to include with this protocol at this time. There are no instruments formally developed for the participants in the sessions, only ones they may create and use.
Verbal consent will be attained with any participants. Handouts will outline this process. (see attached)
Are there risks to the participants?
Steps to Minimize Risks
We minimize risk by clearly outlining and providing a written guide to constraints for the session projects (much like in a classroom setting). This includes not collecting any identifying information (including names and institutions) of any form and eschewing discussion of any topics that could in some way pose personal, social, material, or political risk to the participants.
Any digitally identifying information is immediately purged from the data corpus before analysis, and any images or audio or video that is collected as part of the research is not shared publicly either through presentation or through inclusion in any written products of this work. If any group were inspired by the designed projects at the conference, they would need to submit separate IRB’s and replicate the research for any separate publication/s.
Any info that is stored concerning the group presentations and findings will be filed and stored in an external password protected hard-drive kept by the PI’s on this project.
Medical or Professional Intervention
Possible Benefits to the Participants
The possible benefits to participants are both immediate, short term and long term. First, because many professionals in the games industry are also game players and avidly interested in their own learning processes (as well as the processes of other players), one immediate benefit from participation is simply the opportunity to talk about investigation, meet colleagues in the field, create a collaborative project, and have an authentic assessment in the presentation of their work.
Second, because we share our general findings at the end of the conference that participants have chosen to attend, they have the opportunity to immediately see the outcomes the work that their participation has made possible. Oftentimes these aggregated findings provide an interesting context for reflection on one’s own views.
Finally, because subjects are academic and industry professionals in the field of games (and learning), the findings of these small pilot studies are of immediate benefit to participants professionally in that they add to our collective knowledge about this new emerging field. RTR workshops provide a venue for exploratory and educational collaboration on research topics of interest across disparate disciplines. In our experience so far, individuals who have participated have had overwhelmingly positive things to say about both their personal involvement and they value they feel it brings to our profession. Many subjects wear the “I subjected” stickers with pride and encourage others to volunteer because participation is seen as both informative and fun.
Benefits to Society
RTR workshops provide scholars and designers in the field of games and learning an opportunity to work together on interesting questions and pilot attempts to answer those questions with minimal investment of professional time and resources. It fosters conversation across domains, which in new fields of study in particular, is especially important. As “games and learning” becomes an ever increasingly popular topic of academic and public interest, innovative hands-on educational workshops like RTR can help stave off the disciplinary “silo’ing” so detrimental to forward knowledge by fostering conversation, collaboration, and the exchange of ideas across areas that otherwise not in conversation.
To date, RTR has been a very big success as an educational workshop; our main goal in formalizing the research through IRBs now is to enable us to insure that our handling of data involving participants is appropriate so as to enable broader distribution of our methods and findings in the form on an online book through ETC press (who came to us with an offer for publication given RTR’s strong reputation and success).
Products/materials used in the study:
1. RTR: Research guide and consent script
2. RTR: Follow up (consent for participation)
3. RTR: Report (guide)
4. RTR: Cards
5. RTR: Supplies
The IRB process required the initial submission and edits based on follow up from the committee, two panel members consulted with us and helped guide those revisions. Finally, we got the approval for RTR and were able to take the RTR projects as data for publication.
Special Notes or Instructions: After discussions between [IRB representatives] and the research team, this protocol has been submitted. [IRB representatives] have determined that the research team has done an excellent job in addressing any IRB concerns. Therefore, this protocol is determined to be exempt pursuant to 45 CFR 46.101(b)(1).
Preparing for the session
The first and primary requirement for RTR is your understanding of how it works, and being excited about iterative research. In the previous chapter Eric and Kurt laid out the premise and basic design of RTR. Beyond this, many practitioners would be comfortable running with it and making adjustments on the fly. Your design choices will customize and make RTR come alive in your setting.
For others, you may be asking for more detail and a look inside the ‘on the floor’ implementation. Here is a short but useful ‘to-do’ list of sorts. As the IRB work is simply laid bare, below are the lists and notes we built over time to make sure everything was in place.
During the RTR sessions we kept track of both ideas for the future and needs that emerged for the sessions. With each of the three iterations this document became more refined and useful for our practice. Moreover, we can easily share it with you.
• Emphasize that RTR needs should go through graduate students, not
• Have some example studies that we present in our intro
• Customize the card deck each year.
• power strips (for many active laptops)
• set of cards - color glossy printing on card stock
• keynote presentation w/ samples of RTR and template for groups
• templates of data collection tools
• sample consent scripts
• group information forms (to record contact information)
• “do’s and don’ts” of research
Booth or Handy Resources for RTR teams:
• big sign (“RTR: Real Time Research w/ GLS logo) Lamenated 2 - 2’ x 3’
• internet connection
• e-mail address for RTR (so folks can send files to have printed)
• clipboards (10)
• paper (1-2 reams)
• stapler, markers, pens, notepads, string, tape, portable file/organizer, file
folders, easel w/ sheets of paper
For next year:
• full time RTR grad student (or two) w/ parking passes (for supply runs)
• access to copy machine
• separate table, close to registration
• set up RTR forum/wiki/type thing for folks to stay in touch if wanted
• video cameras
• digital cameras
• tag boards/White boards w/ easels/public wall space
• add a panel of judges to the final presentation & give out awards for various categories
• create a new category of cards called “material constraints”
At the end of this book, you’ll find everything else you’ll need - including cards, handouts, and templates. Use them all as starting points for your own RTR project.
The RTR projects
Before digging in, enjoy a few samples of work. Our collection of RTR alumni are excited to present their findings and analysis of their RTR projects. Enjoy them for the interesting investigations that they are and use them as case studies of RTR in action and get to know RTR from the student’s perspective so you can move towards your own use of RTR.
You should know a few things before you read on. First, these groups were invested enough in a few days of collaboration to return to this writing months later. Second, the value of sharing, writing, and working together were all the incentive available. Often the topics included here were for fun, but not necessarily in line with their research work professionally. Finally, the project chapters you’re about to read were written by very diverse teams. What may appear somewhat consistent in method is actually a combination of researchers, game designers, students, teachers, administrators, and technology specialists - and I suspect some closet artists are included too. These are professionals from a few different walks that have set aside time, energy, and a bit of love to share a few days of their “play” with you.
Use these as examples for your own practice. If done right, this is the sort of work possible. More exciting is that your learners will probably improve on these. Your iterations will add to what RTR is now.
With ETC Press we had the chance to offer the RTR groups a chance to write and share their work. Many did just this. Whether or not you are working towards publication, the process of writing and analysis over time extends the learning for those involved. This sort of revisiting of the work also extends the initial relationships built by those involved.
In order to engage participants in a writing reunion of sorts, I simply sent out e-mails to the groups and invited them to participate. At the end of this book you’ll find the template (Tool 6, page _) that was attached so writing could be consistent. This template also made the process more accessible because the effort only required ‘editing’ their presentations from the RTR sessions into a more formal context.
Invariably the groups saw this as an opportunity to add in what they didn’t have time for when we ran the sessions. Groups took the time to cite the writers that influenced their inquiry thread and methods. In addition, it was exciting to see what a new look at the data produced. Most groups used the comments from our experts at the conference to revise their work in writing.
Once the groups had a working chapter put together, the drafts were presented to Constance, our expert reader, for another round of suggestions and edits. Groups cleaned them up and submitted what you see here.