SpeakEasy

One of the more unique challenges presented by a game where the player can form their own sentences is not only how easy it is for the player to mess up their sentence, but how many ways there are to do it!  Everything from wrong-gendered pronouns to disagreeing pronouns and verbs to incorrect word order to valid sentences but wrong content…

The way we handle this is with language parsing.  We actually break down the player’s answer into its words, look at their parts of speech, and figure out what properties they have (gender, number, and person are the properties we care about in our prototype).  We do the same for the known correct answer – we now have syntax token lists.

From there we launch into a series of structure comparisons, gates by which we determine what kind of wrong the player’s answer is.  Depending on how their answer is syntactically incorrect or differs from the expected answer, we’re able to deliver a clear and targeted pre-authored response, which is customized based on the words and sentence in question.  A diagram of the process (for the sample incorrect answer of “Ella la cocinan”, where “cocinan” does not agree with “Ella”):

The way that we encode the property information is completely extensible – it’s a seamless process adding new properties such as tense, voice, etc., because part-of-speech analysis and agreement is completely data-driven.  The next step for this parsing system to make it applicable to hypothetical future exercises that delve into more advanced Spanish territory would be to encode grammar in a more formal way and represent syntax as a tree to account for clauses, but that’s getting into territory where 3rd party frameworks would become worth investigating.

Our input system isn’t hard to learn – neither are the bite-sized Spanish concepts we introduce in our exercises.  But if you were to present players with our icon interface and ask them to play, they wouldn’t have a clue what to do – this is inherent in the design of Minspeak, where we favor the expressiveness of the icon set over the initial guessability.  In much the same way, if we asked players to read text or view a clip about how to use the icons, their eyes would glaze over.  The way Minspeak is taught in a disability context is through coaching the player and having them work through small examples – in the digital world, the analog to this is an interactive tutorial.

Our tutorial is one of the most sensitive areas of our prototype – any failings in the tutorial lead to the player either being unnecessarily frustrated or completely stymied in the game.  Careful tutorial design and iteration was essential for achieving the success rates we’ve seen – to support this, we created a data-driven state machine-based tutorial system that lets our actual transition logic mirror the flow presented in our tutorial design.

The pseudo-code in the diagram above demonstrates the Tutorial States are specified declaratively entirely in terms of the actions they take and their transition logic.  One key we found to maintaining a sane architecture is ensuring that no state is shared between entrances to a Tutorial State, as they are inherently re-entrant.  Java doesn’t support anonymous functions or closures (“lambdas” in some languages), but its facility for anonymous subclasses allows us a slightly more verbose alternative – we use these to provide a method to generate a new State every time we need that state.

Another point of note about the architecture of our Tutorial system (in particular the state machine architecture we built to base it on) is the gradual shift we went through from deep inheritance hierarchies to aggregation.  Specifically we found that the types of actions and effects that Tutorial States have (like posting messages, showing arrows, changing animations, adjusting UI elements) tend to be very common ones that can be shared across many states – instead of specializing it makes much more sense to simply attach the ones we need.  At the same time, inheritance served us well even after our architectural shift, in that we could bundle common effects and transition logic in a way that supported easy future modification.

We’ve created a general-purpose rendering library on top of jPCT and raft’s Bones extension that makes it dead simple to load and display skeletally animated models on Android.  The ETC’s “7” project team already has plans to use our EasyBones library for an Android augmented reality game.

Features include skeletal animation (loading OgreXML format), picking, billboards, and alpha-blended textures.  We’ve also created accompanying tools that make it simple for artists to view their OgreXML exports in-engine, and a specific Speakeasy tool that we call the SpeakeasyPackager that simplifies our asset pipeline and allows us to reduce our Android package size drastically.

After deciding on jPCT as our underlying 3d engine, we had a lot of work remaining until we could robustly use it in our gameplay code.  jPCT isn’t super-low-level, but it’s also quirky and under-documented, and we needed another layer (or two) on top of it in order to be able to concisely load models and have them display, animate, and transform properly.  Further work was necessary to add the ability to load texture data to raft’s modified version of jMonkeyEngine’s OgreXml importer, as well as to transfer that data into his Bones importer.

We’ve been using our rendering library successfully in our own prototypes.  Since our character appearances are data-driven from a spreadsheet, our SpeakeasyPackager only packages up the textures specifically referenced, helping shave 8mb off of our package size (right now about a 75% savings).  It’s also eliminated a number of classes of difficult asset bugs by standardizing flow of data (we run the packager whenever we sync new assets from Perforce, and it clears out the old package and refreshes it with newly converted models and textures).

Although on general principal we recommend Unity for teams looking to develop 3d games on mobile platforms, all 3 ETC teams developing 3d Android games this semester have required access to native Android functionality that Unity prevents.  Developing a bridge plugin for Unity to make Android API calls is technically possible but no team that’s attempted it has had success.  We’re extremely happy with our choice to build on jPCT, and we think our work will help make life easier for future teams that choose this route.

Original size 128×128

These diagrams break down the character structure.

The paper-doll style allows for alternating the characters to easily create more characters without having to rig and animate for each.

The next image shows the locations of the characters for the first exercise. The bones on the ground are accessed by the programmer to connect our characters to the scene.

The next part for the game will focus on the shell component and requires creating the animation set-up for the baskets and ball.

All character assets are ready to go!