Many Kinect games that use martial arts in a magical setting use something like this: If you throw a punch, a fireball comes out. Throw another, and the same fireball comes out. What if you could change that fireball or lightning bolt by how you moved?

For these nuanced gestures, we began to be inspired by media that used martials arts in such an approach. In “Avatar: The Last Airbender”, martial artists manipulate elements like water, fire, air and earth. Their martial arts form flow from one gesture to another continuously to change whether water turns to ice, the direction of the water, and how it behaves.

Methods

Participants

Our target audience was a public open house from the university research center, where participants
could interact with our demo in an informal, walk-up-and-play format. Roughly 30 individuals attended the open house, ranging in age from young children to older adults and covering the gamut of technical/domain/martial arts knowledge.

---

Design

We chose Tai-Chi as a safe, gentle martial art, under recommendations from a subject-matter expert(SME); we did not strive to be a dojo replacement, but rather, be inspired by martial arts.

We used Karen Holtzblatt’s “what makes things cool?” paper as our guideline for what is coolness.

As for access, we deliberated between using the Microsoft Kinect, the Playstation Move controllers, or both. The dual setup granted us far better tracking of the hands and may give users a sense of holding a staff.

When it came to the hands, back then, the Kinect alone tracked the wrists. We anticipated that when the Kinect had better hand tracking, we could track open and closed palms, and in Tai Chi, these gestures are crucial. Thus, we chose a Microsoft Kinect-only setup, in being true to Tai Chi.

We designed the gestures to be dynamic and seamlessly flow from one gesture into another. While we expected beginners to pause for a brief moment to recall the next gesture, we ideally wanted the player to perform these movements smoothly without pause. We also continuously consulted with a Tai Chi SME to assure our gestures are true to the art and safe for as many people to try as possible.

---

Insights

The nuanced, continuous gestures in our system were also well received. When they were able to perform the gestures successfully, players felt in control of the water in a more fine-grained way than with traditional gesture controls. The interaction was described as feeling like magic.

Nearly every user was able to perform at least a subset of the nuanced gestures with a small amount of coaching or simply by watching
others.

Many users were able to master the whole gesture set, sometimes even figuring out sequences themselves without instruction. Further, the users nearly universally enjoyed the experience, providing enthusiastic feedback despite technical
issues.

Even when the system misclassified a gesture, still reported enjoying interesting interactions with the system.

On occasion, when a user intended to perform one gesture, the system would sometimes recognize it as a sibling gesture in the gesture tree. For example, a freeze gesture might be interpreted as a release.

The users’ actions were still mapped to changes in the water they were manipulating, and they seemed to enjoy even the misrecognized gestures.

Reflections

This project showed me how powerful fun can be in design; even when the system malfunctioned, the vast majority of participants praised the system, even using terms like “magic”. I was surprised that the fun factor helped make glitches enjoyable.

In addition to doing more controlled studies to evaluate the interactions, we got feedback on many venues, including the Youtube comments section, to generate a turnkey, easily downloadable solution. This project was bound to constraints in academia, and more often than not, turnkey, downloadable solutions are not common. Butm I would be happy to partner with an organization to make this available for all.