Video of Pixelbending in action, inspired by martial arts from Avatar: The Last Airbender and Legend of Korra.

Many Kinect games that use martial arts in a magical setting use something like this: If you throw a punch, a fireball comes out. Throw another, and the same fireball comes out. What if you could change that fireball or lightning bolt by how you moved?

For these nuanced gestures, we began to be inspired by media that used martials arts in such an approach. In “Avatar: The Last Airbender”, martial artists manipulate elements like water, fire, air and earth. Their martial arts form flow from one gesture to another continuously to change whether water turns to ice, the direction of the water, and how it behaves.

How can we incorporate the nuanced, continuous gestures from martial arts, into a novel and fun interaction in a VR video game environment?
Role: Gesture Designer/recognizer developer, co-developer, co-researcher



  • Design 3D interactions(hand, arm, and body movements) for:
    • continuous, nuanced gestures
    • martial arts-like movements
    • Fun: incorporate video game elements
    • Broad Access: Portable, off-shelf tech


  • How do we design continuous, nuanced gestures into full-body interactions in a video game?


Our target audience was a public open house from the university research center, where participants
could interact with our demo in an informal, walk-up-and-play format. Roughly 30 individuals attended the open house, ranging in age from young children to older adults and covering the gamut of technical/domain/martial arts knowledge.


Images of scroll outlining Tai-Chi inspired waterbending from Avatar: The Last Airbender (c) Nickelodeon.

We chose Tai-Chi as a safe, gentle martial art, under recommendations from a subject-matter expert(SME); we did not strive to be a dojo replacement, but rather, be inspired by martial arts.

We used Karen Holtzblatt’s “what makes things cool?” paper as our guideline for what is coolness.

As for access, we deliberated between using the Microsoft Kinect, the Playstation Move controllers, or both. The dual setup granted us far better tracking of the hands and may give users a sense of holding a staff.

When it came to the hands, back then, the Kinect alone tracked the wrists. We anticipated that when the Kinect had better hand tracking, we could track open and closed palms, and in Tai Chi, these gestures are crucial. Thus, we chose a Microsoft Kinect-only setup, in being true to Tai Chi.


Gesture Designer/recognizer developer, co-developer, co-researcher in a team of 2


2 months


  • Microsoft Kinect
  • 3D TV
  • Unreal Development Kit
  • C#


  • Design proc
: Gesture tree showing all possible gesture combinations. Water manipulation comprises gathering the water (gather),
manipulating it (freeze, compress), and releasing it (wave, water ball, two-handed press and water jet).

We designed the gestures to be dynamic and seamlessly flow from one gesture into another. While we expected beginners to pause for a brief moment to recall the next gesture, we ideally wanted the player to perform these movements smoothly without pause. We also continuously consulted with a Tai Chi SME to assure our gestures are true to the art and safe for as many people to try as possible.


Three gestures show how a water jet works; the three gestures occur continuously.

The nuanced, continuous gestures in our system were also well received. When they were able to perform the gestures successfully, players felt in control of the water in a more fine-grained way than with traditional gesture controls. The interaction was described as feeling like magic.

Nearly every user was able to perform at least a subset of the nuanced gestures with a small amount of coaching or simply by watching

Many users were able to master the whole gesture set, sometimes even figuring out sequences themselves without instruction. Further, the users nearly universally enjoyed the experience, providing enthusiastic feedback despite technical

Even when the system misclassified a gesture, still reported enjoying interesting interactions with the system.

On occasion, when a user intended to perform one gesture, the system would sometimes recognize it as a sibling gesture in the gesture tree. For example, a freeze gesture might be interpreted as a release.

The users’ actions were still mapped to changes in the water they were manipulating, and they seemed to enjoy even the misrecognized gestures.


This project showed me how powerful fun can be in design; even when the system malfunctioned, the vast majority of participants praised the system, even using terms like “magic”. I was surprised that the fun factor helped make glitches enjoyable.

In addition to doing more controlled studies to evaluate the interactions, we got feedback on many venues, including the Youtube comments section, to generate a turnkey, easily downloadable solution. This project was bound to constraints in academia, and more often than not, turnkey, downloadable solutions are not common. Butm I would be happy to partner with an organization to make this available for all.