Gesture Tutorial

<< Back to the TouchID Toolkit page

In this tutorial you learn how to use gestures in an application.

Therefore, we will implement a gesture training application together. A label in the center of the screen will show the name of a gesture and a handpart. The gesture is randomly picked from the gesture repository (My Documents/GroupLab/TouchID Toolkit/Gestures). The handpart is randomly picked from the 15 key hand parts on a Fiduciary Glove.

The user then has to perform the displayed gesture with the displayed hand part in order to move on to the next random gesture and hand part.

Prerequisite 1: At least one glove configuration has been created

Glove configurations can be conveniently created with the GloveConfigurator which stores the glove configuration files also directly in the glove repository (My Documents/GroupLab/TouchID Toolkit/Gloves) right away. If you have glove configuration files in a different folder and want them to work with this example, copy them into the glove repository.

Prerequisite 2: A set of gestures has been created

Gestures can be conveniently created with the GestureConfigurator which stores them also in the gesture repository (My Documents/GroupLab/TouchID Toolkit/Gestures) right away. If you have gesture configuration files in a different folder and want them to work with this example, copy them into the gesture repository.

Step 1: Creating a new TouchID Application

For detailed information, please read: How to start a new TouchID Application

The TouchID Application template creates the following code skeleton for you:

public partial class Window1 : TouchIDWindow{
public Window1(){
InitializeComponent();

//Creates a glove for every glove model in the glove repository
this.LoadHands();

//Alternatively you can also instantiate glove by glove;
//Hand hand = new Hand(this, "Hand1");

//Creates posture recognizer and loads all postures from the posture repository into the recognition engine
this.PostureRecognizer = new PostureRecognizer();
this.PostureRecognizer.LoadPostures(PostureRepository);

//Creates gesture recognizer and loads all gesture from the gesture repository into the recognition engine
this.GestureRecognizer = new GestureRecognizer();
this.GestureRecognizer.LoadGestures(GestureRepository);

this.HandpartDown += new EventHandler<GloveTouchEventArgs>(Window1_HandpartDown);
}

void Window1_HandpartDown(object sender, GloveTouchEventArgs e){
Console.WriteLine(e.Glove.Hand + " " + e.Handpart.Identifier);
}
}

Step 2: Deleting unnecessary parts of the template code

Since we do not use hand parts or gestures in this example, we delete the code for the posture recognizer, the HandpartDown event and the corresponding callback method Window1_HandpartDown.

Step 3: Adding a label for displaying the name of the randomly picked gesture and hand part

// Center label for the name of randomly picked gesture and hand part
private Label centerLabel;

public partial class Window1 : TouchIDWindow{
public Window1(){

InitializeComponent();
this.LoadHands();
this.GestureRecognizer = new GestureRecognizer(this);
this.GestureRecognizer.LoadGestures(GestureRepository);

this.centerLabel= new Label();
this.centerLabel.Width = 400.0;
this.centerLabel.Height = 50.0;
this.centerLabel.FontSize = 25.0;

// Position label in screen center
double centerX = (this.Width - this.centerLabel.Width)/2.0;
double centerY = (this.Height - this.centerLabel.Height)/2.0
this.centerLabel.RenderTransform = new TranslateTransform(centerX, centerY);

this.Container.Children.Add(centerLabel);
}
}

Step 4: Adding a instance variable that stores the currently picked gesture and hand part

private Gesture WantedGesture;

// We use Handpart.Identifiers instead of Handpart because we want to store only the general hand part (i.e. just index finger)
// and not a particular hand part on a particular glove (index finger on the right glove of user Bob)
private Handpart.Identifiers WantedHandpart;

Step 5: Creating a method that sets a new random gesture and hand part and displays both in the center label.

private void SetRandomGestureAndHandpart(){
Random random = new Random(DateTime.Now.Millisecond);
this.WantedGesture = this.GestureRecognizer.Gestures.ElementAt(random.Next(0, this.GestureRecognizer.Gestures.Count));

Array handparts= Enum.GetValues(typeof(Handpart.Identifiers));
this.WantedHandpart = handparts.GetValue(random.Next(0, handparts.Length));

this.centerLabel.Content = this.WantedGesture.Identifier + " with " + this.WantedHandpart;
}

Step 6: Calling the SetRandomGestureAndHandpart() method in the constructor

This way, a first random gesture and handpart are picked for the start of the training. Make sure to call this method after the GestureRecognizer has loaded the gestures!

So far we have an application that shows a randomly picked gesture and hand part in the center of the screen. What we need now is a mechanism that changes the label text to another randomly picked gesture and hand part when the currently wanted gesture has been performed with the wanted hand part.

Step 8: Subscribing to the GestureRecognized event

Therefor we subscribe to the GestureRecognized event. This event is raised whenever a gesture has been performed.

// Center label for the name of randomly picked gesture and hand part
private Label centerLabel;

public partial class Window1 : TouchIDWindow{
public Window1(){

InitializeComponent();
this.LoadHands();
this.GetureRecognizer = new GestureRecognizer(this);
this.GestureRecognizer.LoadGestures(GestureRepository);

this.centerLabel= new Label();
this.centerLabel.Width = 400.0;
this.centerLabel.Height = 50.0;
this.centerLabel.FontSize = 25.0;

// Position label in screen center
double centerX = (this.Width - this.centerLabel.Width)/2.0;
double centerY = (this.Height - this.centerLabel.Height)/2.0
this.centerLabel.RenderTransform = new TranslateTransform(centerX, centerY);

this.Container.Children.Add(centerLabel);

this.SetRandomGestureAndHandpart();

this.GestureRecognized += new EventHandler<HandGestureEventArgs>(Window1_GestureRecognized);
}
}

Step 8: Adjusting the callback method of the GestureRecognized event

If the callback method wasn't automatically created, do this manually. Then we adjust it, so it checks whether the recognized gesture and hand part matches the wanted counterparts or not. If so, the SetRandomGestureAndHandpart() method is called and the application switches to the next randomly picked gesture and hand part.

void Window1_GestureRecognized(object source, HandGestureEventArgs e){
if(e.RecognizedGesture.Identifier.Equals(this.WantedGesture.Identifier)){
if(e.Handpart.Identifier.Equals(this.WantedHandpart.Identifier)){
this.SetRandomGestureAndHandpart();
} else {
Console.WriteLine("Correct gesture, but wrong hand part.");
}
} else {
Console.WriteLine(e.RecognizedGesture.Identifier + " is not " + this.WantedGesture.Identifier +".");
}
}

The HandGestureEventArgs contain also a list of all gestures that were recognized (e.RecognizedGestures). This way you can also access gestures that have been recognized with a lower confidence as the best result that is exposed as separate value (e.RecognizedGesture). The recognized gestures provide not only their identifiers, but also the recognition confidence, as well as the gesture start and end position.

This is where the gesture tutorial ends. You've learned how to set up an application that uses gesture recognition and how to subscribe to the GestureRecognized event.