Advanced Mappings

<< Back to the GSI Demo page

Tutorial - Advanced Mappings

Below is a brief walkthrough of mapping speech and gesture functionality to Mozilla Firefox. this tutorial assumes that you already know how to create the simple mappings shown in the Creating Mappings example.

In this tutorial you will learn how to

• Map an inverted gesture
• Map a bimanual selection gesture

Some support material is also provided below.

1. Before starting (and if you have not already done so), make sure that you have run through Steps 1-7 in the Getting Started Tutorial.
2. Open Mozilla Firefox
3. Open the Input Recorder through the Start menu \ ... \ GSI Demo \ GSI Recorder.
4. The default set of recognized hand postures are: One Finger, Two Fingers, Five Fingers, One Hand, One Chop (hand side), One Fist, and One Arm. Lets map a fist to the middle mouse button. Say "Computer when I do" and drag a fist along the application, then say "you do" and middle click along the same area. Finally, say "okay" this should create our first mapping. You should need a new saved command called "One Fist gesture"
   [ SPEECH AND GESTURE ]
   Total Events 2
   GestureDown OneFist 924,568
   GestureUp OneFist 922,659
   
   [ KEYBOARD AND MOUSE ]
   Total Events 2
   MouseDown Middle 895,421
   MouseUp Middle 920,670
5. Next, lets map an inverted gesture. In this case five fingers to an inverted middle mouse button. Start by saying "Comptuer when I do" and then drag a five finger posture from the middle of the screen to the bottom. Then say "you do" abd then drag your mouse from the middle of the screen to the top and say "okay". See the video for a demonstration of this action. You should see a new saved command called "Five Finger gesture inverted". Inverted gestures are detected by looking at the position of the up and down events for both the gesture and mouse commands. In general, GSI Demo ignores all gesture dynamics and only uses information found in the up and down events.
   [ SPEECH AND GESTURE ]
   Total Events 2
   GestureDown FiveFinger 913,402
   GestureUp FiveFinger 935,650
   
   [ KEYBOARD AND MOUSE ]
   Total Events 2
   MouseDown Middle 842,408
   MouseUp Middle 793,98
6. Now we are going to map a bimanual selection to a click drag event. Say "Computer when I do" and then place one hand side on the table, wait a second and place the second hand side on the table and then remove both hands simultaneously. Now say "you do" and click and drag over the exact area that was selected between your hands. A new saved command called One Chop Bimanual Selection should be added. This particular command may take several tries, make sure that the apropriate gestures are recognized and that your mouse selection is fairly close to the gesture area.
   [ SPEECH AND GESTURE ]
   Total Events 2
   GestureDown OneChop 629,518
   GestureUp OneArm 632,551
   
   [ KEYBOARD AND MOUSE ]
   Total Events 2
   MouseDown Left 611,410
   MouseUp Left 976,627
7. Now lets add some speech commands "home" (alt-home), "back" (alt-left arrow), "forward" (alt-right arrow) using the same procedure are described in the Creating Mappings example.
8. Now we've finished our mapping for FireFox, click on the Textbox above Save Commands and type in a file name FireFox then press save commands. The system will automatically add the .config extension.