Creating Mappings

<< Back to the GSI Demo page

Tutorial - Creating Mappings

Below is a brief walkthrough of mapping speech and gesture functionality to Windows Solitaire. In this tutorial you will learn how to

• Use GSI recorder to map an existing single user application
• Learn some of the limitations of GSI Demo

Some support material is also provided below.

1. Before starting (and if you have not already done so), make sure that you have run through Steps 1-7 in the Getting Started Tutorial.
2. Open Windows Solitaire through the Start menu \ Games \ Solitaire
3. Open the Input Recorder through the Start menu \ ... \ GSI Demo \ GSI Recorder.
   
   Once the Input recorder is opened, you should notice a change in the Speech Client's list of speech options. There are two key commands "Computer when I do [gesture]" and Computer when I say [speech command]". We will use this for our mapping.
4. First, we want to map a single finger gesture to a left mouse click. Say "Computer when I do" and wait for the audio cue and then do a one finger gesture across the table. Then say "you do" and drag the mouse cursor over the place that you last touched with the left mouse button and complete the mapping with "okay". This does not need to be perfect, GSI Demo will generalize from this sample and will create a one finger gesture mapping (right). If you want you can delete this particular mapping at any time.
   Try the mapping out, you should now be able to use a single finger to move cards around in solitaire. Take a look at the current command on the left. It should say something like
   [ SPEECH AND GESTURE ]
   Total Events 2
   GestureDown OneFinger 952,652
   GestureUp OneFinger 752,648
   
   [ KEYBOARD AND MOUSE ]
   Total Events 2
   MouseDown Left 941,644
   MouseUp Left 682,625
   This means that it has mapped a one finger gesture to a left mouse click. this mapping can be deleted by selecting the command and clicking on the remove selection button
5. Next, lets try a simple speech mapping. First, make sure that solitaire is the application in focus by clicking on it. Then say "Computer when I say" and type new game, then say "you do" and type F2 then say "okay".
   [ SPEECH AND GESTURE ]
   Total Events 1
   SpeechRecognitionEvent new game
   
   [ KEYBOARD AND MOUSE ]
   Total Events 2
   KeyDown F2
   KeyUp F2
   Please note that I chose to use the keyboard shortcut for new game instead of going to the file menu and selecting new game for two reasons. First, the F2 key is more direct and would not result in a screen flicker when run. Second, mouse clicks are stored in absolute coordinates, thus if I moved the window this mouse sequence would no longer work.
6. Lets create another mapping for suit stacking (as seen in the video). Say "Computer when I say", type suit stack, then say "you do" and type Ctrl-A. Repeat the same procedure for next card which corresponds to the spacebar key.
7. Now we've finished our mapping for solitaire, click on the Textbox above Save Commands and type in a file name Solitaire then press save commands. The system will automatically add the .config extension.