Sample Application
Tutorial - Multiuser Multimodal Drawing Application
In this tutorial we will develop our first multi user speech and gesture enabled drawing application using Microsoft Visual Studio 2005. Before starting you will need to make sure that both Visual Studio 2005 and GSI Demo have been installed.
This tutorial assumes that you already know the basics of using GSI Demo and creating your own mappings. In this tutorial you will learn how to
- Add GSI Demo to your Visual Studio Toolbox
- Use Gesture Engine events (Posture Down, Gesture Move, Gesture Up)
- Configure and recieve speech events
- Draw and erase lines in C#
While we recommend you create this program from scratch, we also provide executables and source that you can download and try.
Download: Gsi-MultimodalSketch.zip
|
1) Before starting (and if you have not already done so), make sure that the speech client is started by following steps 1 through 6 in the Getting Started tutorial. Also make sure that you have | Included GSI Demo in Visual Studio Toolbox.
2) Start Microsoft Visual Studio 2005 and create a new C# project called Multimodal Sketch
3) You should see an Multi User Multimodal Integrator in the Toolbox in the Tab where you had previously installed it. Click on that component and drag an instance onto the form. Your screen should look like below.
4) Now click on the event tab of the properties window (the lightning bolt) and double click on the PostureDown event handler. This will automatically generate a Posture Down event stub in the code window. Switch back to the design view (Form1.cs [Design]) and repeat step 4 for the GestureMove, GestureUp and SpeechRecognized events.
5) Now you'll need to add some references in order to get GSI Demo to work. Open the solution explorer and right click on the references folder. Select Add Reference and when the new windows appears select the browse tab. Goto the GSI Demo bin directory (default c:\program files\ilab\GSI Demo\bin\) and select GestureEngine.dll, AxInterop.DIAMONDTOUCHLib.dll, and MathNet.Numerics.dll. These components are needed by the MERL Gesture Engine (which is contained in the multi user multimodal integrator).
6) We are now ready to start coding. Switch to the code tab (called Form1.cs). At the top of page add the three using references
using GestureEngine;
using GsiDemo;
7) Next add some initialization code to the constructor. Our program will allow multiple people to have different colour pens specified by a speech command. The last point of each speech command must be saved so that we can draw a line between the last two points in our drawing application.
private Color[] UserColors = { Color.Red, Color.Green, Color.Blue, Color.Yellow };
private Point[] LastPoint;
private int TotalUsers;
public Form1()
{
InitializeComponent();
multiModalMultiUserIntegrator1.Start();
multiModalMultiUserIntegrator1.Started += new EventHandler(multiModalMultiUserIntegrator1_Started);
}
8) After the integrator has started we need to initialize the LastPoint variable and set the speech commands
{
TotalUsers = multiModalMultiUserIntegrator1.GetTotalGestureClients();
LastPoint = new Point[TotalUsers];
for (int i = 0; i < LastPoint.Length; ++i)
{
LastPoint[i] = new Point(-1, -1);
}
multiModalMultiUserIntegrator1.SetSpeechCommands("red pen,green pen,blue pen,yellow pen,clear screen,");
}
9) Next, we will add some speech event handling code. Go to the SpeechRecognition event handler, the following code plays a sound cue after each recognized speech event and sets the apropriate colour, or clears the screen.
{
//play an audio cue
System.Media.SoundPlayer sp = new System.Media.SoundPlayer(@"c:\windows\media\Windows XP Information Bar.wav");
sp.Play();
if (SpeechIndex < 4)
UserColors[UserId] = OriginalColors[SpeechIndex];
else //clear the screen
this.CreateGraphics().Clear(this.BackColor);
}
10) Now, we are ready to implement some basic drawing code, we will start with the Posture Down and Gesture Up events. We use Posture Down instead of Gesture Down because Posture Down is activated the moment that a finger touches the table while Gesture Down is only activated once a gesture has been successfully recognized over several input frames. When someone touches on the table we will set the Last Point to a new point and when it is released we will reset that variable to its default value.
{
LastPoint[e.receiverId] = new Point(e.x, e.y);
}
private void multiModalMultiUserIntegrator1_GestureUp(GestureEventArgs gesture, _DDiamondTouchEvents_TouchEvent e)
{
LastPoint[e.receiverId] = new Point(-1, -1);
}
11) Now for the drawing code. First we check to see if the LastPoint variable has been set. Then we check to see if a single finger is placed on the Diamond Touch, if it is we draw a line between the current point and the last point using the user specified colour. If it is not a single finger we erase the bounding region of the current touch.
{
if (!(LastPoint[e.receiverId].X == -1 && LastPoint[e.receiverId].Y == -1))
{
Graphics g = this.CreateGraphics();
if ((int)GestureIndicies.OneFinger == gesture.Index)
{
Pen userPen = new Pen(new SolidBrush(UserColors[e.receiverId]));
userPen.Width = 5;
userPen.EndCap = System.Drawing.Drawing2D.LineCap.Round;
g.DrawLine(userPen, new Point(e.x, e.y), LastPoint[e.receiverId]);
}
else //erase comments
g.FillRectangle(new SolidBrush(this.BackColor), new Rectangle(e.left, e.top, e.right-e.left, e.bottom-e.top));
}
LastPoint[e.receiverId] = new Point(e.x, e.y);
}
12) At this point, we have completed our simple multimodal sketching example. You can run the project by pressing F5. It should look something like below. Press Alt-F4 to exit.
- It is important to note that GSI Demo is a wrapper around the MERL Diamond Touch Gesture Engine and Microsoft's Speech SDK. Instructions for using the MERL Gesture Engine can be accessed through the Start Menu \ All Programs \ MERL \ Gesture Engine \ Documentation.