Thinking of what are the hottest buzz-words in last years “touch screen” and “gestures” come up in mind and many companies work on supporting them in their products. Let’s look at “Gestures” – Microsoft has support for touch-screen based gestures in upcoming Windows 7, Opera has mouse gestures for a long time and might have face gestures in future (or not 
)
Anyway it sounds like Qt should also benefit from having an cross-platform and extensible Gestures API allowing us to use both native (aka “system-generated”) gestures (Zoom gesture on Windows and Pinch gesture on Macs) and custom gestures that are defined by the application developer.
We are talking about mouse- and finger-based gestures, i.e. strokes that are drawn with a mouse pointer or with a finger on a touch screen/touchpad (which are translated to mouse events) – from simple double-tapping on a screen (which can also be considered a gesture) to multi-touch gestures (i.e. that are made using several input points – with two fingers).
I’ve written some mockup code showing how gestures could be used:
void ImageViewWidget::gestureEvent(QGestureEvent *event) { if (event->gesture(Qt::TapGesture)) { // we were tapped! } else if (const QGesture *g = event->gesture(Qt::DoubleTapGesture)) { scaleTo(3.0, 3.0); update() } else if (const QGesture *g = event->gesture(Qt::PanGesture)) { if (g->state() == Qt::GestureStarted) setCursor(Qt::SizeAllCursor); else setCursor(Qt::ArrowCursor); panByOffset(g->pos() - g->lastPos()); update(); } else if (const QSlideGesture *g = static_cast<QSlideGesture*>(event->gesture(Qt::SlideGesture))) { // converted from base class QGesture to QSlide gesture that // contains extended info about this particular gesture. if (g->state() == Qt::GestureFinished) { if (g->direction() == Qt::RightDirection) loadNextImage(); else if (g->direction() == Qt::LeftDirection) loadPreviousImage(); update(); } } else if (const QGesture *g = event->gesture(Qt::TapAndHoldGesture)) { if (g->state() == Qt::GestureFinished) { QMenu menu; menu.addAction(loadNextImageAction); menu.addAction(loadPreviousImageAction); menu.addAction(toggleZoomAction); menu.exec(mapToGlobal(g->hotSpot())); } } event->accept(); }
However there are a lot of unanswered questions – could it be useful? What are the use cases for gestures? What kind of gestures should work out of the box? (ok, ok, I suppose finger scrolling should work where appropriate).
And more technical ones – how should gestures behave regarding mouse/touch input – consider a Tablet PC with a touch screen where the user taps and moves a finger – should mouse events delivery be delayed until we decide whether it is a gesture, or should we deliver mouse down event, several mouse move events and then switch to delivering gesture events instead? What about ambiguous gestures – what should we do if we are not sure whether it is a Tap or Panning gesture, or we should wait until DoubleTap gesture is finished.
Do you have any other interesting use cases?
Possibly related posts:
11 comments
Whatever you do, please do not forget users of real tablet PCs (without a keyboard), where an active pen is used to replace the mouse (allows hovering, unlike touch screens.) Of course it would be nice to have some way to get system supplied gestures. KDE for example, now has integrated EasyStroke gesture recognition, and being able to define hundreds of custom gestures for all applications will really improve my life. And maybe we can get handwriting recognition as a Qt accessibility application
Your code looks a bit funny, why not use a system of predefined “actions” that can be assigned to either a key sequence, a gesture, a voice command, a D-Bus event, etc…
So any application that has a zoom action could either support +/- keys, a zoom level drop down, or a two finger gesture automatically.
Crucial to this is multi-touch gestures as well. You have to support the dropping of one or more fingers, along with their coords.
You need an event to reflect if a finger has been added (0-1, 1-2, 2-3, 3-4, etc) or removed, then we need to code sequences (state machine) to emit certain standard events.
Rotate – drop 1 finger, drop 1 more finger, finger 2 moves theta degrees.
Scale – drop 1 finger, drop 1 more finger, fingers 1 & 2 move.
Tap – drop 1 finger, remove one finger, if pos delta=0 finishes event.
Double tap – - drop 1 finger, remove one finger, if pos delta=0 drop 1 finger, remove one finger, if pos delta=0, if the removed are in a threshold of time finishes event.
Tap Drag (pan, slide) – drop 1 finger, move, remove one finger, if pos delta>0
Tap Hold – drop 1 finger, delta=0 time > threshold
A double-finger scale would average deltas of fingers 1,2 with 3,4, for finer control.
Double tap can be repeated with any number of fingers, just as tap hold.
However the trick is there may be several events going on with different widgets at the same time. So we have to group these events by the widgets/items that they map to. There may be many people using the same device at the same time. If we constrain to bounding rects, then we can service several events at once.
It would be awesome if Qt could provide default handlers for widgets. Something as easy as scale would be mapped to setSize(). Rotations could only work in QGraphicsView, unless you wanted to add that to the widget’s paintEvent().
But whatever you do, start planning on multi-touch!
How is slide different from pan?
I think it would make more sense for QGestureEvent to provide a gesture function for retrieving the QGesture object that does not take an argument. Then the QGesture object could provide a type() function that returns a value from an enum like Qt::TapGesture and so on. Thus at the top of your event handler you could get a handle on the QGesture object, then just check the type using a switch statement if your implementation can really be that simple.
I’m quite excited about this blog entry. I’m very interested in supporting pinch zooming and three-finger swiping in an application I’m developing down the road. If we could create our own multi-fingered gestures as well that would be killer. I’m curious if that is possible on “older” MacBook Pro’s, that support two-fingered scrolling but do not support pinch zooming. Might it be possible to develop custom pinch zooming gestures for these slightly older machines?
Talking about gestures, am not sure if people have heard or used an application named Sensiva. That site no longer exists now. But it was one of the best gesture based application I had come across, allowing one to make own actions and symbols.
Something similar in Qt would be rocking.
I did something like that a while back. You even featured an article on it in QtQuarterly. You can find the (lame) project page here: http://thelins.se/index.php?title=Mouse_Gesture_Recognizer .
@Carina: one doesn’t exclude another – there might be a set of pre-defined actions (actually we already have something similar – standard actions -http://doc.trolltech.com/4.5-snapshot/qkeysequence.html#StandardKey-enum) which gestures can be mapped to. However I am not sure that it will work well in general case – for example pinching defines zooming by a specific scale depending on a distance between two fingers – and it can be tricky to activate a standard action and provide enough information to it at the same time.
The gesture api is indended to be extensible, which means the user can implement his own “gesture recognizer” – a state machine that translates a set of any events into a QGestureEvent. (I’ll blog about planned extensibility of the gesture api later).
However it definitely make sense to support most common gestures (like aforementioned flick scrolling) in standard Qt widgets, or at least provide an easy way to add a gesture to a widget (notice the QSlideGesture in the code above – gesture can provide extended information like scaleFactor or rotationAngle for Pinch gesture).
@scrop1us: @Will: well currently I focus on single-user single touch and multitouch (with Windows 7 on a Dell laptop) gestures. And indeed several gestures can occur at the same time, that’s the reason why QGestureEvent::gesture() takes an argument to specify which gesture we are looking for in a list of currently going on gestures.
Slide and Pan look almost the same, I haven’t decided whether they should be merged into one gesture or not. The only difference I see is that Pan can be implemented as a multi-touch gesture – for example panning with two fingers.
@Will: as I mentioned above – gestures can be ambiguous – there can be several gestures going on at the same time – that’s why QGestureEvent stores a list of QGesture objects and the user can check if the gesture he is interested in is contained in the list by calling QGestureEvent::gesture() function which returns non-NULL value on success.
Regarding older MacBooks – I am not sure – afair on these machines gestures that are made on a touch pad are translated to mouse wheel events automatically by the driver, so that the application cannot tell a difference if it was a normal mouse wheel or a touch pad.
@Johan: your article on QQ was one of the first things I read about gestures, it was very inspiring, thank you!
Well, the code is not really up to date. My I kind of like the approach of an event filter, and having gestures being some sort of action object. There was some work done to separate the recognizer from the input handling (to allow live gestures – w/o having to press any buttons) but then work has died off.
I wrote a gesture system for the Nokia N8x0
It works ‘today’
http://www.youtube.com/watch?v=OPQ1VlyMPYU
(well, that was february, today’s version is better!)
To add a “left right” and a “right left” gesture to any widget (label, button etc):
GestureWatcher* gw = GestureWatcher::getGestureWatcher(); // get the (singleton) watcher
gw->connect(desc, Gesture(“r l “),
this, SLOT(desc_clicked())); // watches events on ‘desc’ and sends a signal to this.desc_clicked()
gw->connect(desc, Gesture(“l r “),
this, SLOT(desc_clicked()));
I based it on some of Johan’s ideas (thanks!) and I hope I improved it a bit.
The code lives here:
http://repo.or.cz/w/shopper.git?a=tree;f=src/ui;h=4547f5a22c156648d17847f7bc26a0c13453aba4;hb=gestures
The problem I’m working on at the moment is getting gestures to work with kinetic scrolling
I use a ‘replay events’ approach and queue events until I decide an event has started.
An event starts if I get movement over a certain amount within a small time.
If no move event occurs or if a release occurs then I replay all the events in order so clicks and even drag’n'drop still works.
I used a similar approach in the kinetic scrolling which I’m trying to get added to QT4.5 on maemo (we’re still testing it)
The test suite was all the qtdemo apps here are some:
http://www.youtube.com/watch?v=4TxAIScXQvk
@Johan where exactly is you article in QQ – which number? Thanks
The article Johan was talking about is here: http://doc.trolltech.com/qq/qq18-mousegestures.html
Comments on this entry are closed.