Logging Qt User Actions

[Frank J. Lhota]

[ This was previously posted to qtforum.org ]

We are currently developing a somewhat large Qt application. One of our requirements is to include a facility for recording each user action. For example, there should be a log entry for each time the user presses a button, or edits the text in a line edit, or moves a slider. These log records should have a time stamp indicating when the user action took place.

Given that this application has a lot of displays, we looked into some way of automating the process of creating this log. To do this, we created recorder classes that exist solely to record the user actions of its parent widget. For example, we created a push button recorder class whose instances are children of push buttons. When the push button recorder is created, it connects a slot to its parent's clicked() signal. When the parent button is clicked, the push button recorder creates the corresponding user action record. We created similar classes for recording line edit widgets, sliders, combo boxes, and other widgets.

We then created and an event filter and installed it on the application object. This event filter does not filter any events; we use this filter to process Polish events. When a widget is polished, we look at its class (using the qobject_cast template) and create a corresponding recording object if possible.

This worked well in some aspects. By writing a moderate amount of centralized code, we were able to record user actions for everything in our application, including actions in predefined dialogs such as instances of QMessageBox and QWizard. But we found a serious problem with this scheme: the actions are sometimes recorded out of order with incorrect time stamps.

To explain why this is so, imagine that we have a form with a push button labeled "Build Rome". When the user presses the "Build Rome" button, a complex wizard appears so that the user can enter parameters for the city of Rome. The "Build Rome" button's clicked() signal will have two slots connected to it:

- The push button recorder slot for recording the button clicks; and
- The slot used for the button action, i.e. presenting the city of Rome parameters wizard.

As the QT documentation indicates, when a signal is emitted, the order in which the connected signals and slots are called is not guaranteed. Assume the user clicks the "Build Rome" button. The two slots listed above will be called in some order. If the push button recorder slot is called first, then the user action log looks OK. But what happens if the slot that presents the city of Rome parameters wizard is called first? Then the log will record the user actions on the wizard dialog before it records the "Build Rome" button click! Moreover, the time stamp for the "Build Rome" button click will actually be the time when the other slot returns, which can be considerably later than when the user clicked the button.

Can this approach be salvaged? If not, is there another approach to creating a log of the user actions? One cannot help but think that we are not the first developers who needed to do something like this.

[Methedrine]

I am not quite sure if a signal/slot mechanism is the right approach towards your problem.

Did you try taking a look at the command pattern? Qt supports that via its Undo Framework. The name is a bit misleading, but it does effectively allow recording of actions performed by the user.

Another possibility might be to use a signal proxy to which you connect all emitted signals so you could do log processing there (instead of attaching a record action slot to relevant signals).

[Frank J. Lhota]

First of all, thank you for the suggestion.

I looked into the Qt Undo Framework, but I am not sure how it helps with creating the user action log. It would be easy to define subclasses of QUndoCommand that can create a user action log entry. But how do we then create the corresponding QUndoCommand instances whenever a user action takes place?

Could this problem be solved by creating a separate thread for recording the user actions? This approach requires that all recorder objects would belong to this recorder thread. The connections between the widget signals and recorder slots would therefore be of type QueuedConnection. If queued connections are processed before direct connections, this approach should work, but AFAIK the Qt documentation makes no such guarantees.

[Ginsengelf]

Hi, to use your example:

you will not differentiate between the action logging and the building of Rome, instead you create a subclass of QUndoCommand that builds Rome in its redo() method and Qt will automatically log the action for you.

Maybe a generally helpful link about QtUndo framework (though not directly connected to your problem):
http://doc.trolltech.com/qq/qq25-undo.html

Ginsengelf

[caduel]

While it is true that the Undo-framework can help with the logging (the storing part of the logging, to be more specific) of user input, I do not see how it could help with the (more complicated) part of detecting of user actions.
[Side note: As the logging should work unintrusively, I think the Undo-framework is not appropriate here. Still the framework does have its uses.]

If the logging should work on existring classes like QMessageBox, then obviously the solution can't modify the watched classes.

So, how can we monitor arbitrary classes?
I see just two ways:

the easy one: attach to emitted signals
(This assumes that the relevant information is emitted via signals. A problem is the unspecified order: how to make sure the logging takes part before the action (if the action takes some time to complete)?)
install an event filter: this solves the problem of the unspecified order; but you have to relate events like key pressed with more abstract actions like "input changed"


I know that this just re-states your problem and does not solve it.
Perhaps you can find some inspiration on how to do that in some of the (Qt) GUI testing tools, like squish: http://www.froglogic.com/squish/. These have to log user actions, too.

[wysota]

A low-level approach to what you require would be to install an event filter on the application object. You would then be notified of every event that happens in your application, so you could have a chance to record practically everything. Another case is to filter what you get to leave only what you want.

Edit: Blah... I see you already did that... I must start reading posts more carefully :) Sorry for the fuzz...

Edit 2: You must be doing something wrong. Event filters are triggered before signal emissions are performed. If you have a single event filter and then let your recorder objects (if you really need them) register with the filter and not apply a filter on your own (or whatever you are doing), you should be notified of the event before the destination object even has a chance to know it was clicked on (or whatever). There is no way you could receive events in a wrong order if you use the event filter properly.

[caduel]

A problem with the event based approach might be, that it also records mouse events of disabled buttons and such like. The events recorded might not be what is wanted in the "user action log".
(Ok, I'm not the OP, so I'm guessing a bit.)

[wysota]

A problem with the event based approach might be, that it also records mouse events of disabled buttons and such like.
This is easy to detect, so that's not a big problem.

[Frank J. Lhota]

We looked into using the event filter to record user input events, i.e. to look at each mouse and keyboard operation. The problem is that the customer really wanted a higher level view of the operations. They did not want to see something like


"The user dragged the mouse from point (6, 23) to point (10, 54) on widget theSlider."

Instead, they want something like


"The user changed widget theSlider value from 85 to 70."

The signal / slot mechanism did provide this sort of output. The only problem with our signal logging was the ordering problem.

My examination of the Qt 4.3.2 source code indicates that they seem to have some hooks in the object model (e.g. the function qt_register_signal_spy_callbacks) for doing this sort of logging, but I really don't want to use something that is intended for internal use, and therefore may go away in future versions. Oddly enough, it appears that the QSignalSpy class does not use these logging hooks in version 4.3.2.

Has anyone seen something else in the QtTest module that can be used for this purpose?

[Frank J. Lhota]

The event filter we use looks at the Polish events. Depending on the class of the widget involved, we create and attach the appropriate recorder object (e.g. push button recorder, slider recorder etc.). The widget signals that we are interested in are attached to corresponding slots in the recorder object. The signal / slot mechanishm then does the logging for this widget.

The event filter does no filtering; it just does this extra processing for the Polish events.

[wysota]

We looked into using the event filter to record user input events, i.e. to look at each mouse and keyboard operation. The problem is that the customer really wanted a higher level view of the operations. They did not want to see something like


"The user dragged the mouse from point (6, 23) to point (10, 54) on widget theSlider."

Instead, they want something like


"The user changed widget theSlider value from 85 to 70."

Maybe it would be simple enough to interpret those values properly? If you know the widget, you know its class name, you know what it does and you can try to predict what is going on. Signals are fired only when an event is already delivered and accepted, so that's too late. What you could do is you could tweak Qt's source code and provide some hook you might use, but you'll have to go through all places in code that contain useful signal emissions (or tweak the emission code itself).

[Frank J. Lhota]

Maybe it would be simple enough to interpret those values properly? If you know the widget, you know its class name, you know what it does and you can try to predict what is going on. Signals are fired only when an event is already delivered and accepted, so that's too late. What you could do is you could tweak Qt's source code and provide some hook you might use, but you'll have to go through all places in code that contain useful signal emissions (or tweak the emission code itself).

Some low-level events can be easily turned into high-level user action entries. But the task of doing this in general can sometimes be rather complex and tricky. Consider the key events: the result of pressing an arrow key / tab key / enter key / escape key can depend on not just the widget with the focus: it can also depend on the siblings and parentage of that widget. Let's face it: there is a non-trivial amount of code in Qt and the operating system to translate low-level input actions to high level windows events. We do not want to be obliged to shadow this code.

Given that we want to maintain this software for many years, we don't really have the option of customizing the Qt code.

[wysota]

Given that we want to maintain this software for many years, we don't really have the option of customizing the Qt code.

Why not? If you apply a patch, you can apply it again on subsequent releases. Code that deals with handling signals is not likely to change easily, so patching once should be sufficient until the end of Qt4.

[bootchk]

Just an idea. Could you log the events also? Then you would have a sequence of timed events and signals. You also remember which events are associated with a widget to which you have added an extra signal slot. In a post processing step, you reorder the signals, so that the associated signals are immediately after the associated event. You also futz with the time of the moved signals.

I'm not saying it is pretty. Just doable.

I'm also trying to understand similarities with other GUI toolkits. Aren't Qt signals like actions in OSX? Do other GUI toolkits allow multiple actions (signals) per event, and also not allow an ordering of the signals?

As mentioned previously, there are tools like Squish that seem to accomplish what you want to do. And for multiple platforms. It seems like they would have an abstraction for the whole mess that would be cross platform. Just as you object to hacking the GUI toolkit, I doubt they hack.

[iraytrace]

I picked up and modified some code from at Stanislaw Adaszewski (http://http://algoholic.eu/recording-and-replaying-qt-input-events/) that does some of this. It works most of the time.

The only problem I have found is that when using QFileDialog some of the QKeyPress events never show up if you type the filename. They seem to get stolen by QFileDialog and are never recorded. However if you double-click on file names you can open files from the resulting recorded event log.