Automated GUI Testing (Testability, Squish) - has anyone used it?

[japcrword]

We're developing a commercial application. I'm seeking a way to automate qt gui testing. I know only 2 qt-specific tools for that:
1) Testability Driver (open source)
http://projects.developer.nokia.com/Testabilitydriver
http://gitorious.org/tdriver
2) Froglogic Squish (comm€rcial)
http://www.froglogic.com/

I saw also some references to hooq (http://gitorious.org/hooq/hooq). But it looks for me a little obscure due to the lack of documentation.

I admit we've never done automated gui testing. But supporting many platforms suggests looking for some automation of regression testing. One thing that makes me curious is the absence of discussion of the topic on the Internet. There are no topics on this forum, for example. And judging by Testability commit history it doesn't look like it's being actively developed. I ran it with their test applications and then added support to our application (default configuration, no tweaking). I admit it's not very stable when running it out of the box: several times it raised ruby exceptions, led to 100% CPU consumption and once crashed. It's open source, so hopefully it's possible to do some tweaks about it. And perhaps it's possible to make the application under test be more testability-friendly by writing custom traversers/fixtures. Though I'm not sure about it, hence this post.

So does anyone has experience using this tool? Can you share it? Is it worth spending time on it? I know about unit testing, but I'm interested now in functional testing. Perhaps someone has experience using Squish or other similar tool? Thanks.

[wysota]

I have played with Squish a bit. It was more of a test drive rather than real practical use but I can say I liked it very much. It's quite expensive but in my opinion it is worth its price if you use everything it offers. But first be sure what you are testing and how you wish to test it. Squish is definitely not a tool for non-technical person that is used to doing "bussiness testing", it requires some programming skills to use the tool to the maximum.

[japcrword]

Thank you for your reply. I agree that test automation is half-programming. The question that puzzles me is - if this is so good why nobody uses it? At least it seems so and especially with TDriver. As I understand both Squish and Testability Driver have similar approaches. But Squish is more advanced, it offers a broad language suit for testing (whereas TDriver uses only Ruby), it's better tested. I'd definitely prefer it if it was free, but I'm not sure I can persuade our employers to buy the license. But what I'm sure of is that if there were large market demands in tools like Squish there would be free alternatives, perhaps less advanced, but allowing solving similar problems (probably with a bit more efforts). This is how TDriver looks for me. But at the moment it seems abandoned while being far from perfection.

A typical example of what is needed would be like... Put several QGraphicsItems on QGraphicsScene, do some application-defined operations with them, then check that z-values of the QGraphicsItems are correct (as expected). This is a simple scenario, in practice it may be more complex. This is all done manually at the moment. I'm sure that both Squish and TDriver designed to allow it.

But as for Squish, did you use it on a regular basis during development? Did it prove worth it from the viewpoint of your employers?

Perhaps there's someone who still dealt with TDriver. Your comments would be also very welcome.

[wysota]

The question that puzzles me is - if this is so good why nobody uses it?

Why did you come to such conclusion? If you refer to lack of threads on the subject then that's most probably because Squish is a commercial product that not everyone can afford thus asking questions about it on a public forum such as this where most people answering are programmers and not testers and they often use Qt after work or during their studies, such questions would probably remain unanswered. I suspect Froglogic provides commercial support for its tool so most questions are probably answered by their engineers through some internal channels. As for "nobody uses it" -- if I remember correctly then Squish is used by Qt team to test Qt, at least that was true some time ago.

A typical example of what is needed would be like... Put several QGraphicsItems on QGraphicsScene, do some application-defined operations with them, then check that z-values of the QGraphicsItems are correct (as expected).

Well... that you can do with QtTestLib that comes with Qt itself. It requires some programming but it can be done without paying significant amounts of money for an external tool.

I'm sure that both Squish and TDriver designed to allow it.

I don't know about TDriver but Squish is surely capable of it.

But as for Squish, did you use it on a regular basis during development?

No, I have only evaluated it for myself without paying for a license. I couldn't afford it myself You can probably ask for an evaluation test of Squish and see if it works for you.

Did it prove worth it from the viewpoint of your employers?

I recommended getting that tool to my employer but because of non-technical reasons we didn't go that way.

[japcrword]

Why did you come to such conclusion?

Ok, when I said that I actually thought of TDriver. I meant that usage of this kind of tools is much less than I would expect. Besides there are no actively developed free / open source tools. I'm sure Squish has users (they say they are 1500+ companies using this tool)

Well... that you can do with QtTestLib that comes with Qt itself. It requires some programming but it can be done without paying significant amounts of money for an external tool.

I agree. There should be such tests as well. But in order to test you are bound to calling methods from your application instead of doing that the-user-way i.e. via gui controls. I admit, it's an option. But still I think that if possible to do it via gui it's better to do it so, because there is some probability that something go wrong, like

• someone disconnect the slot you call in your test, yet calling obsolete slot makes a test pass;
• there are some problems with gui or control may be invisible or unresponsive, yet calling a slot makes a test pass;
• controls may behave differently under different OS environments and so on;
• the result of the test relies on the methods / slots you call, so if a test is poorly written like it calls methods that are impossible to invoke via gui (for example to save time for coding a test), it doesn't prove anything, so tests should be written with some additional care.

Perhaps these reasons may seem not very convincing. But there's also one thing that might be important: testing via gui is more persuasive for the management (who decides how much time should be spent on testing) and hence it may reduce testing burden. So I consider automated gui testing an interesting option (especially with tools like TDriver which is also free).

[wysota]

But in order to test you are bound to calling methods from your application instead of doing that the-user-way i.e. via gui controls.

That's not entirely true. You can post input events to controls to trigger the functionality exactly the same way a user would do it (e.g. send mouse click to widget A, then mouse click to B, then strike "abc" on the keyboard, etc.).

I admit, it's an option. But still I think that if possible to do it via gui it's better to do it so, because there is some probability that something go wrong, like

• someone disconnect the slot you call in your test, yet calling obsolete slot makes a test pass;
• there are some problems with gui or control may be invisible or unresponsive, yet calling a slot makes a test pass;
• controls may behave differently under different OS environments and so on;
• the result of the test relies on the methods / slots you call, so if a test is poorly written like it calls methods that are impossible to invoke via gui (for example to save time for coding a test), it doesn't prove anything, so tests should be written with some additional care.

QtTestLib can handle all of the above. Just look at Qt's auto test suite.

Perhaps these reasons may seem not very convincing. But there's also one thing that might be important: testing via gui is more persuasive for the management (who decides how much time should be spent on testing) and hence it may reduce testing burden. So I consider automated gui testing an interesting option (especially with tools like TDriver which is also free).

That's true, managers like to see things happen on screen rather than in the logs

I kept thinking about the subject after I wrote my previous post. There are many ways to have a gui testing tool without spending a penny on it (well.. sort of -- excluding your own work costs). To gain control over the application you only need to access the QApplication instance of the application. This can be done in a number of ways (simplest of which is just loading the test library directly in your app if it is present). Then you can easily implement features like macro recording (to record a test case) and replay (to perform the test), object inspection (for QObject trees and QGraphicsScene items), etc. You can even embed a script engine in your app to execute test scripts "from inside" of the application under test. I would risk saying one can implement such a simple tool in a matter of 10-20 hours of work.

[japcrword]

That's not entirely true. You can post input events to controls to trigger the functionality exactly the same way a user would do it (e.g. send mouse click to widget A, then mouse click to B, then strike "abc" on the keyboard, etc.).

I agree. Thanks for the hints

[anda_skoa]

A couple of my customers use Squish and they are quite satisfied with it.
Since it was basically created to test Qt applications (and later on expanded for other frameworks), it it has all kinds of convenience built-in when it comes to recording and programming test cases.

E.g. it understands QObjects and their properties, tests can easily check and manipulate values, etc.

One of my customers wasn't sure about using it either and playing with the trial version didn't yield any certainty either. They opted to send one of the test engineers to a Squish training offered by some Froglogic partner, which obviously also cost money but allowed them to learn about the potential of Squish in a more structure way than playing around or following tutorials.

Cheers,
_

[japcrword]

A couple of my customers use Squish and they are quite satisfied with it.
Since it was basically created to test Qt applications (and later on expanded for other frameworks), it it has all kinds of convenience built-in when it comes to recording and programming test cases.

E.g. it understands QObjects and their properties, tests can easily check and manipulate values, etc.

Thank you very much for your feedback. I'm sure Squish is a good solution. So I think I'll make a proposal (it probably will be rejected due to its price). Although currently I'm choosing between TDriver and QtTestLib.