Open source automated testing tool for Qt App

[William Wilson]

Is there a good open source automated testing tool for Qt applications? Squish by froglogic is a great tool, but very secretive. They claim they offer free Squish for KDE Apps which is not actually accurate. I'm looking for a similar tool like Squish. May be the open source community needs to build one!
Thanks in advance.

[wysota]

They claim they offer free Squish for KDE Apps which is not actually accurate.

Really? What is not so accurate about the statement? That you don't get the source? They didn't say the application is open source but that you receive it for free (as in free beer).

I'm looking for a similar tool like Squish. May be the open source community needs to build one!

You won't find one and the community won't build one because it's a too big effort for anybody who handles open source software that is usually not tested at all (excluding some simple unit testing)...

You can use QTestLib that is part of Qt though.

[Uwe]

You won't find one and the community won't build one because it's a too big effort for anybody who handles open source software that is usually not tested at all (excluding some simple unit testing)...

Yes Squish is the only real testing tool for Qt. There is also KD Executor, but AFAIR it is only a record/replay tool. Maybe it's possible to find other record/replay tools, that interfere at a level below Qt ( f.e X11 ).

I totally agree with you, that a testing tool is more or less for the commercial world only ( maybe beside huge projects like KDE ), but it is an interesting project and a substantial gap in the free Qt tool chain.

About the effort: a couple of years ago I wrote a Qt plugin for SilkTest (http://en.wikipedia.org/wiki/SilkTest). Writing such a plugin means more or less, breaking down high level operations into event chains. I needed ~1 month to have something that covered ~70% of the Qt3 widgets.
Beside this you need to embed a script language and to write the application around it.

Of course writing such a testing tool is nothing for beginners, but it is not more difficult than writing an IDE. If I could chose between seeing a good testing tool or one more IDE ...

Uwe

[wysota]

Yes Squish is the only real testing tool for Qt. There is also KD Executor, but AFAIR it is only a record/replay tool.

A very nice tool, I might add Unfortunately commercial as well.

Maybe it's possible to find other record/replay tools, that interfere at a level below Qt ( f.e X11 ).

Possibly, but they won't be able to test some Qt quirks which can't be mapped onto X events.

I totally agree with you, that a testing tool is more or less for the commercial world only ( maybe beside huge projects like KDE ), but it is an interesting project and a substantial gap in the free Qt tool chain.

Oh yes, definitely. Unfortunately there is not much we can do about it. Take a look at the IDE projects we have and their state and cooperation. Then realize that a testing tool is much more complex and requires deep knowledge of Qt. I'm really fond of Squish, unfortunately it's ugly and expensive.

Beside this you need to embed a script language and to write the application around it.

Right now that's not a problem as Qt gives us ECMAScript for free and using Python is quite straightforward as well.

Of course writing such a testing tool is nothing for beginners, but it is not more difficult than writing an IDE. If I could chose between seeing a good testing tool or one more IDE ...

I'm not sure I can agree with that. An IDE doesn't really deal with platform dependent quirks like controlling the AUT which I belive is different depending on the platform.

[Uwe]

I'm not sure I can agree with that. An IDE doesn't really deal with platform dependent quirks like controlling the AUT which I belive is different depending on the platform.

Well, the communication between the user and the application is done by Qt events. In my SilkTest plugin I wrote a class that mapped the basic mouse and keyboard operations into Qt event chains. This was the difficult part, as you have to take care of all these enter/leave/focus-in/out stuff.
Then I wrote a proxy class for each Qt widget, that exported high level operations ( like PushButtonProxy:ress() ) to the testscript language. The implementation of these operation was simply a sequence of delays and the mouse/keyboard operations, I had implemented before.

I'm pretty sure, that there are use cases, that can't be handled with such a design, but it is completely platform independent ( pure Qt code ) and should cover most of the usual ways of scripted testing.

Writing an application around it is still a lot of work, but standard Qt/GUI coding. In the end you have a "Poor Man's Test Tool" for Qt applications, that is not complete, but very useful - and as soon as you can offer a working tool, it will be improved ( or replaced by something better ) by the community.

Uwe

[wysota]

Squish has nice features such as when you can point the widget to be selected using mouse with a red border around it, you can inspect objects of the application, etc.

I don't know how exactly the tool you mentioned works, but I suspect obtaining such capabilities with it would be very difficult. Translating events is one thing and its the least such a gui testing tool must do, but it's the added value that makes a tool unique. Look at IDEs - one might say that to write an IDE you need a text editor and buttons to perform compilation and execution and that's basically true, but it's all those little things that make tools such as Visual Studio, Eclipse or KDevelop loved by many.

[Uwe]

Squish has nice features such as when you can point the widget to be selected using mouse with a red border around it, you can inspect objects of the application, etc.

I don't know how exactly the tool you mentioned works, but I suspect obtaining such capabilities with it would be very difficult.

By far the most important thing about a testing tool is, that you can implement automated regression tests. ( This is something you can do with Squish, but not with simple tools like KD Executor. ) My implementation was only for this aspect.

In commercial projects you often find a QA team, that implements these test suites and runs them for every new release of your product. I have worked in an environment, where these test suites ran unattended every night on the development branch, sending the developer a mail, when a test fails.

It is very important, that you can implement your test scripts on an abstract level, to avoid that they break, with every layout change in a new release. That's why record/replay is of limited use. All manual testing is of low importance ( you don't need a tool for this) and debugging functionalities, like the one you have mentioned, should be part of a debugging tool ( what Squish seems to be too ).

Uwe

[wysota]

I think you misunderstood me. The features I was talking about were not debugging functionality. They are part of the test creation process. For instance you have an application that is a calculator and you want to test addition. So you create a test where you press "2", "+", "2", "=" and then you want to check whether you obtained "4" as the text of a proper QLineEdit. To do that you need to be able to read properties of widgets and you need to be able to choose widgets that are interesting for you. Take a look at the demo video available on Froglogic site - it demonstrates the use of those features.

[Uwe]

To do that you need to be able to read properties of widgets and you need to be able to choose widgets that are interesting for you.

Well, in the Qt3 solutions package was a Qt Object Inspector
( http://doc.trolltech.com/solutions/q.../overview.html ) and AFAIR there were other tools around dumping the meta object. It's one of these features, that are adequate for demos, because they show something nice, without having to implement much.

I don't want to insist on a definition of debugging and testing, but I'm sure a QA engineer wouldn't debug/test properties. And to be honest: I never felt, that I ( as developer) need a tool like this, when I have a real debugger.

But let me repeat: a test tool is about automated testing.

Uwe

[wysota]

But let me repeat: a test tool is about automated testing.

So what tests do you consider "automated"? Because the case I'm talking about (with properties, etc.) is definitely regression testing and I'm pretty sure it is automated.

The problem with Solutions is that the code is licenced and we are talking about an open source approach to the problem. Belive me, I would gladly see an open source automated testing tool. I just don't see anyone willing to spend that much of his time writing it - it's tedious and open source developers tend not to touch tedious work.

One more thing - the functionality (I think) you find required for a test tool is contained in QTestLib. So basically one could make a frontend for it that would generate a test program and then compile it. You could even run it from within the frontend and inspect results as they are given in XML.

[Uwe]

So what tests do you consider "automated"?

Automated means, that execution and verification of a test is done unattended.

The people running the test suite often don't know much about the application, they are testing. I've met QA engineers that were excellent developers too, but more often you find people with a very different background. F.e. in my current project the QA job is done offshore by collegues who don't know what UNIX or Qt is.

They push a button to start and after it is finished, they have a report with a summary which tests passed or failed. Analyzing the failed tests is not part of their job.

Uwe

[William Wilson]

Really? What is not so accurate about the statement? That you don't get the source? They didn't say the application is open source but that you receive it for free (as in free beer).

Well.. How does one define a KDE application? An application that runs on the KDE desktop? Apparently, that is not the definition as per froglogic. I don't know what kind of validation that they do before running the AUT under Squish but a regular Qt application will fail to run on Squish/KDE. But their example programs (addressbook app for example) which are pure Qt applications will run fine. When I contacted froglogic they said Squish will support only a *real* KDE app.

We do have excellent free/open source solutions for compiler, IDE (Eclipse), GUI frame work (Qt), performance analysis/memory detection(Valgrind) , database (SQLite) etc.. but not an automated testing tool. There is a big gap here. It's not about the complexity, Valgrind is quite complex; I think no one has felt the real need for it. I don't know why.

[wysota]

Well.. How does one define a KDE application? An application that runs on the KDE desktop?

KDE applications are probably ones that use KApplication instead of QApplication.

There is a big gap here. It's not about the complexity, Valgrind is quite complex; I think no one has felt the real need for it. I don't know why.

How many times have you used Valgrind? And how many times did you implement a full test suite for an open source GUI application? By the way, Valgrind has a significant limitation - it only works on x86 Linux.

[William Wilson]

KDE applications are probably ones that use KApplication instead of QApplication.

Actually, that's fair to assume but not a *real* KDE app as per froglogic. The KApplication also needs to have a KAboutData filled in correctly. So, I wrote a real simple KDE App with these but Squish failed with an error saying "AUT is not an open source application". And froglogic failed to completely define a *real* KDE App. Well.. may be they want the open source developer to keep trying. This is why I wrote "they are very secretive". Try getting the *free* Squish/KDE 4 version and you can see this.

[wysota]

Actually, that's fair to assume but not a *real* KDE app as per froglogic. The KApplication also needs to have a KAboutData filled in correctly. So, I wrote a real simple KDE App with these but Squish failed with an error saying "AUT is not an open source application". And froglogic failed to completely define a *real* KDE App. Well.. may be they want the open source developer to keep trying. This is why I wrote "they are very secretive".

You are much too hard on the Frogs. They are really helpful people and at least some of them contribute to KDE itself. I don't know what problems you had, but I'm sure a bit of good will is enough to solve them. The problems you mention are probably a kind of protection against abusing the software which I can totally understand.

Try getting the *free* Squish/KDE 4 version and you can see this.

Actually I got the Qt4 version for evaluation without any big problems. There was a problem with a licence, but it was easily solved.

[Uwe]

We do have excellent free/open source solutions for compiler, IDE (Eclipse), GUI frame work (Qt), performance analysis/memory detection(Valgrind) , database (SQLite) etc.. but not an automated testing tool. There is a big gap here. It's not about the complexity, Valgrind is quite complex; I think no one has felt the real need for it. I don't know why.

Me too, but I have some ideas:

1) Writing a test suite is comparable to developing an application. Often a QA team takes about 20-50% of the man power in a project. So it's obvious, that only commercial projects would use it. And writing a open source project for the commercial market only is not very motivating.

2) Win Runner, X Runner, Silk Test etc. are expensive products, nobody would buy for fun. So probably most developers have never seen such a thing.

3) Obviously the Qt market is too small to be a business case for vendors of test tools. So we don't have competition leading to a dual license.

But in the end it needs only one person to start such a project.

Uwe