Open source automated testing tool for Qt App

[William Wilson]

Is there a good open source automated testing tool for Qt applications? Squish by froglogic is a great tool, but very secretive. They claim they offer free Squish for KDE Apps which is not actually accurate. I'm looking for a similar tool like Squish. May be the open source community needs to build one!
Thanks in advance.

[wysota]

They claim they offer free Squish for KDE Apps which is not actually accurate.

Really? What is not so accurate about the statement? That you don't get the source? They didn't say the application is open source but that you receive it for free (as in free beer).

I'm looking for a similar tool like Squish. May be the open source community needs to build one!

You won't find one and the community won't build one because it's a too big effort for anybody who handles open source software that is usually not tested at all (excluding some simple unit testing)...

You can use QTestLib that is part of Qt though.

[Uwe]

You won't find one and the community won't build one because it's a too big effort for anybody who handles open source software that is usually not tested at all (excluding some simple unit testing)...

Yes Squish is the only real testing tool for Qt. There is also KD Executor, but AFAIR it is only a record/replay tool. Maybe it's possible to find other record/replay tools, that interfere at a level below Qt ( f.e X11 ).

I totally agree with you, that a testing tool is more or less for the commercial world only ( maybe beside huge projects like KDE ), but it is an interesting project and a substantial gap in the free Qt tool chain.

About the effort: a couple of years ago I wrote a Qt plugin for SilkTest (http://en.wikipedia.org/wiki/SilkTest). Writing such a plugin means more or less, breaking down high level operations into event chains. I needed ~1 month to have something that covered ~70% of the Qt3 widgets.
Beside this you need to embed a script language and to write the application around it.

Of course writing such a testing tool is nothing for beginners, but it is not more difficult than writing an IDE. If I could chose between seeing a good testing tool or one more IDE ...

Uwe

[wysota]

Yes Squish is the only real testing tool for Qt. There is also KD Executor, but AFAIR it is only a record/replay tool.

A very nice tool, I might add Unfortunately commercial as well.

Maybe it's possible to find other record/replay tools, that interfere at a level below Qt ( f.e X11 ).

Possibly, but they won't be able to test some Qt quirks which can't be mapped onto X events.

I totally agree with you, that a testing tool is more or less for the commercial world only ( maybe beside huge projects like KDE ), but it is an interesting project and a substantial gap in the free Qt tool chain.

Oh yes, definitely. Unfortunately there is not much we can do about it. Take a look at the IDE projects we have and their state and cooperation. Then realize that a testing tool is much more complex and requires deep knowledge of Qt. I'm really fond of Squish, unfortunately it's ugly and expensive.

Beside this you need to embed a script language and to write the application around it.

Right now that's not a problem as Qt gives us ECMAScript for free and using Python is quite straightforward as well.

Of course writing such a testing tool is nothing for beginners, but it is not more difficult than writing an IDE. If I could chose between seeing a good testing tool or one more IDE ...

I'm not sure I can agree with that. An IDE doesn't really deal with platform dependent quirks like controlling the AUT which I belive is different depending on the platform.

[Uwe]

I'm not sure I can agree with that. An IDE doesn't really deal with platform dependent quirks like controlling the AUT which I belive is different depending on the platform.

Well, the communication between the user and the application is done by Qt events. In my SilkTest plugin I wrote a class that mapped the basic mouse and keyboard operations into Qt event chains. This was the difficult part, as you have to take care of all these enter/leave/focus-in/out stuff.
Then I wrote a proxy class for each Qt widget, that exported high level operations ( like PushButtonProxy:ress() ) to the testscript language. The implementation of these operation was simply a sequence of delays and the mouse/keyboard operations, I had implemented before.

I'm pretty sure, that there are use cases, that can't be handled with such a design, but it is completely platform independent ( pure Qt code ) and should cover most of the usual ways of scripted testing.

Writing an application around it is still a lot of work, but standard Qt/GUI coding. In the end you have a "Poor Man's Test Tool" for Qt applications, that is not complete, but very useful - and as soon as you can offer a working tool, it will be improved ( or replaced by something better ) by the community.

Uwe

[wysota]

Squish has nice features such as when you can point the widget to be selected using mouse with a red border around it, you can inspect objects of the application, etc.

I don't know how exactly the tool you mentioned works, but I suspect obtaining such capabilities with it would be very difficult. Translating events is one thing and its the least such a gui testing tool must do, but it's the added value that makes a tool unique. Look at IDEs - one might say that to write an IDE you need a text editor and buttons to perform compilation and execution and that's basically true, but it's all those little things that make tools such as Visual Studio, Eclipse or KDevelop loved by many.

[Uwe]

Squish has nice features such as when you can point the widget to be selected using mouse with a red border around it, you can inspect objects of the application, etc.

I don't know how exactly the tool you mentioned works, but I suspect obtaining such capabilities with it would be very difficult.

By far the most important thing about a testing tool is, that you can implement automated regression tests. ( This is something you can do with Squish, but not with simple tools like KD Executor. ) My implementation was only for this aspect.

In commercial projects you often find a QA team, that implements these test suites and runs them for every new release of your product. I have worked in an environment, where these test suites ran unattended every night on the development branch, sending the developer a mail, when a test fails.

It is very important, that you can implement your test scripts on an abstract level, to avoid that they break, with every layout change in a new release. That's why record/replay is of limited use. All manual testing is of low importance ( you don't need a tool for this) and debugging functionalities, like the one you have mentioned, should be part of a debugging tool ( what Squish seems to be too ).

Uwe

[wysota]

I think you misunderstood me. The features I was talking about were not debugging functionality. They are part of the test creation process. For instance you have an application that is a calculator and you want to test addition. So you create a test where you press "2", "+", "2", "=" and then you want to check whether you obtained "4" as the text of a proper QLineEdit. To do that you need to be able to read properties of widgets and you need to be able to choose widgets that are interesting for you. Take a look at the demo video available on Froglogic site - it demonstrates the use of those features.

[William Wilson]

Really? What is not so accurate about the statement? That you don't get the source? They didn't say the application is open source but that you receive it for free (as in free beer).

Well.. How does one define a KDE application? An application that runs on the KDE desktop? Apparently, that is not the definition as per froglogic. I don't know what kind of validation that they do before running the AUT under Squish but a regular Qt application will fail to run on Squish/KDE. But their example programs (addressbook app for example) which are pure Qt applications will run fine. When I contacted froglogic they said Squish will support only a *real* KDE app.

We do have excellent free/open source solutions for compiler, IDE (Eclipse), GUI frame work (Qt), performance analysis/memory detection(Valgrind) , database (SQLite) etc.. but not an automated testing tool. There is a big gap here. It's not about the complexity, Valgrind is quite complex; I think no one has felt the real need for it. I don't know why.

[wysota]

Well.. How does one define a KDE application? An application that runs on the KDE desktop?

KDE applications are probably ones that use KApplication instead of QApplication.

There is a big gap here. It's not about the complexity, Valgrind is quite complex; I think no one has felt the real need for it. I don't know why.

How many times have you used Valgrind? And how many times did you implement a full test suite for an open source GUI application? By the way, Valgrind has a significant limitation - it only works on x86 Linux.

[Uwe]

To do that you need to be able to read properties of widgets and you need to be able to choose widgets that are interesting for you.

Well, in the Qt3 solutions package was a Qt Object Inspector
( http://doc.trolltech.com/solutions/q.../overview.html ) and AFAIR there were other tools around dumping the meta object. It's one of these features, that are adequate for demos, because they show something nice, without having to implement much.

I don't want to insist on a definition of debugging and testing, but I'm sure a QA engineer wouldn't debug/test properties. And to be honest: I never felt, that I ( as developer) need a tool like this, when I have a real debugger.

But let me repeat: a test tool is about automated testing.

Uwe

[wysota]

But let me repeat: a test tool is about automated testing.

So what tests do you consider "automated"? Because the case I'm talking about (with properties, etc.) is definitely regression testing and I'm pretty sure it is automated.

The problem with Solutions is that the code is licenced and we are talking about an open source approach to the problem. Belive me, I would gladly see an open source automated testing tool. I just don't see anyone willing to spend that much of his time writing it - it's tedious and open source developers tend not to touch tedious work.

One more thing - the functionality (I think) you find required for a test tool is contained in QTestLib. So basically one could make a frontend for it that would generate a test program and then compile it. You could even run it from within the frontend and inspect results as they are given in XML.

[Uwe]

We do have excellent free/open source solutions for compiler, IDE (Eclipse), GUI frame work (Qt), performance analysis/memory detection(Valgrind) , database (SQLite) etc.. but not an automated testing tool. There is a big gap here. It's not about the complexity, Valgrind is quite complex; I think no one has felt the real need for it. I don't know why.

Me too, but I have some ideas:

1) Writing a test suite is comparable to developing an application. Often a QA team takes about 20-50% of the man power in a project. So it's obvious, that only commercial projects would use it. And writing a open source project for the commercial market only is not very motivating.

2) Win Runner, X Runner, Silk Test etc. are expensive products, nobody would buy for fun. So probably most developers have never seen such a thing.

3) Obviously the Qt market is too small to be a business case for vendors of test tools. So we don't have competition leading to a dual license.

But in the end it needs only one person to start such a project.

Uwe

[Uwe]

So what tests do you consider "automated"?

Automated means, that execution and verification of a test is done unattended.

The people running the test suite often don't know much about the application, they are testing. I've met QA engineers that were excellent developers too, but more often you find people with a very different background. F.e. in my current project the QA job is done offshore by collegues who don't know what UNIX or Qt is.

They push a button to start and after it is finished, they have a report with a summary which tests passed or failed. Analyzing the failed tests is not part of their job.

Uwe

[wysota]

Automated means, that execution and verification of a test is done unattended.

They push a button to start and after it is finished, they have a report with a summary which tests passed or failed. Analyzing the failed tests is not part of their job.

So it doesn't determine what is tested, only how it is tested. I don't see why regressive tests created using point and click are not to be called automated even if they are then executed from within an automated script...

[Uwe]

I don't see why regressive tests created using point and click are not to be called automated even if they are then executed from within an automated script...

Of course testing with an automated script is automated testing. It is of no importance if it is started from the command line, as cron job, or by "pushing a button" in a test environment.

I'm afraid I can't follow, why we got lost in discussions about terminologies. Isn't it the difference between automated/manual testing obvious for everyone ?

Uwe

[wysota]

Yes, but it's not clear for me why you called what Squish does (at least the part related to the features I mentioned - by the way, have you seen the demo?) as "debugging"

[Uwe]

Yes, but it's not clear for me why you called what Squish does (at least the part related to the features I mentioned - by the way, have you seen the demo?) as "debugging"

For me testing means to check something ( an application, a library or a function ) from outside. As soon as you start looking into internals I would call it debugging. But this might be completely wrong and I don't care much about such a definition.

All I wanted to say was, that (IMHO) something like object inspection fits more into a development tool ( like an IDE ), than in a testing tool. Of course it's nothing wrong with having debugging capabilities there too, but I would prefer a more modular design, where the test tool is responsible for creating and executing test scripts only. If it comes to analyzing a situation it's enough if the test tool attaches the application to external tools ( maybe an IDE with an integrated debugger, object inspection or valgrind support ) with full featured "debugging" capabilities.

Please recall, that this thread is about writing an open source automated testing tool. It's much easier to start with the core functionality instead of trying to include too much at the beginning. For me the core functionalities are:

1) The definition of a test language.
Squish extends standard script languages, but f.e. SilkTest has it's own 4GL language called 4Test.

2) A event chain generator
Something like what I wrote for the SilkText plugin.

3) An interpreter that translates test scripts into event chains.

With 1-3 you can already do full automated testing.

In the next step you could add a GUI around it with all the whistles and bells like an integrated editor for the test language, or run time control of the interpreter ( stepping line by line ). As mentioned before: as soon as you can offer 1-3 you will find support soon.

Uwe

[wysota]

For me testing means to check something ( an application, a library or a function ) from outside. As soon as you start looking into internals I would call it debugging.

Then I guess by testing you mean "black box" testing. But there are also other kinds of tests like "white box" testing where the test takes into consideration the structure (internals) of the tested module.

All I wanted to say was, that (IMHO) something like object inspection fits more into a development tool ( like an IDE ), than in a testing tool. Of course it's nothing wrong with having debugging capabilities there too, but I would prefer a more modular design, where the test tool is responsible for creating and executing test scripts only. If it comes to analyzing a situation it's enough if the test tool attaches the application to external tools ( maybe an IDE with an integrated debugger, object inspection or valgrind support ) with full featured "debugging" capabilities.

I don't think I can agree. If I have a widget, say a line edit that is to have yellow background when its value is even and red if its value is odd, generating events is not enough to test whether the test is passed or not. You either need to have a test that will compare screenshots of the widget or you need some way to actually check what is the background color of the widget. And that's where property inspection steps in. A simple "if" statement based on QLineEdit::text and QWidget::palette does the trick. And I wouldn't call that debugging, because then every test that leads to finding a bug might be called debugging and for what other purpose do we do regression tests than to check if code that used to work, continues to work (or not)?

For me the core functionalities are:

1) The definition of a test language.
Squish extends standard script languages, but f.e. SilkTest has it's own 4GL language called 4Test.

2) A event chain generator
Something like what I wrote for the SilkText plugin.

3) An interpreter that translates test scripts into event chains.

With 1-3 you can already do full automated testing.

All that is contained in QTestLib. Did you try it? Did you like it? Do you find anything missing there?

[Uwe]

I don't think I can agree. If I have a widget, say a line edit that is to have yellow background when its value is even and red if its value is odd, generating events is not enough to test whether the test is passed or not.

Sorry, I forgot to mention it, because for analyzing the state of a widget ( properties or non properties ) you can simply use the methods of the class API ( here QLineEdit ). Of course the test language needs to have getter methods, but in opposite to the setters the implementation is trivial.

All that is contained in QTestLib. Did you try it? Did you like it? Do you find anything missing there?

If you want to push a button you first need to move the mouse over the button generating enter/leave/focus-in/out, hover events. A test tool that tries to simulate the interactive part of the user, needs to generate all these events. Simply posting a mouse event like you can do with the QTestLib is by far not the same.

Uwe