Open source automated testing tool for Qt App

[Uwe]

You won't find one and the community won't build one because it's a too big effort for anybody who handles open source software that is usually not tested at all (excluding some simple unit testing)...

Yes Squish is the only real testing tool for Qt. There is also KD Executor, but AFAIR it is only a record/replay tool. Maybe it's possible to find other record/replay tools, that interfere at a level below Qt ( f.e X11 ).

I totally agree with you, that a testing tool is more or less for the commercial world only ( maybe beside huge projects like KDE ), but it is an interesting project and a substantial gap in the free Qt tool chain.

About the effort: a couple of years ago I wrote a Qt plugin for SilkTest (http://en.wikipedia.org/wiki/SilkTest). Writing such a plugin means more or less, breaking down high level operations into event chains. I needed ~1 month to have something that covered ~70% of the Qt3 widgets.
Beside this you need to embed a script language and to write the application around it.

Of course writing such a testing tool is nothing for beginners, but it is not more difficult than writing an IDE. If I could chose between seeing a good testing tool or one more IDE ...

Uwe

[wysota]

Yes Squish is the only real testing tool for Qt. There is also KD Executor, but AFAIR it is only a record/replay tool.

A very nice tool, I might add Unfortunately commercial as well.

Maybe it's possible to find other record/replay tools, that interfere at a level below Qt ( f.e X11 ).

Possibly, but they won't be able to test some Qt quirks which can't be mapped onto X events.

I totally agree with you, that a testing tool is more or less for the commercial world only ( maybe beside huge projects like KDE ), but it is an interesting project and a substantial gap in the free Qt tool chain.

Oh yes, definitely. Unfortunately there is not much we can do about it. Take a look at the IDE projects we have and their state and cooperation. Then realize that a testing tool is much more complex and requires deep knowledge of Qt. I'm really fond of Squish, unfortunately it's ugly and expensive.

Beside this you need to embed a script language and to write the application around it.

Right now that's not a problem as Qt gives us ECMAScript for free and using Python is quite straightforward as well.

Of course writing such a testing tool is nothing for beginners, but it is not more difficult than writing an IDE. If I could chose between seeing a good testing tool or one more IDE ...

I'm not sure I can agree with that. An IDE doesn't really deal with platform dependent quirks like controlling the AUT which I belive is different depending on the platform.

[Uwe]

I'm not sure I can agree with that. An IDE doesn't really deal with platform dependent quirks like controlling the AUT which I belive is different depending on the platform.

Well, the communication between the user and the application is done by Qt events. In my SilkTest plugin I wrote a class that mapped the basic mouse and keyboard operations into Qt event chains. This was the difficult part, as you have to take care of all these enter/leave/focus-in/out stuff.
Then I wrote a proxy class for each Qt widget, that exported high level operations ( like PushButtonProxy:ress() ) to the testscript language. The implementation of these operation was simply a sequence of delays and the mouse/keyboard operations, I had implemented before.

I'm pretty sure, that there are use cases, that can't be handled with such a design, but it is completely platform independent ( pure Qt code ) and should cover most of the usual ways of scripted testing.

Writing an application around it is still a lot of work, but standard Qt/GUI coding. In the end you have a "Poor Man's Test Tool" for Qt applications, that is not complete, but very useful - and as soon as you can offer a working tool, it will be improved ( or replaced by something better ) by the community.

Uwe

[wysota]

Squish has nice features such as when you can point the widget to be selected using mouse with a red border around it, you can inspect objects of the application, etc.

I don't know how exactly the tool you mentioned works, but I suspect obtaining such capabilities with it would be very difficult. Translating events is one thing and its the least such a gui testing tool must do, but it's the added value that makes a tool unique. Look at IDEs - one might say that to write an IDE you need a text editor and buttons to perform compilation and execution and that's basically true, but it's all those little things that make tools such as Visual Studio, Eclipse or KDevelop loved by many.

[Uwe]

Squish has nice features such as when you can point the widget to be selected using mouse with a red border around it, you can inspect objects of the application, etc.

I don't know how exactly the tool you mentioned works, but I suspect obtaining such capabilities with it would be very difficult.

By far the most important thing about a testing tool is, that you can implement automated regression tests. ( This is something you can do with Squish, but not with simple tools like KD Executor. ) My implementation was only for this aspect.

In commercial projects you often find a QA team, that implements these test suites and runs them for every new release of your product. I have worked in an environment, where these test suites ran unattended every night on the development branch, sending the developer a mail, when a test fails.

It is very important, that you can implement your test scripts on an abstract level, to avoid that they break, with every layout change in a new release. That's why record/replay is of limited use. All manual testing is of low importance ( you don't need a tool for this) and debugging functionalities, like the one you have mentioned, should be part of a debugging tool ( what Squish seems to be too ).

Uwe

[wysota]

I think you misunderstood me. The features I was talking about were not debugging functionality. They are part of the test creation process. For instance you have an application that is a calculator and you want to test addition. So you create a test where you press "2", "+", "2", "=" and then you want to check whether you obtained "4" as the text of a proper QLineEdit. To do that you need to be able to read properties of widgets and you need to be able to choose widgets that are interesting for you. Take a look at the demo video available on Froglogic site - it demonstrates the use of those features.

[Uwe]

To do that you need to be able to read properties of widgets and you need to be able to choose widgets that are interesting for you.

Well, in the Qt3 solutions package was a Qt Object Inspector
( http://doc.trolltech.com/solutions/q.../overview.html ) and AFAIR there were other tools around dumping the meta object. It's one of these features, that are adequate for demos, because they show something nice, without having to implement much.

I don't want to insist on a definition of debugging and testing, but I'm sure a QA engineer wouldn't debug/test properties. And to be honest: I never felt, that I ( as developer) need a tool like this, when I have a real debugger.

But let me repeat: a test tool is about automated testing.

Uwe

[wysota]

But let me repeat: a test tool is about automated testing.

So what tests do you consider "automated"? Because the case I'm talking about (with properties, etc.) is definitely regression testing and I'm pretty sure it is automated.

The problem with Solutions is that the code is licenced and we are talking about an open source approach to the problem. Belive me, I would gladly see an open source automated testing tool. I just don't see anyone willing to spend that much of his time writing it - it's tedious and open source developers tend not to touch tedious work.

One more thing - the functionality (I think) you find required for a test tool is contained in QTestLib. So basically one could make a frontend for it that would generate a test program and then compile it. You could even run it from within the frontend and inspect results as they are given in XML.

[Uwe]

So what tests do you consider "automated"?

Automated means, that execution and verification of a test is done unattended.

The people running the test suite often don't know much about the application, they are testing. I've met QA engineers that were excellent developers too, but more often you find people with a very different background. F.e. in my current project the QA job is done offshore by collegues who don't know what UNIX or Qt is.

They push a button to start and after it is finished, they have a report with a summary which tests passed or failed. Analyzing the failed tests is not part of their job.

Uwe

[wysota]

Automated means, that execution and verification of a test is done unattended.

They push a button to start and after it is finished, they have a report with a summary which tests passed or failed. Analyzing the failed tests is not part of their job.

So it doesn't determine what is tested, only how it is tested. I don't see why regressive tests created using point and click are not to be called automated even if they are then executed from within an automated script...

[Uwe]

I don't see why regressive tests created using point and click are not to be called automated even if they are then executed from within an automated script...

Of course testing with an automated script is automated testing. It is of no importance if it is started from the command line, as cron job, or by "pushing a button" in a test environment.

I'm afraid I can't follow, why we got lost in discussions about terminologies. Isn't it the difference between automated/manual testing obvious for everyone ?

Uwe

[wysota]

Yes, but it's not clear for me why you called what Squish does (at least the part related to the features I mentioned - by the way, have you seen the demo?) as "debugging"