Open source automated testing tool for Qt App

[Uwe]

So what tests do you consider "automated"?

Automated means, that execution and verification of a test is done unattended.

The people running the test suite often don't know much about the application, they are testing. I've met QA engineers that were excellent developers too, but more often you find people with a very different background. F.e. in my current project the QA job is done offshore by collegues who don't know what UNIX or Qt is.

They push a button to start and after it is finished, they have a report with a summary which tests passed or failed. Analyzing the failed tests is not part of their job.

Uwe

[wysota]

Automated means, that execution and verification of a test is done unattended.

They push a button to start and after it is finished, they have a report with a summary which tests passed or failed. Analyzing the failed tests is not part of their job.

So it doesn't determine what is tested, only how it is tested. I don't see why regressive tests created using point and click are not to be called automated even if they are then executed from within an automated script...

[Uwe]

I don't see why regressive tests created using point and click are not to be called automated even if they are then executed from within an automated script...

Of course testing with an automated script is automated testing. It is of no importance if it is started from the command line, as cron job, or by "pushing a button" in a test environment.

I'm afraid I can't follow, why we got lost in discussions about terminologies. Isn't it the difference between automated/manual testing obvious for everyone ?

Uwe

[wysota]

Yes, but it's not clear for me why you called what Squish does (at least the part related to the features I mentioned - by the way, have you seen the demo?) as "debugging"

[Uwe]

Yes, but it's not clear for me why you called what Squish does (at least the part related to the features I mentioned - by the way, have you seen the demo?) as "debugging"

For me testing means to check something ( an application, a library or a function ) from outside. As soon as you start looking into internals I would call it debugging. But this might be completely wrong and I don't care much about such a definition.

All I wanted to say was, that (IMHO) something like object inspection fits more into a development tool ( like an IDE ), than in a testing tool. Of course it's nothing wrong with having debugging capabilities there too, but I would prefer a more modular design, where the test tool is responsible for creating and executing test scripts only. If it comes to analyzing a situation it's enough if the test tool attaches the application to external tools ( maybe an IDE with an integrated debugger, object inspection or valgrind support ) with full featured "debugging" capabilities.

Please recall, that this thread is about writing an open source automated testing tool. It's much easier to start with the core functionality instead of trying to include too much at the beginning. For me the core functionalities are:

1) The definition of a test language.
Squish extends standard script languages, but f.e. SilkTest has it's own 4GL language called 4Test.

2) A event chain generator
Something like what I wrote for the SilkText plugin.

3) An interpreter that translates test scripts into event chains.

With 1-3 you can already do full automated testing.

In the next step you could add a GUI around it with all the whistles and bells like an integrated editor for the test language, or run time control of the interpreter ( stepping line by line ). As mentioned before: as soon as you can offer 1-3 you will find support soon.

Uwe

[wysota]

For me testing means to check something ( an application, a library or a function ) from outside. As soon as you start looking into internals I would call it debugging.

Then I guess by testing you mean "black box" testing. But there are also other kinds of tests like "white box" testing where the test takes into consideration the structure (internals) of the tested module.

All I wanted to say was, that (IMHO) something like object inspection fits more into a development tool ( like an IDE ), than in a testing tool. Of course it's nothing wrong with having debugging capabilities there too, but I would prefer a more modular design, where the test tool is responsible for creating and executing test scripts only. If it comes to analyzing a situation it's enough if the test tool attaches the application to external tools ( maybe an IDE with an integrated debugger, object inspection or valgrind support ) with full featured "debugging" capabilities.

I don't think I can agree. If I have a widget, say a line edit that is to have yellow background when its value is even and red if its value is odd, generating events is not enough to test whether the test is passed or not. You either need to have a test that will compare screenshots of the widget or you need some way to actually check what is the background color of the widget. And that's where property inspection steps in. A simple "if" statement based on QLineEdit::text and QWidget::palette does the trick. And I wouldn't call that debugging, because then every test that leads to finding a bug might be called debugging and for what other purpose do we do regression tests than to check if code that used to work, continues to work (or not)?

For me the core functionalities are:

1) The definition of a test language.
Squish extends standard script languages, but f.e. SilkTest has it's own 4GL language called 4Test.

2) A event chain generator
Something like what I wrote for the SilkText plugin.

3) An interpreter that translates test scripts into event chains.

With 1-3 you can already do full automated testing.

All that is contained in QTestLib. Did you try it? Did you like it? Do you find anything missing there?

[Uwe]

I don't think I can agree. If I have a widget, say a line edit that is to have yellow background when its value is even and red if its value is odd, generating events is not enough to test whether the test is passed or not.

Sorry, I forgot to mention it, because for analyzing the state of a widget ( properties or non properties ) you can simply use the methods of the class API ( here QLineEdit ). Of course the test language needs to have getter methods, but in opposite to the setters the implementation is trivial.

All that is contained in QTestLib. Did you try it? Did you like it? Do you find anything missing there?

If you want to push a button you first need to move the mouse over the button generating enter/leave/focus-in/out, hover events. A test tool that tries to simulate the interactive part of the user, needs to generate all these events. Simply posting a mouse event like you can do with the QTestLib is by far not the same.

Uwe