Qt & Memory Leak

Printable View

25th October 2013, 14:26
fanoI

Qt & Memory Leak
Hello to everybody,

in my corporation we are using Qt to create an Web Application that emulates a Native Application, that is we use HTLM + JS in a QWebKit but there is no web server, you launch the application on the terminal and you have the illusion to use a Native Application.

This simplified a lot the coding as you have used HTLM + CSS to do the GUI and Javascript + Qt for the engine, but the Application if stressed a lot is killed by the infamous Linux Out of Memory Killer :crying: !

I add this details: the Application's Main Window is an instance of QWebKit in which the page header and the footer are written, the body is the 80% of the screen and is, typically, empty.
In this empty part sometimes a new QWebKit runs with the forms in which the operator write his data.
We have menu that are natives QMenu and messagebox, too that are extended by ourselves from QMessageBox.

A part for the QMenu and QMessageBox all it is created on application star sometime hidden, we simply load new urls in them and then show the resource.

In any cases we lose memory but where?
1. Our classes are in C++ but we have avoided the "new" and "malloc" as if they were Satan, or when we have used them the object had an explicit parent that would have deallocated in the future (Qt does this by design, right?).
2. Java script does some "new" but it is a bad clone of Java, right? So it has the Garbage Collector... the Qt 's javascript interpreter is bugged? It leaks memory?
3. The QMessagebox here we are pretty sure to have found a bug in the QMovie object that we have used to show a wait icon (the classical rotating GIF), when the QMessagebox is destroyed (and it is destroyed when it is closed, right?) the QMovie it is not stopped! A timer continued to run... forever :mad: If we call the delete of the QMovie in the QMessagebox destructor the application crashed for a double free :confused:
4. A Memory leak on the Qt Framework itself
The Qt version is: 4.6.2-2 and runs on Linux Cent OS 5.4 running on kernel 2.6.32.1 that is not the official distribution kernel neither the vanilla one (we have changed it adding statically some drivers).

Thanks for your help we have no ideas anymore...
25th October 2013, 15:42
anda_skoa

Re: Qt & Memory Leak

Since you are running on Linux you could use valgrind's memcheck tool to see if you can narrow down the memory leak.

Cheers,
_
25th October 2013, 15:52
spirit

Re: Qt & Memory Leak
Quote:
Originally Posted by fanoI

Our classes are in C++ but we have avoided the "new" and "malloc" as if they were Satan, or when we have used them the object had an explicit parent that would have deallocated in the future (Qt does this by design, right?).
That's correct.
Quote:
Java script does some "new" but it is a bad clone of Java, right? So it has the Garbage Collector... the Qt 's javascript interpreter is bugged? It leaks memory?
I observed some issues when used QtWebKit Bridge.
Quote:
The QMessagebox here we are pretty sure to have found a bug in the QMovie object that we have used to show a wait icon (the classical rotating GIF), when the QMessagebox is destroyed (and it is destroyed when it is closed, right?) the QMovie it is not stopped! A timer continued to run... forever :mad: If we call the delete of the QMovie in the QMessagebox destructor the application crashed for a double free :confused:
Did you provide parent for QMovie?
Quote:
A Memory leak on the Qt Framework itself

The Qt version is: 4.6.2-2 and runs on Linux Cent OS 5.4 running on kernel 2.6.32.1 that is not the official distribution kernel neither the vanilla one (we have changed it adding statically some drivers).

Thanks for your help we have no ideas anymore...
Well, the version of Qt you are using seems to be old enough. Did you try to use 4.8.5 or Qt5?
As, for memory leaks in Qt - since it uses lazy deletion of objects some profiling tools can count this as leaks.
26th October 2013, 13:26
fanoI

Re: Qt & Memory Leak

@anda_skoa

Our application is really complex as it is a sort of Distributed Process, there is a master process that for you is "The Application" that forks a lot of children: the Gui, the Manager, the Storage, Drivers and so on... We have tried to use Valgrind but it stopped on the first fork :(

On a real "OS" as Windows or Mac OSX would be really easy to find a problem as this as we would have a real debugger, but on Linux?

Quote:

Originally Posted by spirit

That's correct.

OK, so we are doing the right thing!

Quote:

Originally Posted by spirit

I observed some issues when used QtWebKit Bridge.

Well this a good and bad news at the same time! I really suspected this...
So QtWebKit Bridge is bugged and cannot be used on production? How I could say this to my PM, now?

Quote:

Originally Posted by spirit

Did you provide parent for QMovie?

Yes I've created a QLabel in my MessageBox (that is I've passed 'this' to it) where normally it is the icon, then I've created a QMovie as son of the QLabel and called the QMovie's method to start the animation... well when the MessageBox is closed we noted that a timer was continuing to run! In the end we have called "stop" in our QMessageBox implementation destructor, but if I delete the QMovie the application crashes so, the object is destroyed by Qt, but the Movie is not stopped! I see this as a bug or at least bad design on the Qt side :mad:

Quote:

Originally Posted by spirit

Well, the version of Qt you are using seems to be old enough. Did you try to use 4.8.5 or Qt5?

The problem is that the OS we use is really old and we cannot update it as my PM says: "if it works don't fix it!", the problem is that "It don't work and I need to fix it!", the native Qt version was 3! To install Qt 4.6 we needed to create the RPMs ourselves!

By the way if no solution arises I'll try to convince them that update is necessary at least to Qt 4.8.5, for Qt 5 myself fear to much incompatibilities to be honest ;)
The problem will be to found the RPMs... oh well we have compiled it one time, we recompile them again... if only we had time, we are in retard in the schedule... again!

Quote:

Originally Posted by spirit

As, for memory leaks in Qt - since it uses lazy deletion of objects some profiling tools can count this as leaks.

OK, but this is Linux itself that sees the Memory Leaks and kills the application! If you use the Linux top command and make run the software with a simulator you see 2 MB on any iterations are lost... after 12 Hours the application is dead and in /var/log/messages you see "xxx killed by Out Of Memory Killer". If Linux thinks that "lazy deletion of objects" is lost memory... well what we could do?

Thanks for the help, it is really appreciated!
Let me know if you have other ideas :cool:
26th October 2013, 14:14
anda_skoa

Re: Qt & Memory Leak

Quote:

Originally Posted by fanoI

On a real "OS" as Windows or Mac OSX would be really easy to find a problem as this as we would have a real debugger, but on Linux?

If you have better tools on other operating systems then I would suggest to use them.
Not sure how a debugger is going to help you there either though.

I personally have found valgrind to be a very capable tool for finding all kinds of runtime issues and would wish it would be available for Windows as well (it is available for OSX I think).
Since you have subprocesses you could narrow down the problem considerably by just running one of them at a time and check which one is leaking the memory.
Or maybe you can already see that in top and valgrind only the problematic component

Quote:

Originally Posted by fanoI

By the way if no solution arises I'll try to convince them that update is necessary at least to Qt 4.8.5, for Qt 5 myself fear to much incompatibilities to be honest ;)
The problem will be to found the RPMs... oh well we have compiled it one time, we recompile them again... if only we had time, we are in retard in the schedule... again!

You could also just build Qt and deploy it as an application dependency.

Quote:

Originally Posted by fanoI

OK, but this is Linux itself that sees the Memory Leaks and kills the application! If you use the Linux top command and make run the software with a simulator you see 2 MB on any iterations are lost... after 12 Hours the application is dead and in /var/log/messages you see "xxx killed by Out Of Memory Killer". If Linux thinks that "lazy deletion of objects" is lost memory... well what we could do?

The operating system does not see memory leaks, it only sees memory allocations. At some point it can't provide the requested memory anymore.

Cheers,
_
27th October 2013, 14:13
fanoI

Re: Qt & Memory Leak

Quote:

Originally Posted by anda_skoa

If you have better tools on other operating systems then I would suggest to use them.
Not sure how a debugger is going to help you there either though.

If only I could change OS, so easily! My client has already migrated from VxWorks to Cent OS Linux and he has already hundreds of machines installed, yes we could update the OS, this could be done but as you could imagine updating production machines from a tecnological and... well political point of view is not easy :(

Quote:

Originally Posted by anda_skoa

I personally have found valgrind to be a very capable tool for finding all kinds of runtime issues and would wish it would be available for Windows as well (it is available for OSX I think).
Since you have subprocesses you could narrow down the problem considerably by just running one of them at a time and check which one is leaking the memory.
Or maybe you can already see that in top and valgrind only the problematic component

To be more clear they are not suprocesses but real processes the master simply forks them, we call the processes togheter "The Application" but from the OS point of view they are normal applications (they have a main(), I see on ps and top them as all applications, they not share memory and so on), it was no Valgrind the problem it was our IPC that uses the process name to communicate... well it sees Valgrind as all member of "The Application" and all died in atrocious sufference (at least we suffered, "The Application" I don't know... but I hope so :p), we tried efence in the past, it find nothing, sadly... we could try again, maybe it help us this time!

We do know thanks to "top" that is the part of "The Application" that is the Qt interface, but we don't know where is the problem...

Quote:

Originally Posted by anda_skoa

You could also just build Qt and deploy it as an application dependency.

Here I don't understood well; do you intend to compile only the .so and put in a know application directory and not in a system, one? I've understood correctly? Well this could help to found a fast solution in this phase of extreme emergency, yes, I could try to propose it...

We could do the RPMs after as we want to use Qt for others "Applications".

Quote:

Originally Posted by anda_skoa

The operating system does not see memory leaks, it only sees memory allocations. At some point it can't provide the requested memory anymore.

Yes, I do know it was an overly simplification on my part... the application (the Qt one) leaks memory and so Linux kill it!

Cheers,
[/QUOTE]

Thanks and Good Week End,
fano
27th October 2013, 14:45
anda_skoa

Re: Qt & Memory Leak

Quote:

Originally Posted by fanoI

If only I could change OS, so easily! My client has already migrated from VxWorks to Cent OS Linux and he has already hundreds of machines installed, yes we could update the OS, this could be done but as you could imagine updating production machines from a tecnological and... well political point of view is not easy :(

Misunderstanding. I am suggesting to change the OS of the device, I am suggesting to test the application on a different machine, using a different OS and different tools since that is what you seem to have available.

I.e. if your debugger on Windows or OSX can be used to find memory leaks, then run the application on those platforms and use those tools.

Quote:

Originally Posted by fanoI

To be more clear they are not suprocesses but real processes the master simply forks them

That was my understanding as well. That is what we usually refer to as a subprocess or child process.

Quote:

Originally Posted by fanoI

We do know thanks to "top" that is the part of "The Application" that is the Qt interface, but we don't know where is the problem...

You could run the Qt interface in valgrind, no?

Quote:

Originally Posted by fanoI

Here I don't understood well; do you intend to compile only the .so and put in a know application directory and not in a system, one? I've understood correctly?

Yes. You can put the library anywhere you want, you only need to point LD_LIBRARY_PATH to that directory before launching the application.
The runtime linker will look there first before falling back to system locations.

Cheers,
_
31st October 2013, 08:53
fanoI

Re: Qt & Memory Leak

Now the problem has become worst, after 12 hours of simulation the Window Manager (Metacity) dies, X dies and my application not founding where to write dies of SEGFAULT!

The impression is that the MessageBox are the problem as in our log you see that one of them is created, then Metacity dies... the same, identical object was created a 1'000'000 of times before.
I've verified: the MessageBox is created on the stack and so it is deleted when the function returns from the exec() method... so "by design" it shall be only one... but IMHO the WM seems to have a limit to Windows he can show!

He not find "an exit": when the application lost more memory it was more stable, if OOM not killed it... it worked forever!
Now it is the OS itself that dies without an apparent motivation...