It seems that Qt does not release unused memory

[iw2nhl]

Hi,
I have this problem: I load a lot of data into memory, then I delete it and reload other data, and so on.

Example (actually it is more complex):
---
open file FILE1
elaborate FILE1 data
show results to the user
close file FILE1 and delete elaborated data

open file FILE2
elaborate FILE2 data
show results to the user
close file FILE2 and delete elaborated data
---

If FILE1 data requires 10 MB of RAM and FILE2 30 MB of RAM, I get this sequence:
open FILE1 -> RAM = 10 MB
close FILE1 -> RAM = 10 MB (not released?)
open FILE2 -> RAM = 30 MB
close FILE2 -> RAM = 30 MB (not released?)

If (opposite) FILE1 data requires 30 MB of RAM and FILE2 10 MB of RAM, I get this sequence:
open FILE1 -> RAM = 30 MB
close FILE1 -> RAM = 30 MB (not released?)
open FILE2 -> RAM = 30 MB
close FILE2 -> RAM = 30 MB (not released?)

I am deleting everything, at least I think so, I also made some tests.
Moreover if this is not the case in both the conditions I would end up with 40 MB of RAM occupied, but this never happens.

My understanding is that Qt knows what has been freed and reuses that memory when needed, but it does not release the memory to the OS once it has been allocated.
I saw this behaviour with both Linux and Windows; both Qt3 and Qt4.

Can someone explain me why this behaviour? Is there a way to avoid it?
Thanks for any help!

[marcel]

If FILE1 data requires 10 MB of RAM and FILE2 30 MB of RAM, I get this sequence:
open FILE1 -> RAM = 10 MB
close FILE1 -> RAM = 10 MB (not released?)
open FILE2 -> RAM = 30 MB
close FILE2 -> RAM = 30 MB (not released?)

Opening a file does not implicitly loads the file data in memory.
The operating system just provides a file handle to the calling process.

Therefore the memory leaks you mention are caused by your code.
Maybe you can post the code?

Regards

[iw2nhl]

Sorry, I thought it was clear that:

open FILE1
close FILE1

summed up the operations described above:

open file FILE1
elaborate FILE1 data
show results to the user
close file FILE1 and delete elaborated data

Posting the code is very difficult, it is a big project!
Anyway I'm pretty sure everything is deleted!
I use almost automatic variables and QObjects with parents, so deletion is not necessary. In the cases in which I use pointers to data allocated in the heap (very seldom) I checked the code and there is a delete for every new.

[jpn]

Posting the code is very difficult, it is a big project!
Then perhaps you could prepare a small example which reproduces the problem? ;)

[iw2nhl]

Then perhaps you could prepare a small example which reproduces the problem? ;)

It's not an easy task, because I don't know what part of the program causes it. Anyway I'll try...

I checked also with Valgrind and no memory leaks have been shown.

Then I did this test:
imagine that I need to load FILE1 and FILE2
FILE1 loaded takes 30 MB of RAM, FILE2 loaded takes 10 MB of RAM
now, if I load FILE1 and FILE2, the program occupies 40 MB of RAM
but if I load FILE1 and unload it, then load FILE2 and unload it, the program occupies 30 MB of RAM

This validates my first description: the memory previously used by FILE1 is marked as deleted and reused by FILE1, but never released to the OS.
If there was a memory leak, I would find 40 MB of RAM occupied in both the cases described above, but it's not the case.

[jacek]

How do you measure the memory used by your application?

[iw2nhl]

How do you measure the memory used by your application?

I use KDE System Guard 1.2.0, part of KDE 3.5.7: in the process table I read the columns "VmSize" and "VmRss".
While on Windows 2000 I use "Task Manager", with "VM Size" and "Mem Usage" columns.

I noticed a difference between Linux and Windows: Linux never releases the memory, while Windows releases most of it, but not all.
Eg.:
If the program, just started takes 10 MB of RAM and I load data for 30 MB (total 40 MB), when I unload the data, the program occupies 12 MB. From this point on, when I load and unload a file, it returns always to 12 MB (and never to 10 MB).
On Linux instead it remains always at the peak memory usage reached (in this example 40 MB), irrespective of what I load/unload.

[aMan]

How do you save your data?
In a QList? Do you delete the container or just empty it?

Maybe Qt doesn't frees the memory of the container but keeps it as a buffer.

[wysota]

On Linux instead it remains always at the peak memory usage reached (in this example 40 MB), irrespective of what I load/unload.

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory. Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

[iw2nhl]

Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.
This is exactly what is happening.

I could try to use all available memory with another program and see if the memory footprint is reduced.
I'll try it asap!
Thanks

[iw2nhl]

Ok, I loaded files up to use 304 MB of VmSize (about 270-280 MB of VmRss), then I closed all of them and I started opening every sort of program.
The result?
Memory usage of the program was identical for VmSize, while VmRss diminished about of 20-30 MB.
More than 130 MB of Swap memory were used (it was never happened before).

I have 512 MB of RAM in my system and a Swap partition of 258 MB.

I don't know how to interpret the result. Is this a behaviour of the kernel? Is this a Qt bug? Is there something wrong in my code?
I really don't know :(

[wysota]

In your case VmSize can go up to over 700MB. I think VmRss is the amount of memory really occupied by the application.

http://www.mozilla.org/projects/footprint/footprint-guide.html

BTW. Increase your swap space. It should be about twice as large as the amount of RAM you have in your machine.

[niko]

you can find memory leaks with this very handy tool:
valgrind (http://valgrind.org/)

its fairly easy to use....


niko

[Brandybuck]

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory.
This is true for Linux, but other Unix systems have differing memory allocation behavior. Try your application under FreeBSD or Solaris for a comparison.

[wysota]

Yes, of course. That's natural as every system uses a different allocator implementation. The author said he was running Linux. Furthermore one Linux based system may behave differently than another, it depends on the kernel it is running (both version and options).

[iw2nhl]

you can find memory leaks with this very handy tool:
valgrind (http://valgrind.org/)
Thanks for the suggestion, but I already used it!
See my post #5.
No memory leaks are shown.
Just a few KB caused by Qt and other linked libraries.

[iw2nhl]

I decided to make a very simple test.
It allocates a QVector, then it shows a message so that you can test memory usage, then the vector is deleted and you can test again memory usage.
In my system, both VmRss and VmSize do not change (actually just some KB).
Please, tell me if I'm doing something wrong!


#include <QtGui>

int main(int argc, char *argv[]) {
QApplication app(argc, argv);

// Memory allocation
QVector<double> *vector = new QVector<double>;
vector->resize(10000);
QMessageBox::information(0, "Memory Allocation Test", "Memory allocated");

// Memory deletion
vector->clear();
vector->squeeze();
delete vector;
QMessageBox::information(0, "Memory Allocation Test", "Memory deleted");

return 0;
}

[iw2nhl]

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.


double *vector = new double[10000];
delete[] vector;

[wysota]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.

[marcel]

On linux, what "ps" or "top" have to say?
Have you tested with other tools at all?

What if after closing the files you open an application that takes a lot of memory? Does vmrss drops in any way?
The vmsize is explainable since the virtual memory manager won't adjust the swap every time a process deallocates some memory.



I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.
I really doubt that a bug of this magnitude could exist in the memory manager.

[wysota]

An hour ago I implemented a test program that was allocating 15000 QVector<int> objects, filling them with data and reporting the memory usage through reading /proc/pidnum/status, then deallocating them (by deleting the vectors) and repeating the procedure a few times. I can't show you the sources as running the application rendered my computer unresponsive and I had to make a hard reboot (and the program was stored in /tmp). Nevertheless I monitored the memory usage and it was not what I expected. VmRSS (physical memory) stayed the same during subsequent iterations (which was expected), but VmSize (virtual memory) kept rising during each iteration (which wasn't expected) until it ate all the swap space and froze the system solid. All that means one of two (or more?) things - either I made some stupid mistake that caused memory not to be released properly or there is something wrong with memory management either in Qt or the kernel or.... I don't know where (compiler?)... Or I don't know what VmSize is...
The app looked essentially like this (+reporting mem usage everywhere):


int main(){
QVector<QVector<int>*> vecs;
for(int r=0;r<3;r++){
for(int i=0;i<15000;i++){
QVector<int> *v = new QVector<int>;
for(int j=0;j<10240;j++)
(*v) << 7;
vecs << v; // or was it vecs[i] = *v; ?
}
for(int i=0;i<15000;i++){
delete vecs[i];
}
}
}

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer. The funny thing is I have 1,2GB swap + 768MB physical memory in my machine which is less than VmSize was reporting as reserved for the process before the system started freezing.

[iw2nhl]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.
Ok, I made a too simple test.
Changing size from 10000 to 1000000 worked as espected. The memory has been allocated and deallocated.
I'll try to make a better test code! Really sorry for my mistake :o.


Have you tested with other tools at all?
Yes, I tried using directly the /proc dirs.
# cat /proc/pid_num/status

Summing up a simple test works as expected, but my program does not (but only with Linux, with Windows works as expected).
I'm trying to enable/disable some portions of my code to find where the problem arises.

[marcel]

Or I don't know what VmSize is...

It is process swap size + the in-use physical memory.

Regards

[fluxar]

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory. Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

I also thought I had a memleak, but after I tested this (loaded a bunch of other applications to fill the memory) my application freed the memory that it no longer used. Thanks wysota!

[Gopala Krishna]

An hour ago I implemented [..]


int main(){
QVector<QVector<int>*> vecs;
for(int r=0;r<3;r++){
for(int i=0;i<15000;i++){
QVector<int> *v = new QVector<int>;
for(int j=0;j<10240;j++)
(*v) << 7;
vecs << v; // or was it vecs[i] = *v; ?
}
for(int i=0;i<15000;i++){
delete vecs[i];
}
}
}



I couldn't understand much on how you extract the info of memory usage but atleast i found this code faulty.
You initially allocate 15k QVector<int> objects(with 10k int each) and push it to "vecs" vector and you just "delete" the 15k vectors in first pass. Note you aren't clearing the pointers in the "vect" vector.
In second pass you append 15k QVector<int> objects i.e now there are 30k pointers in vecs and you delete the first 15k "invalid" pointers which seems wrong.

Probably you could have simplified code like this. (can you try it ? )


int main(){
QVector<QVector<int>*> vecs;
for(int r=0;r<3;r++){
for(int i=0;i<15000;i++){
QVector<int> *v = new QVector<int>;
for(int j=0;j<10240;j++)
(*v) << 7;
vecs << v; // or was it vecs[i] = *v; ?
}
qDeleteAll(vecs);
//You missed this clear statement
vecs.clear();
}
}
Of course you might have just done a cut and paste mistake ! ;)

[wysota]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

[Gopala Krishna]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

I know :)
I just wanted to say you missed vecs.clear();

[wysota]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

[Gopala Krishna]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

Sorry , but have a look at the code again. vecs is declared before the outermost for loop so it wont get cleared. :)
I am not sure how it relates here but deleting an invalid pointer does result in unexpected behavior.

[wysota]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer.

[Gopala Krishna]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

That is the reason i put this is in one of the post above

Of course you might have just done a cut and paste mistake !
Anyways back to topic :)

[wysota]

There was no cut&paste. I had the program written in /tmp which is a ramdisk. And I had to reboot my computer after running the application. Guess the rest :)