It seems that Qt does not release unused memory

[iw2nhl]

Hi,
I have this problem: I load a lot of data into memory, then I delete it and reload other data, and so on.

Example (actually it is more complex):
---
open file FILE1
elaborate FILE1 data
show results to the user
close file FILE1 and delete elaborated data

open file FILE2
elaborate FILE2 data
show results to the user
close file FILE2 and delete elaborated data
---

If FILE1 data requires 10 MB of RAM and FILE2 30 MB of RAM, I get this sequence:
open FILE1 -> RAM = 10 MB
close FILE1 -> RAM = 10 MB (not released?)
open FILE2 -> RAM = 30 MB
close FILE2 -> RAM = 30 MB (not released?)

If (opposite) FILE1 data requires 30 MB of RAM and FILE2 10 MB of RAM, I get this sequence:
open FILE1 -> RAM = 30 MB
close FILE1 -> RAM = 30 MB (not released?)
open FILE2 -> RAM = 30 MB
close FILE2 -> RAM = 30 MB (not released?)

I am deleting everything, at least I think so, I also made some tests.
Moreover if this is not the case in both the conditions I would end up with 40 MB of RAM occupied, but this never happens.

My understanding is that Qt knows what has been freed and reuses that memory when needed, but it does not release the memory to the OS once it has been allocated.
I saw this behaviour with both Linux and Windows; both Qt3 and Qt4.

Can someone explain me why this behaviour? Is there a way to avoid it?
Thanks for any help!

[marcel]

If FILE1 data requires 10 MB of RAM and FILE2 30 MB of RAM, I get this sequence:
open FILE1 -> RAM = 10 MB
close FILE1 -> RAM = 10 MB (not released?)
open FILE2 -> RAM = 30 MB
close FILE2 -> RAM = 30 MB (not released?)

Opening a file does not implicitly loads the file data in memory.
The operating system just provides a file handle to the calling process.

Therefore the memory leaks you mention are caused by your code.
Maybe you can post the code?

Regards

[iw2nhl]

Sorry, I thought it was clear that:

open FILE1
close FILE1

summed up the operations described above:

open file FILE1
elaborate FILE1 data
show results to the user
close file FILE1 and delete elaborated data

Posting the code is very difficult, it is a big project!
Anyway I'm pretty sure everything is deleted!
I use almost automatic variables and QObjects with parents, so deletion is not necessary. In the cases in which I use pointers to data allocated in the heap (very seldom) I checked the code and there is a delete for every new.

[jpn]

Posting the code is very difficult, it is a big project!

Then perhaps you could prepare a small example which reproduces the problem?

[iw2nhl]

Then perhaps you could prepare a small example which reproduces the problem?

It's not an easy task, because I don't know what part of the program causes it. Anyway I'll try...

I checked also with Valgrind and no memory leaks have been shown.

Then I did this test:
imagine that I need to load FILE1 and FILE2
FILE1 loaded takes 30 MB of RAM, FILE2 loaded takes 10 MB of RAM
now, if I load FILE1 and FILE2, the program occupies 40 MB of RAM
but if I load FILE1 and unload it, then load FILE2 and unload it, the program occupies 30 MB of RAM

This validates my first description: the memory previously used by FILE1 is marked as deleted and reused by FILE1, but never released to the OS.
If there was a memory leak, I would find 40 MB of RAM occupied in both the cases described above, but it's not the case.

[jacek]

How do you measure the memory used by your application?

[iw2nhl]

How do you measure the memory used by your application?

I use KDE System Guard 1.2.0, part of KDE 3.5.7: in the process table I read the columns "VmSize" and "VmRss".
While on Windows 2000 I use "Task Manager", with "VM Size" and "Mem Usage" columns.

I noticed a difference between Linux and Windows: Linux never releases the memory, while Windows releases most of it, but not all.
Eg.:
If the program, just started takes 10 MB of RAM and I load data for 30 MB (total 40 MB), when I unload the data, the program occupies 12 MB. From this point on, when I load and unload a file, it returns always to 12 MB (and never to 10 MB).
On Linux instead it remains always at the peak memory usage reached (in this example 40 MB), irrespective of what I load/unload.

[aMan]

How do you save your data?
In a QList? Do you delete the container or just empty it?

Maybe Qt doesn't frees the memory of the container but keeps it as a buffer.

[wysota]

On Linux instead it remains always at the peak memory usage reached (in this example 40 MB), irrespective of what I load/unload.

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory. Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

[iw2nhl]

Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

This is exactly what is happening.

I could try to use all available memory with another program and see if the memory footprint is reduced.
I'll try it asap!
Thanks

[iw2nhl]

Ok, I loaded files up to use 304 MB of VmSize (about 270-280 MB of VmRss), then I closed all of them and I started opening every sort of program.
The result?
Memory usage of the program was identical for VmSize, while VmRss diminished about of 20-30 MB.
More than 130 MB of Swap memory were used (it was never happened before).

I have 512 MB of RAM in my system and a Swap partition of 258 MB.

I don't know how to interpret the result. Is this a behaviour of the kernel? Is this a Qt bug? Is there something wrong in my code?
I really don't know

[wysota]

In your case VmSize can go up to over 700MB. I think VmRss is the amount of memory really occupied by the application.

http://www.mozilla.org/projects/foot...int-guide.html

BTW. Increase your swap space. It should be about twice as large as the amount of RAM you have in your machine.

[niko]

you can find memory leaks with this very handy tool:
valgrind (http://valgrind.org/)

its fairly easy to use....


niko

[Brandybuck]

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory.

This is true for Linux, but other Unix systems have differing memory allocation behavior. Try your application under FreeBSD or Solaris for a comparison.

[wysota]

Yes, of course. That's natural as every system uses a different allocator implementation. The author said he was running Linux. Furthermore one Linux based system may behave differently than another, it depends on the kernel it is running (both version and options).

[iw2nhl]

you can find memory leaks with this very handy tool:
valgrind (http://valgrind.org/)

Thanks for the suggestion, but I already used it!
See my post #5.
No memory leaks are shown.
Just a few KB caused by Qt and other linked libraries.

[iw2nhl]

I decided to make a very simple test.
It allocates a QVector, then it shows a message so that you can test memory usage, then the vector is deleted and you can test again memory usage.
In my system, both VmRss and VmSize do not change (actually just some KB).
Please, tell me if I'm doing something wrong!

#include <QtGui>
 
int main(int argc, char *argv[]) {
    QApplication app(argc, argv);
 
    // Memory allocation
    QVector<double> *vector = new QVector<double>;
    vector->resize(10000);
    QMessageBox::information(0, "Memory Allocation Test", "Memory allocated");
 
    // Memory deletion
    vector->clear();
    vector->squeeze();
    delete vector;
    QMessageBox::information(0, "Memory Allocation Test", "Memory deleted");
 
    return 0;
}

#include <QtGui>

int main(int argc, char *argv[]) {
    QApplication app(argc, argv);

    // Memory allocation
    QVector<double> *vector = new QVector<double>;
    vector->resize(10000);
    QMessageBox::information(0, "Memory Allocation Test", "Memory allocated");

    // Memory deletion
    vector->clear();
    vector->squeeze();
    delete vector;
    QMessageBox::information(0, "Memory Allocation Test", "Memory deleted");

    return 0;
}

To copy to clipboard, switch view to plain text mode 

[iw2nhl]

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.

double *vector = new double[10000];
delete[] vector;

double *vector = new double[10000];
delete[] vector;

To copy to clipboard, switch view to plain text mode 

[wysota]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.

[marcel]

On linux, what "ps" or "top" have to say?
Have you tested with other tools at all?

What if after closing the files you open an application that takes a lot of memory? Does vmrss drops in any way?
The vmsize is explainable since the virtual memory manager won't adjust the swap every time a process deallocates some memory.

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.

I really doubt that a bug of this magnitude could exist in the memory manager.