It seems that Qt does not release unused memory

[wysota]

An hour ago I implemented a test program that was allocating 15000 QVector<int> objects, filling them with data and reporting the memory usage through reading /proc/pidnum/status, then deallocating them (by deleting the vectors) and repeating the procedure a few times. I can't show you the sources as running the application rendered my computer unresponsive and I had to make a hard reboot (and the program was stored in /tmp). Nevertheless I monitored the memory usage and it was not what I expected. VmRSS (physical memory) stayed the same during subsequent iterations (which was expected), but VmSize (virtual memory) kept rising during each iteration (which wasn't expected) until it ate all the swap space and froze the system solid. All that means one of two (or more?) things - either I made some stupid mistake that caused memory not to be released properly or there is something wrong with memory management either in Qt or the kernel or.... I don't know where (compiler?)... Or I don't know what VmSize is...
The app looked essentially like this (+reporting mem usage everywhere):

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

To copy to clipboard, switch view to plain text mode 

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer. The funny thing is I have 1,2GB swap + 768MB physical memory in my machine which is less than VmSize was reporting as reserved for the process before the system started freezing.

[iw2nhl]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.

Ok, I made a too simple test.
Changing size from 10000 to 1000000 worked as espected. The memory has been allocated and deallocated.
I'll try to make a better test code! Really sorry for my mistake .

Have you tested with other tools at all?

Yes, I tried using directly the /proc dirs.
# cat /proc/pid_num/status

Summing up a simple test works as expected, but my program does not (but only with Linux, with Windows works as expected).
I'm trying to enable/disable some portions of my code to find where the problem arises.

[marcel]

Or I don't know what VmSize is...

It is process swap size + the in-use physical memory.

Regards

[fluxar]

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory. Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

I also thought I had a memleak, but after I tested this (loaded a bunch of other applications to fill the memory) my application freed the memory that it no longer used. Thanks wysota!

[Gopala Krishna]

An hour ago I implemented [..]

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

To copy to clipboard, switch view to plain text mode 

I couldn't understand much on how you extract the info of memory usage but atleast i found this code faulty.
You initially allocate 15k QVector<int> objects(with 10k int each) and push it to "vecs" vector and you just "delete" the 15k vectors in first pass. Note you aren't clearing the pointers in the "vect" vector.
In second pass you append 15k QVector<int> objects i.e now there are 30k pointers in vecs and you delete the first 15k "invalid" pointers which seems wrong.

Probably you could have simplified code like this. (can you try it ? )

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    qDeleteAll(vecs);
    //You missed this clear statement
    vecs.clear();
 }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    qDeleteAll(vecs);
    //You missed this clear statement
    vecs.clear();
 }
}

To copy to clipboard, switch view to plain text mode 

Of course you might have just done a cut and paste mistake !

[wysota]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

[Gopala Krishna]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

I know
I just wanted to say you missed vecs.clear();

[wysota]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

[Gopala Krishna]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

Sorry , but have a look at the code again. vecs is declared before the outermost for loop so it wont get cleared.
I am not sure how it relates here but deleting an invalid pointer does result in unexpected behavior.

[wysota]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer.

[Gopala Krishna]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

That is the reason i put this is in one of the post above

Of course you might have just done a cut and paste mistake !

Anyways back to topic

[wysota]

There was no cut&paste. I had the program written in /tmp which is a ramdisk. And I had to reboot my computer after running the application. Guess the rest