It seems that Qt does not release unused memory

[marcel]

On linux, what "ps" or "top" have to say?
Have you tested with other tools at all?

What if after closing the files you open an application that takes a lot of memory? Does vmrss drops in any way?
The vmsize is explainable since the virtual memory manager won't adjust the swap every time a process deallocates some memory.

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.

I really doubt that a bug of this magnitude could exist in the memory manager.

[wysota]

An hour ago I implemented a test program that was allocating 15000 QVector<int> objects, filling them with data and reporting the memory usage through reading /proc/pidnum/status, then deallocating them (by deleting the vectors) and repeating the procedure a few times. I can't show you the sources as running the application rendered my computer unresponsive and I had to make a hard reboot (and the program was stored in /tmp). Nevertheless I monitored the memory usage and it was not what I expected. VmRSS (physical memory) stayed the same during subsequent iterations (which was expected), but VmSize (virtual memory) kept rising during each iteration (which wasn't expected) until it ate all the swap space and froze the system solid. All that means one of two (or more?) things - either I made some stupid mistake that caused memory not to be released properly or there is something wrong with memory management either in Qt or the kernel or.... I don't know where (compiler?)... Or I don't know what VmSize is...
The app looked essentially like this (+reporting mem usage everywhere):

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

To copy to clipboard, switch view to plain text mode 

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer. The funny thing is I have 1,2GB swap + 768MB physical memory in my machine which is less than VmSize was reporting as reserved for the process before the system started freezing.

[Gopala Krishna]

An hour ago I implemented [..]

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

To copy to clipboard, switch view to plain text mode 

I couldn't understand much on how you extract the info of memory usage but atleast i found this code faulty.
You initially allocate 15k QVector<int> objects(with 10k int each) and push it to "vecs" vector and you just "delete" the 15k vectors in first pass. Note you aren't clearing the pointers in the "vect" vector.
In second pass you append 15k QVector<int> objects i.e now there are 30k pointers in vecs and you delete the first 15k "invalid" pointers which seems wrong.

Probably you could have simplified code like this. (can you try it ? )

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    qDeleteAll(vecs);
    //You missed this clear statement
    vecs.clear();
 }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    qDeleteAll(vecs);
    //You missed this clear statement
    vecs.clear();
 }
}

To copy to clipboard, switch view to plain text mode 

Of course you might have just done a cut and paste mistake !

[wysota]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

[Gopala Krishna]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

I know
I just wanted to say you missed vecs.clear();

[wysota]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

[Gopala Krishna]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

Sorry , but have a look at the code again. vecs is declared before the outermost for loop so it wont get cleared.
I am not sure how it relates here but deleting an invalid pointer does result in unexpected behavior.

[wysota]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer.

[Gopala Krishna]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

That is the reason i put this is in one of the post above

Of course you might have just done a cut and paste mistake !

Anyways back to topic

[wysota]

There was no cut&paste. I had the program written in /tmp which is a ramdisk. And I had to reboot my computer after running the application. Guess the rest