PDA

View Full Version : QHash : Virtual memory not released



jbenoit
13th November 2009, 16:54
Hi,

I am using a QHash and I am not sure how to release the virtual memory used by the QHash.

Here is the code of a function using a QHash.



void testMemory(){
QMap<long , int> distances;
for(long x=0; x<2073600; x++){
distances.insert(x, 2);
}
distances.clear();
}


Here is the code to test the memory before and after the call to testMemory() function.



...
#define DIV 1024
#define WIDTH 7
...
MEMORYSTATUSEX statex;
statex.dwLength = sizeof (statex);
GlobalMemoryStatusEx (&statex);
printf ("There is %*ld percent of memory in use.\n", WIDTH, statex.dwMemoryLoad);
printf ("There are %*I64d total Kbytes of physical memory.\n", WIDTH, statex.ullTotalPhys/DIV);
printf ("There are %*I64d free Kbytes of physical memory.\n", WIDTH, statex.ullAvailPhys/DIV);
printf ("There are %*I64d total Kbytes of paging file.\n", WIDTH, statex.ullTotalPageFile/DIV);
printf ("There are %*I64d free Kbytes of paging file.\n", WIDTH, statex.ullAvailPageFile/DIV);
printf ("There are %*I64d total Kbytes of virtual memory.\n", WIDTH, statex.ullTotalVirtual/DIV);
printf ("There are %*I64d free Kbytes of virtual memory.\n", WIDTH, statex.ullAvailVirtual/DIV);

cout << endl << "Call testMem function..." << endl << endl;

testMemory();

statex.dwLength = sizeof (statex);
GlobalMemoryStatusEx (&statex);
printf ("There is %*ld percent of memory in use.\n", WIDTH, statex.dwMemoryLoad);
printf ("There are %*I64d total Kbytes of physical memory.\n", WIDTH, statex.ullTotalPhys/DIV);
printf ("There are %*I64d free Kbytes of physical memory.\n", WIDTH, statex.ullAvailPhys/DIV);
printf ("There are %*I64d total Kbytes of paging file.\n", WIDTH, statex.ullTotalPageFile/DIV);
printf ("There are %*I64d free Kbytes of paging file.\n", WIDTH, statex.ullAvailPageFile/DIV);
printf ("There are %*I64d total Kbytes of virtual memory.\n", WIDTH, statex.ullTotalVirtual/DIV);
printf ("There are %*I64d free Kbytes of virtual memory.\n", WIDTH, statex.ullAvailVirtual/DIV);
...


Here is the console output for this test :



There is 59 percent of memory in use.
There are 2095848 total Kbytes of physical memory.
There are 843764 free Kbytes of physical memory.
There are 4037636 total Kbytes of paging file.
There are 2799952 free Kbytes of paging file.
There are 2097024 total Kbytes of virtual memory.
There are 2062472 free Kbytes of virtual memory.

Call testMem function...

There is 59 percent of memory in use.
There are 2095848 total Kbytes of physical memory.
There are 845320 free Kbytes of physical memory.
There are 4037636 total Kbytes of paging file.
There are 2799572 free Kbytes of paging file.
There are 2097024 total Kbytes of virtual memory.
There are 1801284 free Kbytes of virtual memory.



As you can see, there are many Kbytes of virtual memory that aren't released after the function.

What I do wrong? How can I change the testMemory() function to free all virtual memory?

What did I miss?

Note : if you comment the line 4 of the testMemory() function (the line that insert an item), all the virtual memory is released correctly.

wysota
13th November 2009, 17:51
You didn't do anything wrong. The memory will be released when the system requires it for something else. Until then it will be kept assigned to your process in case it wants to allocate it again. Don't worry about it, the memory has been released properly, you don't even need to clear the map as it will be destroyed when the function returns.

squidge
13th November 2009, 18:21
If you put the memory check in your loop you'll find that it doesn't allocate memory for every single insert, instead it allocates and frees in chunks. Due to this the memory may not be freed directly after the last free, but you can be sure it will be freed eventually.

If you need to check this you can run the memory test before and afterwards, and then cycle the loop a few hundred times. You'll notice the memory does not increase anymore after the first cycle as it will just reuse the memory from before.

wysota
13th November 2009, 18:30
If you put the memory check in your loop you'll find that it doesn't allocate memory for every single insert, instead it allocates and frees in chunks. Due to this the memory may not be freed directly after the last free, but you can be sure it will be freed eventually.

It will be freed immediately (it's C++, not Java, after all). It will just not be unassigned from the process in case it wants to allocate it again.

You can disable swap and allocate more than all the free memory on your system using another process, you will notice the memory assigned to the tested process will decrease as it will be allocated to the "vacuum process". This is actually a known way to defragment memory.

You also have to distinguish between assignment and allocation. You can allocate chunks of memory (just as QMap does) and you will see the amount of allocated memory increase but they will come from the same virtual page of memory assigned to your process. The OS always assigns the whole page, even if you allocate just one byte of memory. This is not something you can test with userspace tools.

jbenoit
13th November 2009, 18:42
First of all, thank you both for your quick answer. This is really appreciated.



... but you can be sure it will be freed eventually. ...


Maybe this is my problem.

Right after the use of the QHash, i need to allocate a huge block a continuous memory (I am using CvMat from openCV).

While my virtual memory seems all available (from task manager), I got an "insufficient memory error".


Here a code example :


int nbDim = 375;
int maxPts = 1073741700/nbDim;
CvMat* dataPts = cvCreateMat( maxPts , nbDim, CV_8U );
cvReleaseMat(&dataPts);

testMemory();

dataPts = cvCreateMat( maxPts , nbDim, CV_8U );
cvReleaseMat(&dataPts);

The first cvCreateMat pass while the second popup a error message : "not enough memory".

How can I know that the memory have been released and that I can creat the new matrice ?

jbenoit
13th November 2009, 18:49
You can disable swap and allocate more than all the free memory on your system using another process, you will notice the memory assigned to the tested process will decrease as it will be allocated to the "vacuum process". This is actually a known way to defragment memory.

Can you give me a reference webpage on how to do this ?

squidge
13th November 2009, 19:03
It will be freed immediately (it's C++, not Java, after all). It will just not be unassigned from the process in case it wants to allocate it again.Maybe I'm showing my age now and things are not done as they used to be done before, but I'm used to the fact that you free() or delete some memory and the memory is returned to the heap for the process, and not to the system. Only when the amout of free memory hit a predefined figure was the memory actually freed to the system, to prevent memory fragmentation. This was when the C compiler ran from a floppy disk however :)

jbenoit: "eventually" will not stop you from allocating further memory. The reserved memory will be used before more is requested from the system. I doesn't hold on to memory knowing you want it for something else.

As for vacuum processes, all you have to do is a loop allocating 100KB at a time until malloc() returns NULL. At that point you have allocated all that is possible from a single process (which depends on page file size, OS, 32/64bit etc). Or you can buy a product that does this for you for between $20 and $99 or so. It'll come with a flashy GUI and lots of flashing lights, but behind the scenes, it does exactly the same as you can do in 5 minutes by writing it yourself.

Defragmenting virtual memory is a nonsense idea anyway, even if you have a 4GB contiguous chunk of memory, theres nothing stopping the OS mapping those contiguous chunks to a random chunks of physical memory. The only way of getting a chunk of contiguous physical memory is by dropping to the device driver ring (ring 0) and requesting it there.

jbenoit
13th November 2009, 20:07
I understand what you are saying, theorically...

Pratically, I am not sure where to begin.

My problem look like a fragmentation problem.

So, how can I defragment my memory in my application ? Pratically speeking...

To be more precise :

All in c++, using QT, on Windows OS

1) In c++, how can I disabled swap during run-time ?

2) In c++, how can I allocate more than all the free memory on your system using another process during run-time?

squidge
13th November 2009, 21:30
Both 1 and 2 are a bad idea for a typical application, so don't do either.

jbenoit
13th November 2009, 21:53
Ok. So, what can I do ?

There is what I want *need* to do in my app :

1) Allocate a large block of continous memory (cvCreateMat, from openCV)
2) I need to allocate many small block (QHash, for example, and others things) that mess up my fragmentation.
3) Deallocated the memory from 1)
4) Allocate large block of memory for PCA analysis.
5) Deallocated memory from 2) and 4)


And I need to repeat step 1 to 5 a couple of times.

The problem is that, when I try to redo step 1 for the second time, my memory is fragmented and I get a "inssuficient memory".

If I don't do step 3 (in order to reuse the matrice for the next iteration), I got an error on step 4 : "insufficient memor".


Any clues on what should I do ?

squidge
13th November 2009, 22:03
Which is the larger amount of memory required - cvCreateMat or the PCA analysis? Allocate it once and use it for both? Why deallocate when your going to need it again shortly anyway? Even if it is for another task.

wysota
13th November 2009, 22:27
Defragmenting virtual memory is a nonsense idea anyway, even if you have a 4GB contiguous chunk of memory, theres nothing stopping the OS mapping those contiguous chunks to a random chunks of physical memory.
That's not exactly true. The idea of defragmenting memory is that most of memory of other processes gets pushed to swap and physical memory gets released. Then you can request a big chunk of memory that otherwise might be problematic due to swapping and the fact that memory is allocated in pages and not bytes. Consider the fact that you have 256 units of memory in total and page size is 16 units. If some process requests one unit of memory, 16 units get assigned to it and other processes can only request 240 units of memory without swapping. If you force the other process to be preemptied and its memory moved to swap, you can allocate 256 units of physical memory instead of 240 units of physical memory and 16 units of swap space. What you gain is that the swapping process is moved in time and you get a continuus chunk of physical memory with perspectives of faster access to physical memory. At least in theory. In practice there are other processes running and interacting with your process so it all becomes less predictable.

As for the original problem, find the border case where the allocation suceeds and make sure the hash is destroyed before that and that all values are correct (watch out for data type limitations).

squidge
13th November 2009, 23:03
Yes, the only real difference is when the delay occurs to swap data to disk - when you allocate memory for no reason to flush the system, or when you allocate the memory that you actually want. It doesn't make much (if any) difference to how much memory you can get.

I do agree however that sometimes you have a problem when the system is busy moving memory around that a memory allocation may fail (this is documented in the MSDN), but you shouldn't try to allocate huge amounts of memory in your app just to get everyone else to purge there buffers to swap space.

To an application, it has no idea whether a contiguous area of memory is physically contiguous or only logically contiguous, as each process has its own memory range which the OS chooses itself where to put in physical memory, and its only when you write to this memory does it actually get allocated (See below).

The other problem is that modern OSs use a "Copy on write" strategy, so they'll quite happily let you allocate a few GB of memory, but will not actually allocate it from the physicall memory store until you store something in it. This is why programs sometimes crash without warning - you write into another page of memory, the OS tries to allocate that page, finds out there no free physically memory left and will then attempt to move other memory to the swap file. If the swap file is full too, the process gets terminated unless its a critical system process, in which case it will pick a process and kill it using a certain algorithm (normally preferrring the big memory eaters)

Try it on Linux without a swap file, you'll find that a system with 512MB of RAM will happily let several processes allocate several GB of memory. Then write to that memory and watch the system kill you.

wysota
14th November 2009, 10:02
It doesn't make much (if any) difference to how much memory you can get.
It doesn't make difference how much virtual memory you can get. Things are different with physical memory (and on systems that have swap disabled it gets more significant as you can't use all the memory in the system).


but you shouldn't try to allocate huge amounts of memory in your app just to get everyone else to purge there buffers to swap space.
True but there are other things one shouldn't do or be able to do but Windows still does it (i.e. FindWindow/PostMessage to every possible window in the system).

squidge
14th November 2009, 13:19
True but there are other things one shouldn't do or be able to do but Windows still does it (i.e. FindWindow/PostMessage to every possible window in the system).It seems in the latest version (Windows 7), security has been tightened up quite a bit, from the docs it seems that you can no longer post to another applications message loop. You have to request such functionality first now, both in your manifest file and via the API, which causes the UAC to prompt the user.

But thats not going to help the OP with there memory problem. Maybe they can run the application under OllyDbg, that way you can see the entire memory map for the process and see exactly where everything is being put. To use it properly however, it does help to have some knowledge of the x86 instruction set, as thats its primary output mode.

wysota
14th November 2009, 14:56
I'm still waiting for him to detect the border case. I doubt the hash has anything to do with it.

squidge
14th November 2009, 17:35
I agree, once the hash has been freed, the memory should be pretty much the same as before.