Hitting memory limits for malloc() at about 1.7GB?

[Skywalker]

I have Vista 64 Ultimate and installed the latest Qt SDK, 2009.03. I seem to be hitting memory limits when using malloc() even though I have 8GB of physical RAM, and task manager shows only 2GB of it is used. My program crashes when I try to allocate large amounts of memory.

The exact limit seems to vary, but it seems to be about 1.74GB. The following code causes my program to crash:

#include <iostream>
#include <qglobal.h>
 
using namespace std;
 
int main(int argc, char *argv[]){
	double* test = (double*) malloc(1863000000);
	cout<<"Test"<<endl;
	test[0] = 1.0;
}

#include <iostream>
#include <qglobal.h>

using namespace std;

int main(int argc, char *argv[]){
	double* test = (double*) malloc(1863000000);
	cout<<"Test"<<endl;
	test[0] = 1.0;
}

To copy to clipboard, switch view to plain text mode 

With a slightly smaller number the program executes properly. I don't think Vista Ultimate has a memory limit for processes. Is the latest Qt compiler 64-bit?

How can I utilize my RAM to its fullest extent using malloc() and Qt?

[wysota]

Qt has nothing to do with malloc(). The latter is a C function which is implemented in the C runtime and calls the system routine deep inside the kernel of the operating system. It must be your kernel that is preventing the allocation and I'd say you simply have a limit on a continuous chunk that can be allocated. What happens if you try to allocate that memory is two chunks half that size each?

[Skywalker]

I tried allocating 4 chunks of 500MB each. The program crashes when trying to write values to the 4th chunk. Then I tried 8x250MB and it crashed on the 7th block. At 25x80MB it crashed trying to write to the 25th. I also tried 100x40MB but it crashes writing to the 51st.

So breaking into smaller blocks seems to help, but only slightly, and I keep hitting a limit right around 2GB.

The program I'm trying to make has to read a couple of 800MB files into memory, then create some arrays/matrices for numerical analysis. It actually worked on a Linux system but crashes on Windows while trying to allocate one of the arrays. And the entire reason I moved to my home system was because it was 64bit with 8GB RAM I was running out of RAM on the Linux system.

Are there maybe any other reasons this could be happening, and things I can do to get around it?

[wysota]

I don't know how it is in 64 bit Vista but in 32 bit version there is a limit set on the maximum amount of memory a process can allocate. There is a system command to override it but I don't remember it right now. I suggest browsing Vista programming forums for solutions. Do one more thing - what happens if you use the new operator instead of malloc()?

[faldzip]

hmm I tried something loike this:

#include <iostream>
 
int main()
{
	int *array = new int[600000000];
	std::cout << "Test 1: sizeof(int) = " << sizeof(int) << std::endl;
	array[0] = 5;
	std::cout << "Test 2: array[0] = " << array[0] << std::endl;
	system("pause");
	delete[] array;
	return 0;
}

#include <iostream>

int main()
{
	int *array = new int[600000000];
	std::cout << "Test 1: sizeof(int) = " << sizeof(int) << std::endl;
	array[0] = 5;
	std::cout << "Test 2: array[0] = " << array[0] << std::endl;
	system("pause");
	delete[] array;
	return 0;
}

To copy to clipboard, switch view to plain text mode 

and it is working fine on Vista Business 64. sizeof(int) is 4 so 4*600000000 is about 2.4GB - didn't try more, because I have 2GB of RAM in my notebook.
But my program was compiled with Visual Studio compiler with target machine x64 (so it is 64-bit executable - not working on 32-bit windows).

And make it clear:

Is the latest Qt compiler 64-bit?

Qt it is a bunch of C++ classes so there is no Qt C++ compiler (just some tools making use of Qt easier) so if there are some limitations of a compiler - it is only depend on your compiler. If you have downloaded Qt SDK then I suppose that you have MinGW - I don't know this one, but I think it is not generating 64-bit code (by default).

P.S. I tried also you code from first post and it is working good even for 2000000000.

[Skywalker]

So it's looking like this is probably a 32 bit compiler issue. Thanks.

Do one more thing - what happens if you use the new operator instead of malloc()?

It also crashes, though oddly after the crash I now get the message "This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.", while I got no message with malloc().

If you have downloaded Qt SDK then I suppose that you have MinGW - I don't know this one, but I think it is not generating 64-bit code (by default).

Yes I have the SDK. I've read that Qt is 64 bit since 3.3 - don't really get how that would help with a 32 bit compiler?

Anyway, what 64 bit compilers is Qt compatible with? And how do I change the compiler Qt uses by default?

[wysota]

I've read that Qt is 64 bit since 3.3 - don't really get how that would help with a 32 bit compiler?

It means it can be compiled with a 64 bit compiler.

Anyway, what 64 bit compilers is Qt compatible with?

MinGW has a 64 bit version.

And how do I change the compiler Qt uses by default?

You use the proper QMAKESPEC directory and make sure paths are setup correctly.

[Skywalker]

It was indeed a 32 bit compiler issue - I was able to successfully run a test program compiled with MinGW 64.

I'm not quite sure how to properly set up Qt to work with MinGW64, but I'll make a different thread for that.

Thank you very much