Is there a maximum vector length other than a memory limit?

[agerlach]

I am using Win 7 64 bit with 12 GB RAM.
I want to allocate a QVector<double> of length 714838*256 by:

QVector<double> *qwert = new QVector<double>(714838*256);

QVector<double> *qwert = new QVector<double>(714838*256);

To copy to clipboard, switch view to plain text mode 

but I get a bad_alloc() exception. So I tried it with just:

double * qwert = new double[714838 * 256];

double * qwert = new double[714838 * 256];

To copy to clipboard, switch view to plain text mode 

And I still get an error. After a few trials I found that I can allocate 178077685 doubles but not 178077686. But,

QVector<double> *qwert = new  QVector<double>(178077685);

QVector<double> *qwert = new  QVector<double>(178077685);

To copy to clipboard, switch view to plain text mode 

still throws the same exception. I show this to be ~1.32 GB of doubles wihich is << 12 GB or RAM available.
I've looked all day online to find a reason why I cannot allocate a double array any larger and the only explanation I can find is a RAM limitation, which is not the case here.

[tbscope]

There are more limits on your pc besides the amount of ram. (as in, do you know exactly where this memory is reserved?)

Why, dear oh dear, why?
Why do you need that much memory?

To me this screams "Complete and utter bad design"
Can you tell why you need to do this?

[SixDegrees]

What happens if you try to allocate a simple array of doubles, as in

double* t = new double[178077686];

double* t = new double[178077686];

To copy to clipboard, switch view to plain text mode 

Are the results different?

[agerlach]

What happens if you try to allocate a simple array of doubles, as in

double* t = new double[178077686];

double* t = new double[178077686];

To copy to clipboard, switch view to plain text mode 

Are the results different?

I may have been unclear but I tried that. This works:

double* t = new double[178077685];

double* t = new double[178077685];

To copy to clipboard, switch view to plain text mode 

but this does not:

double* t = new double[178077686];

double* t = new double[178077686];

To copy to clipboard, switch view to plain text mode 

[agerlach]

There are more limits on your pc besides the amount of ram. (as in, do you know exactly where this memory is reserved?)

Why, dear oh dear, why?
Why do you need that much memory?

To me this screams "Complete and utter bad design"
Can you tell why you need to do this?

The 714838*256 size represents 714838 16x16 "images" called a training stack. This is used by taking another 16x16 "image" and correlating it with all the images in the stack. This needs to be performed fast. I currently have this algorithm built in Matlab and I have no problem allocating this much memory. Based on the time it takes to do this in Matlab and based on the speed ups I have experienced in the past from moving to C++ I expect to do this in < 1 s. Due to the speed requirements I would rather not have to write intermediate steps to the hard disk. Plus this code will not be deployed so I can control any hardware constraints.

If anybody has any suggestions on betters ways to do this that will not sacrifice speed, I am open to ideas.

What other limits would I have other than RAM? And what do you mean by "do you know exactly where this memory is reserved?"

[tbscope]

What other limits would I have other than RAM? And what do you mean by "do you know exactly where this memory is reserved?"

What sits between your physical ram and your program code?
It's a chain with limits.

For example, you will not be able to use the 12GB of ram on a 32 bit system

[agerlach]

I understand that I will not be able to use 12GB RAM on a 32 bit system, but this is on a 64 bit OS.

[tbscope]

What are your stack and heap sizes?

[SixDegrees]

I'm not a Windows guy, but the failure of even Plain Old C Arrays to work suggests some sort of OS limit imposed on your processes. In Unix, both the OS and the user can fiddle with the maximum memory size a given process can allocate; this is normally set to "unlimited," but I've seen it restricted on some systems. I suspect Windows is probably "helping" you by imposing such a limit. No idea what the command would be to increase it, though.

Here's an article on the subject - sounds like this might be worth investigating.

[agerlach]

What are your stack and heap sizes?

I'm sorry, but could you tell me how to check that?

[agerlach]

I'm not a Windows guy, but the failure of even Plain Old C Arrays to work suggests some sort of OS limit imposed on your processes. In Unix, both the OS and the user can fiddle with the maximum memory size a given process can allocate; this is normally set to "unlimited," but I've seen it restricted on some systems. I suspect Windows is probably "helping" you by imposing such a limit. No idea what the command would be to increase it, though.

Here's an article on the subject - sounds like this might be worth investigating.

You are on to something, it looks like my program is compiling at a 32 bit application which is limited to 2 GB of RAM. I created an empty project in VS2010 and compile as x64 and I was able to allocate significantly more memory. I was using Qt creator for development but I don't really know how to set it up to build my projects as x64. Would it be easier for me to just start using VS instead?

[SixDegrees]

Again, I'm not Windows-savvy, but make sure your QMAKESPEC is pointing to the 64-bit version of its files, not the 32-bit version.

[ChrisW67]

As I understand it, the bundled MingW environment in the Qt SDK is 32-bit only.

[agerlach]

OK, for my project I am using Qt, VTK, and the newmat C++ matrix library. I compiled newmat to x64 and downloaded the source for Qt and ran configure.exe -> nmake -> nmake install from the VS2008 x64 command prompt. Everything built fine. I then built VTK directed at the x64 Qt build in x64. Again, everything built fine. I guess my question is on how to start using the x64 version of Qt I built in Qt Creator and how do I handle QMAKESPEC. I don't see anything in the mkspecs folder to seems appropriate. There is only win32 for msvc2008.

[JohannesMunk]

The problem is that the vector requires all that data to be in one big chunk.

Your correlation will probably be interframe for each pixel? So that you need fast access to different frames of the same pixel.

Why don't you manage your data as a 256 long QList of 714838 double vectors?

Access won't be slower. But you allow for some fragmentation.

Joh

[agerlach]

OK, for my project I am using Qt, VTK, and the newmat C++ matrix library. I compiled newmat to x64 and downloaded the source for Qt and ran configure.exe -> nmake -> nmake install from the VS2008 x64 command prompt. Everything built fine. I then built VTK directed at the x64 Qt build in x64. Again, everything built fine. I guess my question is on how to start using the x64 version of Qt I built in Qt Creator and how do I handle QMAKESPEC. I don't see anything in the mkspecs folder to seems appropriate. There is only win32 for msvc2008.

Apparently Qt Creator handles the QMAKESPEC for you based on the build of qmake you point it to. Everything works good except debugging. I download cdb for x64 and pointed Qt Creator to it but it has the following error:

Unable to load the debugger engine library 'C:\Program Files \Debuggint Tools for Windows (x64)\dbghelp.dll': Cannot load library C:\Program Files\Debugging Tools for Windows (x64)\dbghelp.dll

even though these files exist at that location.

[agerlach]

The problem is that the vector requires all that data to be in one big chunk.

Your correlation will probably be interframe for each pixel? So that you need fast access to different frames of the same pixel.

Why don't you manage your data as a 256 long QList of 714838 double vectors?

Access won't be slower. But you allow for some fragmentation.

Thanks, that makes sense, but what do you mean by "interfram for each pixel". The following equation is how the correlation is calculated. P is the randomly selected image and Q is one of the 714838 images in the training stack. pi and qi are the ith pixel values of P or Q respectively. Their are 256 pixels per image. Normally W^2 = 256, but in this case it equals the number of pixels that are nonzero in both P and Q.
l Correlation..png

[JohannesMunk]

Ah ok. So you correlate whole frames. And you need to correlate every new frame against your complete training?

Then you should definitely store the individual pixel and squarepixels sums to speed up calculation of the denominator of your formula.

Store a struct/class for each frame with the rawpixel data and the precalculated values.

Put all your frames into a QList. That will effectively be an array of pointers to your structs/classes. As that array will be quite small and each frame-struct is small, you should not have a problem allocating that memory.
As you will need to resolve those only once for each much more expensive correlation-calculation, that won't have an impact on performance, either.

HIH

Johannes

[agerlach]

Ah ok. So you correlate whole frames. And you need to correlate every new frame against your complete training?

Then you should definitely store the individual pixel and squarepixels sums to speed up calculation of the denominator of your formula.

Store a struct/class for each frame with the rawpixel data and the precalculated values.

Put all your frames into a QList. That will effectively be an array of pointers to your structs/classes. As that array will be quite small and each frame-struct is small, you should not have a problem allocating that memory.
As you will need to resolve those only once for each much more expensive correlation-calculation, that won't have an impact on performance, either.

Just to make sure I understand everything, when you say "... as that array will be quit small" you are refering to its size in memory since it is just a array of pointers? If so, this make a ton of sense and is a huge help. Thanks!

On a second note, I never thought about pre calculating the sums in the equation. I never did in the past because only the pixel values in Q that correspond to non zero pixel values in P are used in the calculation. So if Q = {1 2 3} and P = {0 0 1} then sum(Q) = 3 since Q(0) and Q(1) are not used. But now that I think about it since the images are never all zero it would be more efficient to calculate the sum and then subtract the elements that correspond to a 0 value in P.

[JohannesMunk]

Just to make sure I understand everything, when you say "... as that array will be quit small" you are refering to its size in memory since it is just a array of pointers? If so, this make a ton of sense and is a huge help. Thanks!

Thats exactly what it means! Of course the overall memory usage will be equally big, but now the memory manager can decide where to put all the little pieces(=frames), and is not forced (and doesn't fail) to allocate a big continuous chunk.

On a second note, I never thought about pre calculating the sums in the equation. I never did in the past because only the pixel values in Q that correspond to non zero pixel values in P are used in the calculation. So if Q = {1 2 3} and P = {0 0 1} then sum(Q) = 3 since Q(0) and Q(1) are not used. But now that I think about it since the images are never all zero it would be more efficient to calculate the sum and then subtract the elements that correspond to a 0 value in P.

If Q={1,2,3} then sum(Qi)=6. if P={0,0,1} sum(Pi*Qi)=3. I was only talking of the denominator of your formula. There are no mixed (p and q) sum expressions there.

Johannes