QVector crashes when array size is big

[Sheng]

#include<QApplication>
#include<iostream>
#include<QString>
#include<QVector>

struct myStruc
{
QString name;
int array[3000*2000];
};

int main(int argc,char* argv[])
{
QApplication app(argc,argv);
QVector<myStruc> myVector;
myStruc currentStruc;
for (int i=0;i<5;i++)
{
currentStruc.name+="a";
currentStruc.array[0]=i;
myVector.push_back(currentStruc);
}
for(int i=0;i<5;i++)
{
std::cout<<myVector[i].name.toStdString()<<" "<<myVector[i].array[0]<<std::endl;
}
myVector.clear();
return app.exec();
}



Compile fine, got "segmentation fault" error when run it.
It works OK if I decrease array size to 3000, any solutions?
Thanks in advance.

[abernat]

Have you stepped through it in a debugger or added qDebug() output to determine where you're getting the seg fault?

[talk2amulya]

what happens when u try other container classes like QList?

[Sheng]

what happens when u try other container classes like QList?

Same thing happens (QVector,QList,QLinkedList, std::vector, std::list). Did some search and found this looks like a notorious problem of STL container. I guess Qt Container might use STL during implementation since STL Containers crash as well.
Anyway, I will write a simple list container tomorrow.

[Sheng]

Have you stepped through it in a debugger or added qDebug() output to determine where you're getting the seg fault?

The segmentation fault comes when push_back function is called.

[talk2amulya]

well, the issue is that QVector and other containers allocate CONTIGUOUS memory..and ur every array takes 24+MB of memory plus QString which is also dynamic..so there are definitely chances that when u try to push back one more..it being another contiguous location enters into another segment..thus a segmentation fault..a solution to this would be PROBABLY create a container that doesn't allocate contiguous memory but spreaded..but still its risky cuz amount being allocated is huge...maybe some guru can give a better insight on this

[wysota]

6*3000*2000*sizeof(int) = 36M*sizeof(int) = 144 (or 288) MB just for creating the array. Aren't you running out of memory? If not, are you sure C++ copies arrays? I doubt that so copying the struct might lose its integrity.

[lni]

The crash is not because it is QVector. You program will crash without QVector in pure C++..

Don't put such a huge array in stack. You should use memory in the heap. But it is going to swap if you don't have enough memory...

Do this:


struct myStruc
{
QString name;
int *array;

myStruc() : array( new int[ 3000*2000] ) {
}

// or with size argument in constructor
myStruct( size_t sz ) : array( new int[ sz ] ) {
}

~myStruc() {
delete [] array;
}

};

[wysota]

Actually QVector allocates its data on the heap even if the vector object itself is on the stack.

[Sheng]

6*3000*2000*sizeof(int) = 36M*sizeof(int) = 144 (or 288) MB just for creating the array. Aren't you running out of memory? If not, are you sure C++ copies arrays? I doubt that so copying the struct might lose its integrity.

the Container will crash even I only push_back one object.

[drhex]

QVector probably allocates memory for several objects at a time, to avoid having to reallocate at every push_back(). How much RAM is there in your testbox, Sheng?

Does it work if you supply an initial vector size, e.g:


QVector<myStruc> myVector(5);

[Sheng]

QVector probably allocates memory for several objects at a time, to avoid having to reallocate at every push_back(). How much RAM is there in your testbox, Sheng?

at least 1G.
It does not work either with reserve or resize.

[Sheng]

Just wrote and tested my own linked list, it works fine.

[wysota]

Just wrote and tested my own linked list, it works fine.

Which proves exactly nothing.

[Sheng]

Which proves exactly nothing.

It at least demonstrated the problem is not from the limitation of the system, but from the container.

[drhex]

I've experimented some more and abbreviated the program:


#include<QVector>

struct myStruc {
int array[3000*2000];
};

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

QVector<myStruc> myVector;
myStruc currentStruc;
myVector.push_back(currentStruc);
}

For array-sizes upto 3000*232 (=2.8MB), the above program works as expected.

For array-sizes between 3000*233 and 3000*698, it prints "hello" and then segfaults.

For array-sizes 3000*699 (=8.4MB) and up, it segfaults immediately, without even printing "hello".

Adding a QApplication object did not help.

Tested with gcc 4.3.2 on Ubuntu 8.10 using Qt-4.5.0-beta1.

Sheng, you've found the struct-of-death! ;)

[Sheng]

I've experimented some more and abbreviated the program:


#include<QVector>

struct myStruc {
int array[3000*2000];
};

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

QVector<myStruc> myVector;
myStruc currentStruc;
myVector.push_back(currentStruc);
}

For array-sizes upto 3000*232 (=2.8MB), the above program works as expected.

For array-sizes between 3000*233 and 3000*698, it prints "hello" and then segfaults.

For array-sizes 3000*699 (=8.4MB) and up, it segfaults immediately, without even printing "hello".

Adding a QApplication object did not help.

Tested with gcc 4.3.2 on Ubuntu 8.10 using Qt-4.5.0-beta1.

Sheng, you've found the struct-of-death! ;)

Nice research, thanks.

[drhex]

#include <stdio.h>
#include <unistd.h>

struct myStruc {
int array[3000*2000];
};


void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc currentStruc;
foo(currentStruc);
}

The above also crashes without printing hello, and now there isn't any Qt code left! Seems like a compiler or OS problem.

[Sheng]

#include <stdio.h>
#include <unistd.h>

struct myStruc {
int array[3000*2000];
};


void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc currentStruc;
foo(currentStruc);
}

The above also crashes without printing hello, and now there isn't any Qt code left! Seems like a compiler or OS problem.

try use heap to see what happens. Should work fine, then try to push back to the container.
Here is what I got:
hello
0
segmentation fault

[drhex]

#include <stdio.h>
#include <sys/resource.h>

int main(int , char *[])
{
struct rlimit rlim;
getrlimit(RLIMIT_STACK, &rlim);
printf("max stack size=%d bytes\n", (int)rlim.rlim_cur);
}

This prints

max stack size=8388608 bytes

on my machine. One learns something every day!

[drhex]

try use heap to see what happens. Should work fine, then try to push back to the container.
Here is what I got:
hello
0
segmentation fault

Try using a pointer or a QVector in myStruc instead of that huge int array.

[wysota]

Or a type shorter than int.

[lni]

Actually QVector allocates its data on the heap even if the vector object itself is on the stack.

QVector is fine. He has a huge array in the stack, exceeding stack limit.

[Sheng]

QVector is fine. He has a huge array in the stack, exceeding stack limit.

Read my post #19, use heap instead of stack, container crashes.

[Sheng]

Or a type shorter than int.

Yes, actually I used uint8_t in my code.


Try using a pointer or a QVector in myStruc instead of that huge int array.

I can not use pointer since I need to save image data in the queue.

[lni]

Read my post #19, use heap instead of stack, container crashes.


How do you use heap? Try this:



#include <stdio.h>
#include <unistd.h>

struct myStruc {
int *array;
myStruc() {
array = new int[ 3000*2000 ];
}
~myStruc() {
delete [] array;
}
};


void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc currentStruc;
foo(currentStruc);
}

[Sheng]

How do you use heap? Try this:



#include <stdio.h>
#include <unistd.h>

struct myStruc {
int *array;
myStruc() {
array = new int[ 3000*2000 ];
}
~myStruc() {
delete [] array;
}
};


void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc currentStruc;
foo(currentStruc);
}



It also crashes when you push back structure into vector.
Here is the code:


#include <stdio.h>
#include <unistd.h>
#include <QVector>

struct myStruc {
int *array;
myStruc() {
array = new int[ 3000*2000 ];
}
~myStruc() {
delete [] array;
}
};


void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc currentStruc;
foo(currentStruc);
QVector<myStruc> myVector;
myVector.push_back(currentStruc);
}

[lni]

It crashes because you don't have copy constructor. The QVector needs a copy constructor and it uses a default shadow copy constructor. Your "array" pointer is deleted twice when it goes out of scope...

Run the following program and see how many times the constructor is called and how many times the destructor is called.



#include <stdio.h>
#include <unistd.h>
#include <QVector>
#include <QDebug>

struct myStruc {
int *array;
myStruc() {
qDebug() << "myStruc";
array = new int[ 3000*2000 ];
}

~myStruc() {
qDebug() << "~myStruc";
delete [] array;
}
};


void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc currentStruc;
foo(currentStruc);
QVector<myStruc> myVector;
myVector.push_back(currentStruc);
}

[Sheng]

OK, thanks, you are right, both of programs below works fine.

1.


#include <stdio.h>
#include <unistd.h>
#include <QVector>
#include <QDebug>

struct myStruc {
int *array;
myStruc() {
qDebug() << "myStruc";
array = new int[ 3000*2000 ];
}

myStruc(const myStruc& struc1) {
qDebug() << "copyStruc";
array = new int[3000*2000];
for(int i=0;i<3000*2000;++i)
array[i]=struc1.array[i];
}

~myStruc() {
qDebug() << "~myStruc";
delete [] array;
}
};


void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc currentStruc;
foo(currentStruc);
QVector<myStruc> myVector;
myVector.push_back(currentStruc);
}


2.




int main(int,char* [])
{
QVector<int> array(3000*2000);
QVector<QVector<int> > myVector;
myVector.push_back(array);
}

[lni]

You 2nd example is a no-op for the myStruct. You define a "myStruct" but it is not used in your main example code. QVector<int> array(3000*2000) is allocated in heap by QVector.

More over, even if you have enough stack size, the code is dangerous because your program could run in other machine which doesn't have enough stack size, it would be a nightmare to debug because it runs in your development machine, and crash in the client machines.

Avoid excessive use of stack memory, certainly not 2000x3000 size of int, which is 6,000,000 int array!

[Sheng]

You 2nd example is a no-op for the myStruct. You define a "myStruct" but it is not used in your main example code. QVector<int> array(3000*2000) is allocated in heap by QVector.

More over, even if you have enough stack size, the code is dangerous because your program could run in other machine which doesn't have enough stack size, it would be a nightmare to debug because it runs in your development machine, and crash in the client machines.

Avoid excessive use of stack memory, certainly not 2000x3000 size of int, which is 6,000,000 int array!

Why this one crashes? I thought everything is in heap, no?



struct myStruc {
QString myString;
int array[3000*2000];
};

void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc* currentStruc=new myStruc;
foo(*currentStruc);
QVector<myStruc> myVector;
myVector.push_back(*currentStruc);
}

[lni]

myStruc* currentStruc=new myStruct is in heap, but the element inside myStruct is in stack!

[Sheng]

myStruc* currentStruc=new myStruct is in heap, but the element inside myStruct is in stack!

I do not think so, in my own linked list, I use this and works fine. The maximum stack size is around 8M. the imageData is larger than 10M.

I defined my structure as follows:


struct imageStruc
{
QString fileName;
uint8_t imageData[IMAGE_SIZE];
imageStruc* next;
};

[wysota]

but the element inside myStruct is in stack!

No, it's not. If it were on the stack and you returned from the function the frame of which it would be positioned in, the data would get destructed which is of course not the case.

What is on the stack is probably the local copy of the structure when push_back is called (although I'm not sure as push_back probably takes a const reference and not a copy).

Sheng: Why do you need that array there in the first place? Can't you use a vector that dynamically allocates its data as it is needed?

By the way, if you don't need a vector but a linked list, use QLinkedList and not QVector :)

[fullmetalcoder]

myStruc* currentStruc=new myStruct is in heap, but the element inside myStruct is in stack!
:eek: AFAIK if you allocate a struct on heap ALL its field can only be allocated on heap...

Now some thoughts that may or may not be relevant :


big structures like that should never ever be passed by value anywhere... allocate them on heap and use a QVector<myStruct*> instead of a QVector<myStruct> (or whatever container you may like more)
Storing the data directly in the struct instead of a pointer to it is a HUGE design mistake IMO because it forces it to be allocated no matter what and that gets even worse when some copy has to be made. Using a pointer and allocating the data when it is needed would probably be a better idea, would allow sharing and would very probably cause a dramatic decrease of memory usage and a dramatic increase in speed
huge contiguous arrays of memory are not a very wise design choice. Maybe you can split that data in smaller pieces? And do you need such an amount of memory anyway (I assume you want several instances of that struct since you are then using containers)?

[Sheng]

No, it's not. If it were on the stack and you returned from the function the frame of which it would be positioned in, the data would get destructed which is of course not the case.

What is on the stack is probably the local copy of the structure when push_back is called (although I'm not sure as push_back probably takes a const reference and not a copy).

Sheng: Why do you need that array there in the first place? Can't you use a vector that dynamically allocates its data as it is needed?

By the way, if you don't need a vector but a linked list, use QLinkedList and not QVector :)

Yes, actually I used linked list, just use QVector for the example since all of them crashes.

[lni]

No, it's not. If it were on the stack and you returned from the function the frame of which it would be positioned in, the data would get destructed which is of course not the case.
QLinkedList and not QVector :)

In his example, the stack is contained inside the heap. The stack will only get destructed when the heap is deleted, it will not go out of scope if the heap is no deleted. When you do "new myStruct" to construct the object, where does the array[ 2000x300 ] come from? It comes from the stack, and it will crash immediately at the "new myStruct".

As I said, the element inside the structure is in the stack, change it to heap and it should work.

[wysota]

Yes, actually I used linked list, just use QVector for the example since all of them crashes.
Still you didn't say what is the array meant for.


In his example, the stack is contained inside the heap.
I'm sorry, but are you sure you know what the stack is? Because based on your post I think you don't.

[Sheng]

:eek: AFAIK if you allocate a struct on heap ALL its field can only be allocated on heap...

Now some thoughts that may or may not be relevant :


big structures like that should never ever be passed by value anywhere... allocate them on heap and use a QVector<myStruct*> instead of a QVector<myStruct> (or whatever container you may like more)
Storing the data directly in the struct instead of a pointer to it is a HUGE design mistake IMO because it forces it to be allocated no matter what and that gets even worse when some copy has to be made. Using a pointer and allocating the data when it is needed would probably be a better idea, would allow sharing and would very probably cause a dramatic decrease of memory usage and a dramatic increase in speed
huge contiguous arrays of memory are not a very wise design choice. Maybe you can split that data in smaller pieces? And do you need such an amount of memory anyway (I assume you want several instances of that struct since you are then using containers)?


Copy has to be made since I need to copy the image from the buffer to somewhere else, the buffer is updated quickly.

[lni]

:eek: AFAIK if you allocate a struct on heap ALL its field can only be allocated on heap...


FALSE.

Try this and tell me why it works.



#include <QVector>
#include <QString>

struct myStruc {
QString myString;
int *array;
myStruc() {
array = new int[ 3000*2000 ];
}

};

void foo(const myStruc &v)
{
printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
printf("hello\n");
fflush(stdout);
sleep(1);

myStruc* currentStruc=new myStruc;
foo(*currentStruc);
QVector<myStruc> myVector;
myVector.push_back(*currentStruc);
}

[Sheng]

In his example, the stack is contained inside the heap. The stack will only get destructed when the heap is deleted, it will not go out of scope if the heap is no deleted. When you do "new myStruct" to construct the object, where does the array[ 2000x300 ] come from? It comes from the stack, and it will crash immediately at the "new myStruct".

As I said, the element inside the structure is in the stack, change it to heap and it should work.

no, it's not, and it works in my code. the crash comes not from "new myStruct", but from push_back.

[lni]

no, it's not, and it works in my code. the crash comes not from "new myStruct", but from push_back.

Your computer may have enough resource to hold 1 copy of array[ 2000 x 3000 ], and crash if you have two copies, that is precisely what QVector does, it copies your structure when you do push_back the object (value based object).

Go back review my post on calling the destructor and copy constructor prinout, and how you add a copy constructor to fix the crash.

[wysota]

FALSE.

Try this and tell me why it works.

:-)

Explain this:


#include "stdio.h"
#include <QString>

struct A {
QString x;
int array[1000];
};

int main(){
A a;
A *b = new A;
printf("Addr of a: 0x%08x\n", (unsigned int)&a);
printf("Addr of a.array: 0x%08x\n", (unsigned int)a.array);
printf("Addr of b: 0x%08x\n", (unsigned int)b);
printf("Addr of b->array: 0x%08x\n", (unsigned int)b->array);
return 0;
}

Result:
Addr of a: 0xbfa5008c
Addr of a.array: 0xbfa50090
Addr of b: 0x089fe1b8
Addr of b->array: 0x089fe1bc

As you see a and a.array are on the stack and b and b->array are on heap.

Your "example" works because even if "myStruc" is allocated on the stack, its array member will be allocated on the heap. But it doesn't mean that if myStruc is allocated on the heap, array is (or can be) allocated on the stack :)

[wysota]

By the way, push_back is not a problem. This crashes on the QVector constructor and I think I know why :)


#include <QVector>
#include <QString>

struct StructOfDeath {
QString some;
QString innocent;
int variables;
double And;
int memberOfDeath[3000*2000];
StructOfDeath(){ variables = 0; And = 1;}
};

int main(){

QVector<StructOfDeath> vector;
vector.reserve(30);
for(int i=0;i<10;i++)
vector.push_back(StructOfDeath());
printf("Done\n");
return 0;
}

[lni]

:-)

As you see a and a.array are on the stack and b and b->array are on heap.

Your "example" works because even if "myStruc" is allocated on the stack, its array member will be allocated on the heap. But it doesn't mean that if myStruc is allocated on the heap, array is (or can be) allocated on the stack :)

Do some google search on "array in stack overflow", or at
http://www.devx.com/tips/Tip/14276

[Sheng]

Your computer may have enough resource to hold 1 copy of array[ 2000 x 3000 ], and crash if you have two copies, that is precisely what QVector does, it copies your structure when you do push_back the object (value based object).

Go back review my post on calling the destructor and copy constructor prinout, and how you add a copy constructor to fix the crash.

There are more than 10 images in the memory, not just two.

[faldzip]

Do some google search on "array in stack overflow", or at
http://www.devx.com/tips/Tip/14276
I can't see any connection between that article and things you are saying :P
As for me it's simple:

If there's code like this:


A a;

object a of type A is created on stack. If that code appears as a local variable in some function/method then at the end of the function/method object a will be deleted from stack, because program is returning to the caller, to place where call appeared.
(that's why there's stack overflow in infinite recursion -> infinite storing the called method variables and no returns, which will remove these variables from stack)

But if there's code like this:


A *a = new A;

instead of previously shown, then it's different story :] it means that your local variable created on a stack is a pointer (32-bit address = 4B) which points to the object of type A created on heap - not afecting stack in any way.

So if there is an array in struct created dynamically on a heap:


struct myStruc {
QString myString;
int *array;
myStruc() {
array = new int[ 3000*2000 ];
}

};

then this struct size is just a size of myString and size of pointer to an array. An array itself is always on heap (beacause of operator new).


If there is any mistake in my understanding of stack/heap, feel free to provide any interesting links on this topic. :D

[wysota]

Do some google search on "array in stack overflow", or at
http://www.devx.com/tips/Tip/14276

Honestly I don't see any connection between the issue of the array being on the stack or not and the article. I think the example I gave you is a proof that can't be undermined.

Stack addresses have high values because in most (all?) modern architectures the stack expands "downwards" (to lower memory addresses) so there has to be enough space for the stack to expand until it reaches the heap space (that is (usually?) between the global variable address space and the stack address space). The example clearly shows that a.array is positioned 4 bytes away from the beginning of the structure in the same adress region so it has to be inside the struct as sizeof(A) will return a value larger than 4 bytes so a logical conclusion follows that both "objects" have to be positioned in the same "part" of memory. The same goes for the second variant, the difference is also 4 bytes and the size of the structure remains the same.

Here is a result of the extended version of my previous example. Note the difference in structure sizes and addresses.

Addr of a: 0xbfcaebb0
Addr of a.array: 0xbfcaebb4
Addr of b: 0x0808d1b8
Addr of b->array: 0x0808d1bc
Size of struct A: 4004

Addr of c: 0xbfcafb54
Addr of c.array: 0x0808e168
Addr of d: 0x0808f110
Addr of d->array: 0x0808f120
Size of struct B: 8

And the code itself...


#include "stdio.h"
#include <QString>

struct A {
QString x;
int array[1000];
};

struct B {
QString x;
int *array;
B(){ array = new int[1000]; }
};

int main(){
A a;
A *b = new A;
B c;
B *d = new B;
printf("Addr of a: 0x%08x\n", (unsigned int)&a);
printf("Addr of a.array: 0x%08x\n", (unsigned int)a.array);
printf("Addr of b: 0x%08x\n", (unsigned int)b);
printf("Addr of b->array: 0x%08x\n", (unsigned int)b->array);
printf("Size of struct A: %d\n", sizeof(A));
printf("\n");
printf("Addr of c: 0x%08x\n", (unsigned int)&c);
printf("Addr of c.array: 0x%08x\n", (unsigned int)c.array);
printf("Addr of d: 0x%08x\n", (unsigned int)d);
printf("Addr of d->array: 0x%08x\n", (unsigned int)d->array);
printf("Size of struct B: %d\n", sizeof(B));

return 0;
}

[Sheng]

Thank you all for your posts. I think I am clear of everything about Qt containers.
I also think every one learn something new, if not, you are good enough. :)

[wysota]

Sheng, a side question - why don't you use QImage or QByteArray instead of that huge array of yours?