Stuck creating << and >> operators for my custom class.

I need to create non-member operator<< and operator>> functions for my custom class so that I can stream them to a QByteArray to push as a BLOB in a mySQL database but I am having some problems in creating my << and >> functions.

I have the following classes

Code:

---

class siStorage : public QObject
{
...
QVector<float> *sImage;
float sImageSum;
float sImage2Sum;
}

---

Code:

---

class siStack : public QObject
{
...
QList<siStorage*> *stack;
}

---

All siStorage in the QList in stack are always the same length, for example 256. And the length of stack can be large, for example > 50,000.

I would like to write << and >> operators that read and write an siStack object.

This is what I currently have:

Code:

---

QDataStream &operator<< (QDataStream &stream, const siStorage &storage)
{
stream << *(storage.sImage) << storage.siSum << storage.si2Sum;
return stream;
}
 
QDataStream &operator<< (QDataStram &stream, const siStack &stack)
{
for( int = ii = 0; ii < stack.stack.size(); ii++) {
stream << *(stack.stack->at(ii));
}
 
return stream;
}

---

Is this the way I should do this? When I check the size of my resulting QByteArray it is about twice the size of what I would expect. I assume there is some overhead by streaming over 50,000 QVectors but I won't expect that much.

As for the >> operator I have no clue on how to implement it. This will be my first time doing anything like this. Any suggestions or examples will be greatly appreciated.

Quote:

---

Is this the way I should do this?

---

In general yes.

Quote:

---

When I check the size of my resulting QByteArray it is about twice the size of what I would expect.

---

Can you show the code for that?

Quote:

---

As for the >> operator I have no clue on how to implement it.

---

Exactly like << just the other way around:

Code:

---

QDataStream &operator>> (QDataStream &stream, siStorage &storage)
{
 stream >> *(storage.sImage) >> storage.siSum >> storage.si2Sum;
return stream;
}

---

Quote:

---

In general yes.

---

Is there a way to construct the operator to work for pointers? In the documentation it states that I can do << QList<T> as long as the << operator is defined for T. In my case T is a pointer. This would simplify things so I only need to write << and >> operators for siStorage and I woudn't need to use for loops

Quote:

---

Can you show the code for that?

---

Almost all the memory used is stored in stack. Each sImage in siStorage has 256 floats and in this example their are 54,877 siStorage* in stack. So 4 bytes / float * 256 floats / siStorage * 54,877 siStorage / stack = 56,194,048 bytes / stack. But if I do this below I get 112,607,608 bytes, where pStack is a siStack*.

Code:

---

QByteArray byteArray;
QBuffer buffer(&byteArray);
buffer.open(QIODevice::WriteOnly);
 
QDataStram out(&buffer);
 
out << *pStack;
 
byteArray.size();

---

Quote:

---

Exactly like << just the other way around:

---

That makes sense. But because of my use of a for loop won't I need the length of the loop first? Another reason why this would be much simpler if I could define << for a siStorage*.

Quote:

---

But because of my use of a for loop won't I need the length of the loop first?

---

Yes, so write it to the output stream before you write out the elements of the stack. This is what the standard containers do.

Is there a particular reason you have allocated your QVector<float> and QList<siStorage *> on the heap rather than as part of the object?

Quote:

---

Is there a way to construct the operator to work for pointers? In the documentation it states that I can do << QList<T> as long as the << operator is defined for T. In my case T is a pointer. This would simplify things so I only need to write << and >> operators for siStorage and I woudn't need to use for loops

---

I am not sure what exactly you are asking about.
You can make the operator take a pointer sure, it will not change much.
However, I am not sure what exactly you have in mind to do with the pointer?
When you serialize the data to the stream, you have to copy the data it self, there is no way around defreferencing the pointer at some stage in the process. (That is, if I understood you correctly)

Code:

---

QDataStream &operator<< (QDataStream &stream, const siStorage *storage)
{
stream << *(storage->sImage) <<storage->siSum << storage->si2Sum;
return stream;
}

---

As you can see, there is little difference.

Quote:

---

Almost all the memory used is stored in stack. Each sImage in siStorage has 256 floats and in this example their are 54,877 siStorage* in stack. So 4 bytes / float * 256 floats / siStorage * 54,877 siStorage / stack = 56,194,048 bytes / stack. But if I do this below I get 112,607,608 bytes, where pStack is a siStack*.

---

Well, you forgot to add other members in siStorage (such as the two floats) and probably other members in siStack that you didn't show in the code.
In addition all the member allocations done by the QVector - if you add all of these, this might be considerably more than what you calculated.
In addition you need to add memory alignment, which might also considerably add if the internal sizes don't align nicely.

Quote:

---

Is there a particular reason you have allocated your QVector<float> and QList<siStorage *> on the heap rather than as part of the object?

---

Not really. I'm kind of learning all of this by the seat of my pants. Can you explain what you mean and why I should to it any different?

Quote:

---

Is there a particular reason you have allocated your QVector<float> and QList<siStorage *> on the heap rather than as part of the object?

---

Not really. I'm kind of learning all of this by the seat of my pants. Can you explain what you mean and why I should to it any different?

Quote:

---

Is there a particular reason you have allocated your QVector<float> and QList<siStorage *> on the heap rather than as part of the object?

---

Not really. I'm kind of learning all of this by the seat of my pants. Can you explain what you mean and why I should to it any different?

Quote:

---

I am not sure what exactly you are asking about.
You can make the operator take a pointer sure, it will not change much.
However, I am not sure what exactly you have in mind to do with the pointer?
When you serialize the data to the stream, you have to copy the data it self, there is no way around defreferencing the pointer at some stage in the process. (That is, if I understood you correctly)

---

This is exactly what I wanted to do. Now my << operator becomes much simpler for my siStack class. I can do this

Code:

---

QDataStream &operator<< (QDataStream &stream, const siStack &stack)
{
stream << *stack.stack;
return stream;
}

---

instead of

Code:

---

QDataStream &operator<< (QDataStream &stream, const siStack &stack)
{
for (int ii = 0; ii < stack.stack->size(); ii ++) {
stream << *(stack.stack->at(ii));
}
return stream;
}

---

I also no longer need handle streaming the size of the list since the container will handle it for me. This also simplifies the >> operator.

Quote:

---

Well, you forgot to add other members in siStorage (such as the two floats) and probably other members in siStack that you didn't show in the code.

---

I guess I should clarify my example. I modified my code so that siStack only streams stack and siStorage only streams the sImage. So the numbers I present are a true representation of the number of floats used.

What do you mean by:

Quote:

---

In addition all the member allocations done by the QVector - if you add all of these, this might be considerably more than what you calculated.
In addition you need to add memory alignment, which might also considerably add if the internal sizes don't align nicely.

---

Thanks for you time.

Your initial doubling in the expected file size is because QDataStream stores floats and doubles as 64-bits (i.e. double) unless told otherwise with QDataStream::setFloatingPointPrecision()
Here is a complete example:

Code:

---

#include <QtCore>
#include <QDebug>
 
class siStorage: public QObject 
{
    Q_OBJECT
public:
    siStorage(QObject * parent = 0):
        QObject(parent),
        sImage(QVector<float>(256, 0.0f)),
        sImageSum(0.0f),  sImage2Sum(0.0f)
    { }
 
    QVector<float> sImage;
    float sImageSum;
    float sImage2Sum;
};
 
QDataStream &operator<< (QDataStream &stream, const siStorage &s)
{
    stream << s.sImage << s.sImageSum << s.sImage2Sum;
    return stream;
}
 
QDataStream &operator>> (QDataStream &stream, siStorage &s)
{
    stream >> s.sImage >> s.sImageSum >> s.sImage2Sum;
    return stream;
}
 
class siStack: public QObject
{
    Q_OBJECT
public:
    siStack(QObject *parent = 0):
        QObject(parent)
    {
    }
 
    QList<siStorage*> stack;
};
 
QDataStream &operator<< (QDataStream &stream, const siStack &s)
{
    stream << s.stack.size();
    foreach (siStorage *storage, s.stack)
        stream << *storage;
    return stream;
}
 
QDataStream &operator>> (QDataStream &stream, siStack &s)
{
    // Make sure it is empty to start
    qDeleteAll(s.stack);
    s.stack.clear();
 
    int stackSize = 0;
    stream >> stackSize;
    for (int i = 0; i < stackSize; ++i)  {
        siStorage *storage = new siStorage(&s);
        stream >> *storage;
        s.stack << storage;
    }
    return stream;
}
 
 
int main(int argc, char *argv[])
{
    QCoreApplication app(argc, argv);
 
    // test data 5 * (256 floats + 2 floats)
    siStack testStack;
    for (int i = 0; i < 5; ++i) {
        siStorage *s = new siStorage(&testStack);
        testStack.stack << s;
    }
 
    QFile f("test.out");
    f.open(QIODevice::WriteOnly);
    QDataStream out(&f);
    out.setFloatingPointPrecision(QDataStream::SinglePrecision);
    //out.setFloatingPointPrecision(QDataStream::DoublePrecision);
    out << testStack;
    f.close();
 
    siStack newStack;
    qDebug() << newStack.stack.size();
    f.open(QIODevice::ReadOnly);
    QDataStream in(&f);
    in >> newStack;
    f.close();
    qDebug() << newStack.stack.size();
}
 
#include "main.moc"

---

On my machine the test file is 5184 bytes (Qt 4.6.3 on Linux), which represents:
5 * ((256 + 2) * 4) = 5160 bytes of float data
4 byte list size for the siStack
5 * 4 byte vector size for sImage

Thanks, the QDataStream precision was the problem with the size of my QByteArray. Unfortunetly I am still having problems with my >> operator, << seems to be working fine. Here are my two classes:

Code:

---

class siStorage: public QObject
{
Q_OBJECT
public:
siStorage(QObject *parent = 0) : QObject(parent)
{
this->sImage = new QVector<float>();
}
 
QVector<float> *sImage;
float sImageSum;
float sImage2Sum;
};
 
QDataStream &operator<<(QDataStream &stream, const siStorage *storage)
{
stream << *(storage->sImage) << sImageSum << sImage2Sum;
return stream;
}
 
QDataStream &operator>> (QDataStream &stream, siStorage *storage)
{
storage = new siStorage();
stream >> *(storage->sImage) >> sImageSum >> sImage2Sum;
return stream;
}

---

Code:

---

class siStack: public QObject
{
Q_OBJECT
public:
siStack(QObject *parent=0) : QObject(parent)
{
stack = new QList<siStorage*>();
}
 
void populateStack() { ... }
void pushToDataBase() {
 //converts to stack to QByteArray and loads to a database 
    QByteArray stackByteArray;
    QBuffer buffer(&stackByteArray);
    buffer.open(QIODevice::WriteOnly);
 
    QDataStream out(&buffer);
    out.setFloatingPointPrecision(QDataStream::SinglePrecision);
 
    out << *(this);
 
    ... // Database code
}
void pullFromDataBase(){ 
//selects BLOB from database, converts to QByteArray, and then places in stack
QByteArray stackByteArray;
 
//... DB code to populate stackByteArray
 
 
    QBuffer buffer(&stackByteArray);
    buffer.open(QIODevice::ReadOnly);
 
    QDataStream in(&buffer);
    in.setFloatingPointPrecision(QDataStream::SinglePrecision);
 
 
    in >> *(this);
 
}
QList<siStorage*> *stack;
};
 
QDataStream &operator<<(QDataStream &stream, const siStack &stack)
{
stream << *(stack.stack);
return stream;
}
 
QDataStream &operator>>(QDataStream &stream, siStack &stack)
{
stream >> *(stack.stack);
return stream;
}

---

But when I test my code it will not allow me to access a siStorage stored in siStack::stack. Here is a short example.

Code:

---

siStack *stack = new siStack();
stack->populateStack();
qDebug() << stack->stack->size; //Example size = 5;
qDebug() << *(stack->stack->at(5)->sImage); // displays ok;
stack->pushToDataBase();
 
stack->stack->clear;
qDebug() <<stack->stack->size; size = 0;
 
stack->pullFromDataBase();
qDebug() <<stack->stack->size; // size = 5 as above!
qDebug() << *(stack->stack->at(5)->sImage); // code crashes with error code -1073741819
stack->pullFromDataBase();

---

Why are you defining sImage as a pointer to a QVector<float> and not simply using a QVector<float>? In your usage, you keep dereferencing the pointer *(storage->sImage), so you are treating it as a QVector<float> anyway.

You've also defined your stack as a QList of pointers. Where are you allocating (new) the pointers in the stack? In the populateStack method?

There is so much unnecessary dereferencing of pointers going on in your code - you seem to be really confused about pointers, and I suspect that most of the problems you are having are because you are trying to reference the contents of empty or uninitialized pointers, or are trying to stream data into something that hasn't been initialized yet. Get rid of the pointers and replace the code with simple stack allocated objects instead and see if your code works better.

this:

Code:

---

QDataStream &operator>> (QDataStream &stream, siStorage *storage)
{
storage = new siStorage();
stream >> *(storage->sImage) >> sImageSum >> sImage2Sum;
return stream;
}

---

in compbination with this:

Code:

---

in >> *(this);

---

is dangerous, and probably your problem.

in your operator, you are giving 'this' as a pointer, then assign a new address to it, and apply your stream on it.
So now your original 'this' points to the new one you allocated in the '>>' operator, and all hell can break loose if you still have pointers pointing to the original address of siStorage.

EDIT:
actually

Code:

---

in >> *(this);

---

should not compile, since you are dereferencing your pointer...

Quote:

---

Why are you defining sImage as a pointer to a QVector<float> and not simply using a QVector<float>? In your usage, you keep dereferencing the pointer *(storage->sImage), so you are treating it as a QVector<float> anyway.

---

I keep dereferencing the pointer when using the stream operator because it requires me to. In all of my other methods I don't dereference it. I don't really have a good reason on why I am using pointers other than that is what I always seem to see in examples and because I read once that it was much more difficult for someone to reverse engineer your software if you use pointers. I don't know if that is true though.

Quote:

---

You've also defined your stack as a QList of pointers. Where are you allocating (new) the pointers in the stack? In the populateStack method?

---

populateStack allocates the pointers in stack when I need to write stack to the database and the pointers are allocated in QDataStream &operator>> (QDataStream &stream, siStorage *storage) when I try to read the stack from the database.

Quote:

---

There is so much unnecessary dereferencing of pointers going on in your code - you seem to be really confused about pointers, and I suspect that most of the problems you are having are because you are trying to reference the contents of empty or uninitialized pointers, or are trying to stream data into something that hasn't been initialized yet. Get rid of the pointers and replace the code with simple stack allocated objects instead and see if your code works better.

---

I cannot argue with you here. Almost all of my programming experience is with Matlab. I never had to worry about pointers there but my code ran so slow I jumped ship to Qt. I think I have a pretty good idea of how to use pointers (thats a big think :p), but like I said above, I am dereferencing to be able to use the default >> and << for the QList container.

I'll test some code without using pointers to see if I can correct this.

Quote:

---

Originally Posted by high_flyer

this:

Code:

---

QDataStream &operator>> (QDataStream &stream, siStorage *storage)
{
storage = new siStorage();
stream >> *(storage->sImage) >> sImageSum >> sImage2Sum;
return stream;
}

---

in compbination with this:

Code:

---

in >> *(this);

---

is dangerous, and probably your problem.

in your operator, you are giving 'this' as a pointer, then assign a new address to it, and apply your stream on it.
So now your original 'this' points to the new one you allocated in the '>>' operator, and all hell can break loose if you still have pointers pointing to the original address of siStorage.

EDIT:
actually

Code:

---

in >> *(this);

---

should not compile, since you are dereferencing your pointer...

---

If you look at the code again you will see that "this" is a *siStack not *siStorage.

Quote:

---

If you look at the code again you will see that "this" is a *siStack not *siStorage.

---

It doesn't change the problem.
Your >> operator will only work correctly for NULL pointers.
Read again what I explained to understand what the problem with your operator is.

P.S
Pointers are good and powerful.
And like with every other great power - it requires great responsibility.
It is good to give pointers (or reference) as parameters since this way you don't have to move by copy large amount of data - so they are efficient.
If you don't need polymorphism at the destination, probably reference or even const reference is even better.
You don't have to be shy about using pointers, but you have to understand how to work with them,when you should, and what are the dangers involved.
Your >> operator is a mistake, even if it has nothing to do with your current problem - but I think it very much does.
You should change your pointer to reference, and with that you will both stay slim, and comply with what d_stranz was taking about.

Well, I got two things from this thread:

1) I used ChrisW67's solution and everything works great!
2) I found out that I have a lot to learn about pointers! :o

Thanks everyone for your time, patience, and suggestions.