Problems converting raw data to struct when using QDataStream

[Momergil]

Hello!

I'm having some problem with correctly creating and interpreting a struct through a QDataStream "operation". The goal here is the convert a struct into a QByteArray, send by socket to another application which will convert the raw data to the struct again. For now I'm using a Qt-based emulator which converts the raw data into a QByteArray before converting it to the struct.

The struct is the following:

struct re8k_ics_settings_inout
{
    unsigned char header;
 
    struct re8k_ics_settings_inout_nominalvoltages
    {
        unsigned int A;
        unsigned int B;
        unsigned int C;
    } nominalVoltages;
 
    enum re8k_ice_nominal_frequency nominalFrequency;
};

struct re8k_ics_settings_inout
{
    unsigned char header;

    struct re8k_ics_settings_inout_nominalvoltages
    {
        unsigned int A;
        unsigned int B;
        unsigned int C;
    } nominalVoltages;

    enum re8k_ice_nominal_frequency nominalFrequency;
};

To copy to clipboard, switch view to plain text mode 

The enum have only two optional numbers (0 or 1). Here is the code containing both the serialiation and the deserialization:

//
            QByteArray serializedData;
            QDataStream dataStream(&serializedData,QIODevice::WriteOnly);
            dataStream.setVersion(QDataStream::Qt_4_8);
 
            re8k_ics_settings_inout structTemp;
            structTemp.header = RE8K_ICDEF_SETTINGS_INOUT;
 
            structTemp.nominalVoltages.A = 400;
            structTemp.nominalVoltages.B = 9999;
            structTemp.nominalVoltages.C = 1;
 
            structTemp.nominalFrequency = Frequency50Hz;    //0
 
            dataStream << structTemp.header
                       << quint32(structTemp.nominalVoltages.A) << quint32(structTemp.nominalVoltages.B) << quint32(structTemp.nominalVoltages.C)
                       << structTemp.nominalFrequency;
 
            emit signalSendToSystem(serializedData);
 
            qDebug() << "Sended:" << serializedData.toHex() << serializedData.size() << (serializedData.size() == RE8K_ICDEF_SETTINGS_INOUT_SIZE) << RE8K_ICDEF_SETTINGS_INOUT_SIZE
                        << structTemp.nominalVoltages.A << structTemp.nominalVoltages.B << structTemp.nominalVoltages.C;
 
            //
            const QByteArray tempBA = serializedData.mid(0,RE8K_ICDEF_SETTINGS_INOUT_SIZE);
            const void* tempPointer = tempBA.constData();
            const re8k_ics_settings_inout* const newStruct = static_cast< const re8k_ics_settings_inout* >(tempPointer);
 
            qDebug() << "Received:" << tempBA.toHex() << tempBA.size() << (tempBA.toHex() == serializedData.toHex()) << newStruct->nominalVoltages.A << newStruct->nominalVoltages.B << newStruct->nominalVoltages.C;
 
            //
            re8k_ics_settings_inout newStruct2;
 
            QDataStream deserialize(tempBA);
            deserialize.setVersion(QDataStream::Qt_4_8);
 
            deserialize >> newStruct2.header
                        >> newStruct2.nominalVoltages.A
                        >> newStruct2.nominalVoltages.B
                        >> newStruct2.nominalVoltages.C;
 
            qDebug() << "Received 2:" << newStruct2.nominalVoltages.A << newStruct2.nominalVoltages.B << newStruct2.nominalVoltages.C;

//
            QByteArray serializedData;
            QDataStream dataStream(&serializedData,QIODevice::WriteOnly);
            dataStream.setVersion(QDataStream::Qt_4_8);

            re8k_ics_settings_inout structTemp;
            structTemp.header = RE8K_ICDEF_SETTINGS_INOUT;

            structTemp.nominalVoltages.A = 400;
            structTemp.nominalVoltages.B = 9999;
            structTemp.nominalVoltages.C = 1;

            structTemp.nominalFrequency = Frequency50Hz;    //0

            dataStream << structTemp.header
                       << quint32(structTemp.nominalVoltages.A) << quint32(structTemp.nominalVoltages.B) << quint32(structTemp.nominalVoltages.C)
                       << structTemp.nominalFrequency;

            emit signalSendToSystem(serializedData);

            qDebug() << "Sended:" << serializedData.toHex() << serializedData.size() << (serializedData.size() == RE8K_ICDEF_SETTINGS_INOUT_SIZE) << RE8K_ICDEF_SETTINGS_INOUT_SIZE
                        << structTemp.nominalVoltages.A << structTemp.nominalVoltages.B << structTemp.nominalVoltages.C;

            //
            const QByteArray tempBA = serializedData.mid(0,RE8K_ICDEF_SETTINGS_INOUT_SIZE);
            const void* tempPointer = tempBA.constData();
            const re8k_ics_settings_inout* const newStruct = static_cast< const re8k_ics_settings_inout* >(tempPointer);

            qDebug() << "Received:" << tempBA.toHex() << tempBA.size() << (tempBA.toHex() == serializedData.toHex()) << newStruct->nominalVoltages.A << newStruct->nominalVoltages.B << newStruct->nominalVoltages.C;

            //
            re8k_ics_settings_inout newStruct2;

            QDataStream deserialize(tempBA);
            deserialize.setVersion(QDataStream::Qt_4_8);

            deserialize >> newStruct2.header
                        >> newStruct2.nominalVoltages.A
                        >> newStruct2.nominalVoltages.B
                        >> newStruct2.nominalVoltages.C;

            qDebug() << "Received 2:" << newStruct2.nominalVoltages.A << newStruct2.nominalVoltages.B << newStruct2.nominalVoltages.C;

To copy to clipboard, switch view to plain text mode 

And here is the result:

Sended: "0e000001900000270f0000000100000000" 17 false 20 400 9999 1 
Received: "0e000001900000270f0000000100000000" 17 true 654311568 15 1 
Received 2: 400 9999 1

Sended: "0e000001900000270f0000000100000000" 17 false 20 400 9999 1 
Received: "0e000001900000270f0000000100000000" 17 true 654311568 15 1 
Received 2: 400 9999 1

To copy to clipboard, switch view to plain text mode 

So as you can see, convert the "received" QByteArray into the struct with QDataStream is actually working (400 9999 1), but if I try the way I'll have to use (using a reinterpret_cast or, in this case, some static_cast for more safety), I get some bizarre errors. Not only this, but the RE8K_ICDEF_SETTINGS_INOUT_SIZE (which is sizeof(re8k_ics_settings_inout)), returns 20 while the QByteArray produced by the QDataStream has a size of 17. So my interpretation here is that is the QDataStream that is doing the mess. And to add a extra comment, the system works fine if the biggest number I put in nominalVoltages substruct is 255 (but still with the different size problem).


How may I solve this problem?

Thanks,

Momergil

[yeye_olive]

There is a big misunderstanding here.

QDataStream is a facility do serialize/deserialize binary data to/from a platform-independent format. It can help you convert between the internal memory representation of the structure in your application and an external exchange format. You need such a format since you plan to exchange data over the network between two applications, which, as far as we know, might be running on different architectures.

Line 21, serializedData.size() == RE8K_ICDEF_SETTINGS_INOUT_SIZE: why do you expect these numbers to be equal (they aren't)? serializedData.size() is the size of the serialized QByteArray (in the external format, then), while RE8K_ICDEF_SETTINGS_INOUT_SIZE (that is, sizeof(re8k_ics_settings_inout)) is the size of the internal representation.

Line 27: you interpret the serialized data (external format) as if it were in the internal format. Why would the concepts of serialization and deserialization even exist if there was no difference between the internal and external formats?

There is no "mess" with QDataStream. It does exactly what it is meant to do: it helps you convert between structured data in an internal, architecture-dependent format and a flat sequence of bytes in a platform-independent format. It does not even make sense to compare the size or structure of those representations.

If you want to send your structure over a socket to another application, you can either:

• use QDataStream to serialize and deserialize data, but then you need to send the size of the QByteArray along with its contents, because you have no control over this size;
• design your own platform-independent exchange format and serialize/deserialize to/from sequences of bytes of a fixed known size. For example, this format could be: header as a 1-byte unsigned integer, then A, B, C, and nominalFrequency as big-endian 4-byte unsigned integers, for a total of 17 bytes.

[Momergil]

Thanks for the reply, yeye.

why do you expect these numbers to be equal (they aren't)?

Well, for a simple reason: I would expect that, in doing the serialization to a QByteArray, the data contained in the QByteArray (which is underneath an array of chars or similar) would have the same data size of the struct (when it's size is converted to char size, which is what sizeof() does). At best I would expect that QDataStream would have problem with that struct "data separation" issue (that //#pragma pack(1) over the struct declaration should take care of), so if this is what is giving me problems, I'ld say that it's comprehensible. Otherwise I don't see any reason why a QByteArray created from a struct by QDataStream would have a different size that a struct containing the same data when its size is taken by sizeof().

Line 27: you interpret the serialized data (external format) as if it were in the internal format. Why would the concepts of serialization and deserialization even exist if there was no difference between the internal and external formats?

Well, not sure I got your point. But the idea here is always to convert a struct in a series of bytes and read that series of bytes in the other side as a struct again, using QDataStream to do part of the work in the first part.

I guess I will just use another trick I use in another application; that should do the trick. But it's still interesting that I should use QDataStream in this case =T


Thanks,

Momergil

[yeye_olive]

You seem to be understanding that serializing with QDataStream is a non-trivial operation, but at the same time you expect deserialization to be done by bit-for-bit reinterpretation. This is dangerously wrong. If you use QDataStream to serialize, you must use QDataStream to deserialize.

You seem to be understanding that serializing with QDataStream is a non-trivial operation, but at the same time you expect deserialization to be done by bit-for-bit reinterpretation. This is dangerously wrong. If you use QDataStream to serialize, you must use QDataStream to deserialize.

---

Added after 5 minutes:

By the way, your code should not depend on the precise layout of the serialized data. Just let QDataStream encode and decode, or design your own format as I explained before. Also, do not use types such as "unsigned int" in your structure because the range of values of this type depends on the architecture. Use a fixed-size type such as uint32_t.

[Momergil]

If you use QDataStream to serialize, you must use QDataStream to deserialize.

Hmm, I'll have that in mind next time... Manuals I read about using QDataStream to serialize structures to QByteArray and vice-versa didn't mentioned this.

By the way, your code should not depend on the precise layout of the serialized data. Just let QDataStream encode and decode, or design your own format as I explained before. Also, do not use types such as "unsigned int" in your structure because the range of values of this type depends on the architecture. Use a fixed-size type such as uint32_t.

To be honest, I never actually understood this point. For instance, if I go to stdint.h I'll find

typedef unsigned int		uint32_t;

typedef unsigned int		uint32_t;

To copy to clipboard, switch view to plain text mode 

Which means that there is no difference between I writing my code with unsigned int or uint32_t, since they are the same. If a difference in architecture appears between two different projects, they will do wrong independent on the name I use since they will be synonyms. Of course, for clarity purposes, it's still worth the use of the typedef nomenclature.

Thanks,

Momergil

[anda_skoa]

Hmm, I'll have that in mind next time... Manuals I read about using QDataStream to serialize structures to QByteArray and vice-versa didn't mentioned this.

Well, you need a compatible deserialization. Whether that is QDataStream or a compatible bytestream parser doesn't matter.
Like when you serialize to XML using QXmlStreamWriter, you can use any XML parser to deserialization, not necessary QXmlStreamReader

To be honest, I never actually understood this point. For instance, if I go to stdint.h I'll find

typedef unsigned int		uint32_t;

typedef unsigned int		uint32_t;

To copy to clipboard, switch view to plain text mode 

Which means that there is no difference between I writing my code with unsigned int or uint32_t, since they are the same. If a difference in architecture appears between two different projects, they will do wrong independent on the name I use since they will be synonyms. Of course, for clarity purposes, it's still worth the use of the typedef nomenclature.

The point of these types is that they are guaranteed to be of the given size.

Cheers,
_

[yeye_olive]

Hmm, I'll have that in mind next time... Manuals I read about using QDataStream to serialize structures to QByteArray and vice-versa didn't mentioned this.

Read the documentation of the QDataStream class itself; it clearly explains that you serialize with << and deserialize with >>. I do not understand how you can expect serialization to be a complex process and deserialization to be trivial at the same time. Suppose you want to serialize a value v of some type (your structure, for example). The value has an internal (architecture and compiler-specific) memory representation A. Serialization using QDataStream gives you another (architecture independent) memory representation B. What you imply is that B is also a valid internal representation for v, which is completely wrong, as you witnessed yourself in your first post.

To be honest, I never actually understood this point. For instance, if I go to stdint.h I'll find

typedef unsigned int		uint32_t;

typedef unsigned int		uint32_t;

To copy to clipboard, switch view to plain text mode 

Which means that there is no difference between I writing my code with unsigned int or uint32_t, since they are the same. If a difference in architecture appears between two different projects, they will do wrong independent on the name I use since they will be synonyms. Of course, for clarity purposes, it's still worth the use of the typedef nomenclature.

Those two types are the same on the specific platform you are compiling for at the moment, but you cannot expect them to be the same everywhere. unsigned int is a typical machine integer, while uint32_t is guaranteed to be 32 bits wide.

[Momergil]

I do not understand...

Well I think I finally got your point My expectation/presupposition was, indeed, doomed to fail some time.

Those two types are the same on the specific platform you are compiling for at the moment, but you cannot expect them to be the same everywhere. unsigned int is a typical machine integer, while uint32_t is guaranteed to be 32 bits wide.

Well, even with you re-stating that + anda_skoa similar comment I still can't see how would that actually work if in all platforms the included header file with such definitions are the same. Of course, if they change from platform to platform (such as e.g. in a given architecture the uint32_t is a typedef for unsigned int and in another is a typedef for unsigned short), than it makes sense, but if they are always a typedef of the same basic 'data type', so changes in the basic 'data type' will affect their typedef equally anyway - which means that a uint32_t in one architecture will be different to a uint32_t in the other anyway.

Thanks,

Momergil

[yeye_olive]

Well, even with you re-stating that + anda_skoa similar comment I still can't see how would that actually work if in all platforms the included header file with such definitions are the same. Of course, if they change from platform to platform (such as e.g. in a given architecture the uint32_t is a typedef for unsigned int and in another is a typedef for unsigned short), than it makes sense, but if they are always a typedef of the same basic 'data type', so changes in the basic 'data type' will affect their typedef equally anyway - which means that a uint32_t in one architecture will be different to a uint32_t in the other anyway.

The typedef varies from platform to platform and guarantees that uint32_t is 32 bits wide everywhere.