Results 1 to 9 of 9

Thread: Problems converting raw data to struct when using QDataStream

  1. #1
    Join Date
    Jun 2011
    Location
    Porto Alegre, Brazil
    Posts
    482
    Thanks
    165
    Thanked 2 Times in 2 Posts
    Qt products
    Qt5
    Platforms
    Unix/X11 Windows

    Default Problems converting raw data to struct when using QDataStream

    Hello!

    I'm having some problem with correctly creating and interpreting a struct through a QDataStream "operation". The goal here is the convert a struct into a QByteArray, send by socket to another application which will convert the raw data to the struct again. For now I'm using a Qt-based emulator which converts the raw data into a QByteArray before converting it to the struct.

    The struct is the following:

    Qt Code:
    1. struct re8k_ics_settings_inout
    2. {
    3. unsigned char header;
    4.  
    5. struct re8k_ics_settings_inout_nominalvoltages
    6. {
    7. unsigned int A;
    8. unsigned int B;
    9. unsigned int C;
    10. } nominalVoltages;
    11.  
    12. enum re8k_ice_nominal_frequency nominalFrequency;
    13. };
    To copy to clipboard, switch view to plain text mode 

    The enum have only two optional numbers (0 or 1). Here is the code containing both the serialiation and the deserialization:

    Qt Code:
    1. //
    2. QByteArray serializedData;
    3. QDataStream dataStream(&serializedData,QIODevice::WriteOnly);
    4. dataStream.setVersion(QDataStream::Qt_4_8);
    5.  
    6. re8k_ics_settings_inout structTemp;
    7. structTemp.header = RE8K_ICDEF_SETTINGS_INOUT;
    8.  
    9. structTemp.nominalVoltages.A = 400;
    10. structTemp.nominalVoltages.B = 9999;
    11. structTemp.nominalVoltages.C = 1;
    12.  
    13. structTemp.nominalFrequency = Frequency50Hz; //0
    14.  
    15. dataStream << structTemp.header
    16. << quint32(structTemp.nominalVoltages.A) << quint32(structTemp.nominalVoltages.B) << quint32(structTemp.nominalVoltages.C)
    17. << structTemp.nominalFrequency;
    18.  
    19. emit signalSendToSystem(serializedData);
    20.  
    21. qDebug() << "Sended:" << serializedData.toHex() << serializedData.size() << (serializedData.size() == RE8K_ICDEF_SETTINGS_INOUT_SIZE) << RE8K_ICDEF_SETTINGS_INOUT_SIZE
    22. << structTemp.nominalVoltages.A << structTemp.nominalVoltages.B << structTemp.nominalVoltages.C;
    23.  
    24. //
    25. const QByteArray tempBA = serializedData.mid(0,RE8K_ICDEF_SETTINGS_INOUT_SIZE);
    26. const void* tempPointer = tempBA.constData();
    27. const re8k_ics_settings_inout* const newStruct = static_cast< const re8k_ics_settings_inout* >(tempPointer);
    28.  
    29. qDebug() << "Received:" << tempBA.toHex() << tempBA.size() << (tempBA.toHex() == serializedData.toHex()) << newStruct->nominalVoltages.A << newStruct->nominalVoltages.B << newStruct->nominalVoltages.C;
    30.  
    31. //
    32. re8k_ics_settings_inout newStruct2;
    33.  
    34. QDataStream deserialize(tempBA);
    35. deserialize.setVersion(QDataStream::Qt_4_8);
    36.  
    37. deserialize >> newStruct2.header
    38. >> newStruct2.nominalVoltages.A
    39. >> newStruct2.nominalVoltages.B
    40. >> newStruct2.nominalVoltages.C;
    41.  
    42. qDebug() << "Received 2:" << newStruct2.nominalVoltages.A << newStruct2.nominalVoltages.B << newStruct2.nominalVoltages.C;
    To copy to clipboard, switch view to plain text mode 

    And here is the result:

    Qt Code:
    1. Sended: "0e000001900000270f0000000100000000" 17 false 20 400 9999 1
    2. Received: "0e000001900000270f0000000100000000" 17 true 654311568 15 1
    3. Received 2: 400 9999 1
    To copy to clipboard, switch view to plain text mode 

    So as you can see, convert the "received" QByteArray into the struct with QDataStream is actually working (400 9999 1), but if I try the way I'll have to use (using a reinterpret_cast or, in this case, some static_cast for more safety), I get some bizarre errors. Not only this, but the RE8K_ICDEF_SETTINGS_INOUT_SIZE (which is sizeof(re8k_ics_settings_inout)), returns 20 while the QByteArray produced by the QDataStream has a size of 17. So my interpretation here is that is the QDataStream that is doing the mess. And to add a extra comment, the system works fine if the biggest number I put in nominalVoltages substruct is 255 (but still with the different size problem).


    How may I solve this problem?

    Thanks,

    Momergil
    May the Lord be with you. Always.

  2. #2
    Join Date
    Oct 2009
    Posts
    483
    Thanked 96 Times in 93 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    There is a big misunderstanding here.

    QDataStream is a facility do serialize/deserialize binary data to/from a platform-independent format. It can help you convert between the internal memory representation of the structure in your application and an external exchange format. You need such a format since you plan to exchange data over the network between two applications, which, as far as we know, might be running on different architectures.

    Line 21, serializedData.size() == RE8K_ICDEF_SETTINGS_INOUT_SIZE: why do you expect these numbers to be equal (they aren't)? serializedData.size() is the size of the serialized QByteArray (in the external format, then), while RE8K_ICDEF_SETTINGS_INOUT_SIZE (that is, sizeof(re8k_ics_settings_inout)) is the size of the internal representation.

    Line 27: you interpret the serialized data (external format) as if it were in the internal format. Why would the concepts of serialization and deserialization even exist if there was no difference between the internal and external formats?

    There is no "mess" with QDataStream. It does exactly what it is meant to do: it helps you convert between structured data in an internal, architecture-dependent format and a flat sequence of bytes in a platform-independent format. It does not even make sense to compare the size or structure of those representations.

    If you want to send your structure over a socket to another application, you can either:
    • use QDataStream to serialize and deserialize data, but then you need to send the size of the QByteArray along with its contents, because you have no control over this size;
    • design your own platform-independent exchange format and serialize/deserialize to/from sequences of bytes of a fixed known size. For example, this format could be: header as a 1-byte unsigned integer, then A, B, C, and nominalFrequency as big-endian 4-byte unsigned integers, for a total of 17 bytes.

  3. The following user says thank you to yeye_olive for this useful post:

    Momergil (3rd July 2014)

  4. #3
    Join Date
    Jun 2011
    Location
    Porto Alegre, Brazil
    Posts
    482
    Thanks
    165
    Thanked 2 Times in 2 Posts
    Qt products
    Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    Thanks for the reply, yeye.

    Quote Originally Posted by yeye_olive View Post
    why do you expect these numbers to be equal (they aren't)?
    Well, for a simple reason: I would expect that, in doing the serialization to a QByteArray, the data contained in the QByteArray (which is underneath an array of chars or similar) would have the same data size of the struct (when it's size is converted to char size, which is what sizeof() does). At best I would expect that QDataStream would have problem with that struct "data separation" issue (that //#pragma pack(1) over the struct declaration should take care of), so if this is what is giving me problems, I'ld say that it's comprehensible. Otherwise I don't see any reason why a QByteArray created from a struct by QDataStream would have a different size that a struct containing the same data when its size is taken by sizeof().

    Quote Originally Posted by yeye_olive View Post
    Line 27: you interpret the serialized data (external format) as if it were in the internal format. Why would the concepts of serialization and deserialization even exist if there was no difference between the internal and external formats?
    Well, not sure I got your point. But the idea here is always to convert a struct in a series of bytes and read that series of bytes in the other side as a struct again, using QDataStream to do part of the work in the first part.

    I guess I will just use another trick I use in another application; that should do the trick. But it's still interesting that I should use QDataStream in this case =T


    Thanks,

    Momergil
    May the Lord be with you. Always.

  5. #4
    Join Date
    Oct 2009
    Posts
    483
    Thanked 96 Times in 93 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    You seem to be understanding that serializing with QDataStream is a non-trivial operation, but at the same time you expect deserialization to be done by bit-for-bit reinterpretation. This is dangerously wrong. If you use QDataStream to serialize, you must use QDataStream to deserialize.

    You seem to be understanding that serializing with QDataStream is a non-trivial operation, but at the same time you expect deserialization to be done by bit-for-bit reinterpretation. This is dangerously wrong. If you use QDataStream to serialize, you must use QDataStream to deserialize.


    Added after 5 minutes:


    By the way, your code should not depend on the precise layout of the serialized data. Just let QDataStream encode and decode, or design your own format as I explained before. Also, do not use types such as "unsigned int" in your structure because the range of values of this type depends on the architecture. Use a fixed-size type such as uint32_t.
    Last edited by yeye_olive; 3rd July 2014 at 16:21.

  6. The following user says thank you to yeye_olive for this useful post:

    Momergil (3rd July 2014)

  7. #5
    Join Date
    Jun 2011
    Location
    Porto Alegre, Brazil
    Posts
    482
    Thanks
    165
    Thanked 2 Times in 2 Posts
    Qt products
    Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    Quote Originally Posted by yeye_olive View Post
    If you use QDataStream to serialize, you must use QDataStream to deserialize.
    Hmm, I'll have that in mind next time... Manuals I read about using QDataStream to serialize structures to QByteArray and vice-versa didn't mentioned this.

    Quote Originally Posted by yeye_olive View Post
    By the way, your code should not depend on the precise layout of the serialized data. Just let QDataStream encode and decode, or design your own format as I explained before. Also, do not use types such as "unsigned int" in your structure because the range of values of this type depends on the architecture. Use a fixed-size type such as uint32_t.
    To be honest, I never actually understood this point. For instance, if I go to stdint.h I'll find
    Qt Code:
    1. typedef unsigned int uint32_t;
    To copy to clipboard, switch view to plain text mode 

    Which means that there is no difference between I writing my code with unsigned int or uint32_t, since they are the same. If a difference in architecture appears between two different projects, they will do wrong independent on the name I use since they will be synonyms. Of course, for clarity purposes, it's still worth the use of the typedef nomenclature.

    Thanks,

    Momergil
    May the Lord be with you. Always.

  8. #6
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,420
    Thanks
    37
    Thanked 1,545 Times in 1,495 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    Quote Originally Posted by Momergil View Post
    Hmm, I'll have that in mind next time... Manuals I read about using QDataStream to serialize structures to QByteArray and vice-versa didn't mentioned this.
    Well, you need a compatible deserialization. Whether that is QDataStream or a compatible bytestream parser doesn't matter.
    Like when you serialize to XML using QXmlStreamWriter, you can use any XML parser to deserialization, not necessary QXmlStreamReader


    Quote Originally Posted by Momergil View Post
    To be honest, I never actually understood this point. For instance, if I go to stdint.h I'll find
    Qt Code:
    1. typedef unsigned int uint32_t;
    To copy to clipboard, switch view to plain text mode 

    Which means that there is no difference between I writing my code with unsigned int or uint32_t, since they are the same. If a difference in architecture appears between two different projects, they will do wrong independent on the name I use since they will be synonyms. Of course, for clarity purposes, it's still worth the use of the typedef nomenclature.
    The point of these types is that they are guaranteed to be of the given size.

    Cheers,
    _

  9. The following user says thank you to anda_skoa for this useful post:

    Momergil (4th July 2014)

  10. #7
    Join Date
    Oct 2009
    Posts
    483
    Thanked 96 Times in 93 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    Quote Originally Posted by Momergil View Post
    Hmm, I'll have that in mind next time... Manuals I read about using QDataStream to serialize structures to QByteArray and vice-versa didn't mentioned this.
    Read the documentation of the QDataStream class itself; it clearly explains that you serialize with << and deserialize with >>. I do not understand how you can expect serialization to be a complex process and deserialization to be trivial at the same time. Suppose you want to serialize a value v of some type (your structure, for example). The value has an internal (architecture and compiler-specific) memory representation A. Serialization using QDataStream gives you another (architecture independent) memory representation B. What you imply is that B is also a valid internal representation for v, which is completely wrong, as you witnessed yourself in your first post.

    Quote Originally Posted by Momergil View Post
    To be honest, I never actually understood this point. For instance, if I go to stdint.h I'll find
    Qt Code:
    1. typedef unsigned int uint32_t;
    To copy to clipboard, switch view to plain text mode 

    Which means that there is no difference between I writing my code with unsigned int or uint32_t, since they are the same. If a difference in architecture appears between two different projects, they will do wrong independent on the name I use since they will be synonyms. Of course, for clarity purposes, it's still worth the use of the typedef nomenclature.
    Those two types are the same on the specific platform you are compiling for at the moment, but you cannot expect them to be the same everywhere. unsigned int is a typical machine integer, while uint32_t is guaranteed to be 32 bits wide.

  11. The following user says thank you to yeye_olive for this useful post:

    Momergil (4th July 2014)

  12. #8
    Join Date
    Jun 2011
    Location
    Porto Alegre, Brazil
    Posts
    482
    Thanks
    165
    Thanked 2 Times in 2 Posts
    Qt products
    Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    Quote Originally Posted by yeye_olive View Post
    I do not understand...
    Well I think I finally got your point My expectation/presupposition was, indeed, doomed to fail some time.


    Quote Originally Posted by yeye_olive View Post
    Those two types are the same on the specific platform you are compiling for at the moment, but you cannot expect them to be the same everywhere. unsigned int is a typical machine integer, while uint32_t is guaranteed to be 32 bits wide.
    Well, even with you re-stating that + anda_skoa similar comment I still can't see how would that actually work if in all platforms the included header file with such definitions are the same. Of course, if they change from platform to platform (such as e.g. in a given architecture the uint32_t is a typedef for unsigned int and in another is a typedef for unsigned short), than it makes sense, but if they are always a typedef of the same basic 'data type', so changes in the basic 'data type' will affect their typedef equally anyway - which means that a uint32_t in one architecture will be different to a uint32_t in the other anyway.

    Thanks,

    Momergil
    May the Lord be with you. Always.

  13. #9
    Join Date
    Oct 2009
    Posts
    483
    Thanked 96 Times in 93 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Problems converting raw data to struct when using QDataStream

    Quote Originally Posted by Momergil View Post
    Well, even with you re-stating that + anda_skoa similar comment I still can't see how would that actually work if in all platforms the included header file with such definitions are the same. Of course, if they change from platform to platform (such as e.g. in a given architecture the uint32_t is a typedef for unsigned int and in another is a typedef for unsigned short), than it makes sense, but if they are always a typedef of the same basic 'data type', so changes in the basic 'data type' will affect their typedef equally anyway - which means that a uint32_t in one architecture will be different to a uint32_t in the other anyway.
    The typedef varies from platform to platform and guarantees that uint32_t is 32 bits wide everywhere.

  14. The following user says thank you to yeye_olive for this useful post:

    Momergil (4th July 2014)

Similar Threads

  1. Problems with marshalling a struct to Qt/DBus
    By laumann in forum Qt Programming
    Replies: 3
    Last Post: 21st January 2014, 10:40
  2. Replies: 6
    Last Post: 24th November 2010, 18:59
  3. Problems with QDataStream
    By Doug Broadwell in forum Qt Programming
    Replies: 1
    Last Post: 6th August 2008, 22:02
  4. QDataStream class/struct & stream operators
    By darksaga in forum Qt Programming
    Replies: 1
    Last Post: 1st August 2008, 19:40
  5. sharing struct data throughout program
    By vonCZ in forum Newbie
    Replies: 3
    Last Post: 9th July 2007, 17:21

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.