Streaming custom QVariant?

[brcain]

Hello,

How do you stream a custom QVariant? I'm getting an error when using QDataStream to output to a binary file. Error is noted in code below.

Where does the type (i.e. QVariant::Type) get specified? Is this done automatically by Q_DECLARE_METATYPE? I was trying to declare my own value (beyond QVariant::UserType), but I couldn't see where to accomplish it. It appears to just be set by defualt to QVariant::UserType, but I need multiple custom types.

I'm hoping to leverage the Qt serialization capabilities ... handling byte-order, string length, etc.

QDataStream & operator<<(QDataStream & out, const sStatePacket & record)
{
   QVariant variant;
   variant.setValue(record);
   out << variant;   // ERROR OCCURS HERE!!!
   return out;
}
 
QDataStream & operator>>(QDataStream & in, sStatePacket & record)
{
   QVariant variant;
   in >> variant;
   record.time = variant.value<sStatePacket>().time;
   record.pos = variant.value<sStatePacket>().pos;
   record.hpr = variant.value<sStatePacket>().hpr;
   return in;
}
 
struct sStatePacket
{
   float time;
   sPosRecord pos;
   sRotRecord hpr;
};
Q_DECLARE_METATYPE(sStatePacket)

QDataStream & operator<<(QDataStream & out, const sStatePacket & record)
{
   QVariant variant;
   variant.setValue(record);
   out << variant;   // ERROR OCCURS HERE!!!
   return out;
}

QDataStream & operator>>(QDataStream & in, sStatePacket & record)
{
   QVariant variant;
   in >> variant;
   record.time = variant.value<sStatePacket>().time;
   record.pos = variant.value<sStatePacket>().pos;
   record.hpr = variant.value<sStatePacket>().hpr;
   return in;
}

struct sStatePacket
{
   float time;
   sPosRecord pos;
   sRotRecord hpr;
};
Q_DECLARE_METATYPE(sStatePacket)

To copy to clipboard, switch view to plain text mode 

Thank you,
Ben

[wysota]

Is there a particular reason why you want to go through QVariant? Why not stream the data directly?

[brcain]

If I use QVariant, I can leverage Qt's serialization and introspection capability. I can inspect what type the data is when I read it back using QVariant::type(). For example, I might have a binary file that has varying record types (not in any particular order).

Here's a potential code snippet:

void cLogStream::seek(float time)
{
   QVariant variant;
   this->device()->seek(0);
   ...
   while(this->device()->atEnd() == false)
   {
      *this >> variant;
      if(variant.type() == sStatePacketType)
      {
         sStatePacket statePacket = variant.value<sStatePacket>();
         if(statePacket.time ...)
            mBuffer.push_back(variant);
      }
      ...
   }
}

void cLogStream::seek(float time)
{
   QVariant variant;
   this->device()->seek(0);
   ...
   while(this->device()->atEnd() == false)
   {
      *this >> variant;
      if(variant.type() == sStatePacketType)
      {
         sStatePacket statePacket = variant.value<sStatePacket>();
         if(statePacket.time ...)
            mBuffer.push_back(variant);
      }
      ...
   }
}

To copy to clipboard, switch view to plain text mode 

[wysota]

The problem is QVariant doesn't know how to serialize your custom type. It could serialize it as a byte array but it would have nothing to do with portability. If you need something like that then do it exactly this way - store the data as a byte array. Of course you will still have problems how to detect that a particular byte array is of your type, but this is achievable if you hand write the code.

[brcain]

So, does Qt not support serialization of custom types?

As long as the type is registered (using the macros, template functions, and stream operators), I would think that it would be able to properly serialize any object. Other serialization libraries do this. They output "header" information for each type ... just like is done with a QString where it writes QVariant::String, string length, then the string character data.

The key is to make sure that the Variant::Type maps appropriately ... from when written to when read back. If the type isn't found, the header information should contain the size, so that it could be skipped ... incrementing the stream to the next record.

Are you sure there is no serialization support? I'm very surprised. In fact it's a bit of a show-stopper!

[wysota]

So, does Qt not support serialization of custom types?

It does but not through QVariant. You can (de)serialize an object and then place it into the variant.

Other serialization libraries do this.

Other serialization libraries are not portable or they have full introspection (such as java).

They output "header" information for each type ... just like is done with a QString where it writes QVariant::String, string length, then the string character data.

Qt does it the same way. But the header for QVariant can't possibly contain every information to recreate every possible structure in the world. That's why you provide your own implementation of streaming to data stream operator.

Are you sure there is no serialization support? I'm very surprised. In fact it's a bit of a show-stopper!

I'm just saying you should not go through QVariant. You might have noticed there is no streaming operator for QVariant, only a constructor that takes a reference to datastream to deserialize an object into the variant.

[brcain]

Maybe I'm confusing QVariant with a meta-type of some sort. It seems to behave like one.

So, for example, when reading a binary file of various custom types, how would one inspect what the record actually is? Qt's serialization would have to support this. Something has to handle the data as a generically sized "block" for the streaming to work. Otherwise, wouldn't know how to iterate through the binary file.

BTW, I realize that QVariant (on its own) can't recreate every possible structure in the world, but through template-programming was thinking it should be able reconstruct any type (from the type identifier in the header). It (or at least the meta-type system) already has type knowledge through the registration mechanism. And, it (stream class) could skip/report unkown types (again by inspecting header information).

The documentation (that I've come across) is limited on serialization. Do you have anything you'd recommend? Seems like it would be a great tutorial (maybe it is).

BTW, thanks for your help ... appreciate it.

[wysota]

So, for example, when reading a binary file of various custom types, how would one inspect what the record actually is? Qt's serialization would have to support this. Something has to handle the data as a generically sized "block" for the streaming to work. Otherwise, wouldn't know how to iterate through the binary file.

QDataStream is a low level mechanism. You want a high level one thus you have to use this low level mechanism to implement a high level mechanism. Simply speaking, if you want to detect what is the type of data in the stream, stream additional data that will let you detect the type. You can stream the name of the type, for example.

QDataStream stream(...);
QString str;
stream << "string" << str;
...
QDataStream inStream(...);
QString type;
stream >> type;
QVariant var;
if(type=="string"){
  QString s;
  stream >> s;
  var.setValue(s);
}

QDataStream stream(...);
QString str;
stream << "string" << str;
...
QDataStream inStream(...);
QString type;
stream >> type;
QVariant var;
if(type=="string"){
  QString s;
  stream >> s;
  var.setValue(s);
}

To copy to clipboard, switch view to plain text mode 

You can use QMetaType to have automatic coversion of the type id to the name of the type to avoid unnecessary comparisons like in my code above.

BTW, I realize that QVariant (on its own) can't recreate every possible structure in the world, but through template-programming was thinking it should be able reconstruct any type (from the type identifier in the header).

I'm not sure how templates would be helpful here. QVariant is a class on its own, you don't create a new "QVariant" class in each application that uses Q_DECLARE_METATYPE.

The documentation (that I've come across) is limited on serialization. Do you have anything you'd recommend? Seems like it would be a great tutorial (maybe it is).

The only relevant docs are in QDataStream reference. This is really a marginal topic in Qt.

[jpn]

See [WIKI]Using custom data types with Qt[/WIKI].

[brcain]

QDataStream is a low level mechanism. You want a high level one thus you have to use this low level mechanism to implement a high level mechanism. Simply speaking, if you want to detect what is the type of data in the stream, stream additional data that will let you detect the type. You can stream the name of the type, for example.

Thanks again. And I hope I don't sound too critical/whiney ...

This seems like standard serialization framework support that should already exist. Maybe I misread what serialization support is actually there. The reason for serialization is so that you don't have to handle the meta-type information that is wrapped around the user type ... but are rather given the meta-type information that you can cast (from some data buffer) to what actually type is.

Shouldn't serialization framework automatically handle Qt types and provide mechanism for custom types? I do override the QDataStream streaming operators, but that doesn't say anything about the header (type, size, etc.) that should be handled automatically.

I can punt on this / quit beating a dead horse ... am just very surprised.

[brcain]

So, how do you stream from a binary file of varying types in non-deterministic order using QDataStream?

Can you even do this with pre-defined Qt types?

[wysota]

Shouldn't serialization framework automatically handle Qt types and provide mechanism for custom types?

It does. But not through QVariant

I do override the QDataStream streaming operators, but that doesn't say anything about the header (type, size, etc.) that should be handled automatically.

That's your task in the implementation of the operator.

QDataStream is a mechanism do stream binary data to/from a device. It is not a general-purpose serialization mechanism but it can be used to create one. It's main task is to provide a way to store (and retrieve) binary data in some well known internal data format where you know what data you expect.

[brcain]

QDataStream [...] is not a general-purpose serialization mechanism but it can be used to create one.

Okay, I think I finally get it. I read too much into the serialization support.

A serialization framework would be a great future capability. It could be used for binary file streams as well as network streams, etc. ... as a tidey way to packetize all records.

Much thanks,
Ben