Problem in converting Large QDomDocument to QByteArray

[yeye_olive]

Actually olive, using a delimiter isn't feasible as many times we need to transfer binary data. There is a possibility of the delimiter being present in the ACTUAL_DATA.

You could use a hybrid protocol using length-prefixing for some messages and delimiters for others. Just prefix each message with a byte indicating which one of the two conventions is used for that message. This allows you to choose length-prefixing for binary data with predictable length, and delimiters for XML. This is not difficult to do, and may be the best approach in your situation.

So, your first option looks the best but then again it's not possible to serialize the whole data before sending because there are chances of memory segmentation happening when no.of records are huge. So we are planning to add the following modifications to our protocol:-

1) Reading from the database, forming the XML and then sending over the socket.....everything will happen frame by frame as told by you.
2) Each frame will have the usual HEADER containing the length of ACTUAL_DATA present in that particular frame.
3) Modification:- The first frame will have the additional info of the TOTAL NO.OF RECORDS that we would be send gradually. So at the receiver end we would keep looping until the no.of parsed records becomes equal to the value present in the HEADER of first frame.

OK, that works too. Besides the receiving end can use the total number of records to optimize the allocation of the internal structures for storing the database.

4) I am planning to skip the following step suggested by you. I hope this shouldn't be an issue:-

streamWriter.writeStartDocument();
streamWriter.writeEndDocument();
Reason-  I don't want the following header in my XML:  <?xml version="1.0" encoding="UTF-8"?>

streamWriter.writeStartDocument();
streamWriter.writeEndDocument();
Reason-  I don't want the following header in my XML:  <?xml version="1.0" encoding="UTF-8"?>

To copy to clipboard, switch view to plain text mode 

Is there a good reason for not including this small header? Frankly it does not weigh much compared to the huge database. The documentation for QXmlStreamWriter does not explicitly state that what you suggest is forbidden, but it does not state that it works as you expect either. I am more concerned about the receiving end. By removing this header (called the XML declaration), you remove the information about the text encoding. It seems that QXmlStreamReader relies on the XML declaration since it does not offer a way to set a codec manually. If I were you I would keep the XML declaration and save myself some trouble.

Finally, if you can completely change the protocol, why do you use XML in the first place? You could encode the database to binary data. For example:

Database = [number of records (integer)][record 1][record 2]...
Record = [UserID (integer)][UserName (string)][UserTye (integer)]
integer = big-endian 32-bit unsigned integer
string = UTF-8 encoded string followed by NUL byte

This format would also be suited to progressive serialization and deserialization.