Problems with TCP Raw Data

[milot]

Dears,

I am writing a server application for some GPS devices and I am having trouble with the receiving data.

The server is working fine, it receives the header as expected and writes to the device, but the data expected to be received is as shown:

080400000113fc208dff000f14f650209cca80006f00d6040004000403010115031603000
1460000015d0000000113fc17610b000f14ffe0209cc580006e00c0050001000403010115
0316010001460000015e0000000113fc284945000f150f00209cd20000950108040000000
4030101150016030001460000015d0000000113fc267c5b000f150a50209cccc000930068
0400000004030101150016030001460000015b0004

080400000113fc208dff000f14f650209cca80006f00d6040004000403010115031603000
1460000015d0000000113fc17610b000f14ffe0209cc580006e00c0050001000403010115
0316010001460000015e0000000113fc284945000f150f00209cd20000950108040000000
4030101150016030001460000015d0000000113fc267c5b000f150a50209cccc000930068
0400000004030101150016030001460000015b0004

To copy to clipboard, switch view to plain text mode 

but the data received is:

7811a7b728990508100000001100000098020508000000000000000069000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001100000058060508000000000000000011000000000000000000000000000000110000006877e3b7000000000000000011000000e874e3b7000000000000000011000000a873e3b7000000000000000011000000a873e3b7010000000000000021000000c80905080040a7b780a2f8b700000000c80905080040a7b780a2f8b71100000002000000010000000000000011000000a873e3b7020000000000000011000000a873e3b7030000000000000039000000c803050802000000c8090508801c5eebe01e5eebd01e5eeb2f7573722f6c69622f67636f6e762f5554462d31362e736f0000000011000000a873e3b7040000000000000011000000a873e3b70500000000000000190000001404050808000000a80c05080900000001000000290000000100000003000000020000000300000001000000e80005083330322f80d4c0b7b4b0e3b711000000a873e3b7060000000000000011000000a873e3b707000000000000003900000001000000040000000300000001000000

7811a7b728990508100000001100000098020508000000000000000069000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001100000058060508000000000000000011000000000000000000000000000000110000006877e3b7000000000000000011000000e874e3b7000000000000000011000000a873e3b7000000000000000011000000a873e3b7010000000000000021000000c80905080040a7b780a2f8b700000000c80905080040a7b780a2f8b71100000002000000010000000000000011000000a873e3b7020000000000000011000000a873e3b7030000000000000039000000c803050802000000c8090508801c5eebe01e5eebd01e5eeb2f7573722f6c69622f67636f6e762f5554462d31362e736f0000000011000000a873e3b7040000000000000011000000a873e3b70500000000000000190000001404050808000000a80c05080900000001000000290000000100000003000000020000000300000001000000e80005083330322f80d4c0b7b4b0e3b711000000a873e3b7060000000000000011000000a873e3b707000000000000003900000001000000040000000300000001000000

To copy to clipboard, switch view to plain text mode 

and the slot (connected to readyRead() signal) that I am handling the received data is as shown:

// Just for verification
qDebug() << "Bytes Available: " << tcpClient->bytesAvailable();
 
  QDataStream in(tcpClient); 
  quint16 header;
 
  in >> header; // reading header
 
  char *data = new char[header];
 
// reading raw data on the char pointer above
  in.readRawData(data, header);
 
  QByteArray b(data, tcpClient->bytesAvailable());
 
// displaying "crazy" data as shown above
  printf(b.toHex());
 
 
// sending accepted message to the device
// otherwise the device won't send you the data back
  QByteArray ba;
  QDataStream out(&ba, QIODevice::WriteOnly);
  out << true;
 
  tcpClient->write(ba);

// Just for verification
qDebug() << "Bytes Available: " << tcpClient->bytesAvailable();

  QDataStream in(tcpClient); 
  quint16 header;

  in >> header; // reading header

  char *data = new char[header];

// reading raw data on the char pointer above
  in.readRawData(data, header);

  QByteArray b(data, tcpClient->bytesAvailable());

// displaying "crazy" data as shown above
  printf(b.toHex());


// sending accepted message to the device
// otherwise the device won't send you the data back
  QByteArray ba;
  QDataStream out(&ba, QIODevice::WriteOnly);
  out << true;

  tcpClient->write(ba);

To copy to clipboard, switch view to plain text mode 

Any suggestions what I am doing wrong?

Sincerely,
Milot.

[borisbn]

1. change in >> header; to

in.readRawData( &header, sizeof( header ) );

in.readRawData( &header, sizeof( header ) );

To copy to clipboard, switch view to plain text mode 

to avoid read header as text
2. after in.readRawData( data, header ); tcpClient->bytesAvailable() seems to be not equal to header
try

QByteArray b( data, header )

QByteArray b( data, header )

To copy to clipboard, switch view to plain text mode 

3. I think it would be better to use QIODevice's peek function to retrieve header, then look if it greater, then tcpClient->bytesAvailable(), and ( if it is ) go further

[milot]

Hi borisbn,

Thank you for your reply. I was testing until now still no luck.

As per your suggestions, the signature of readRawData is readRawData(char *s, int l); and in my code header is quint16 so I cannot use that. I tried replacing bytesAvailable() with header, but no luck because I keep getting header 0 and bytesAvailable() 959!

[wysota]

Why are you using QDataStream?

[milot]

Because of this statement: The QDataStream class provides serialization of binary data to a QIODevice. and the readRawData() function, and I am thinking that this class is helping me...

Do you have any other suggestion?

Thanks.

[wysota]

Because of this statement: The QDataStream class provides serialization of binary data to a QIODevice.

Was the data serialized on the other end with QDataStream as well?

[milot]

Yes,

Because the devices wait for response from the server, so I sent the response.

First I am able to get the IMEI number from the device using this code:

quint16 hd;
 
 
  QDataStream in(tcpClient);
  in >> hd;
 
  qDebug() << "Header: " << hd;
 
  char *data = new char[hd];
  in.readRawData(data, hd);
 
  QString str = QString::fromUtf8(data);
 
  qDebug() << "IMEI: " << str;

quint16 hd;


  QDataStream in(tcpClient);
  in >> hd;

  qDebug() << "Header: " << hd;

  char *data = new char[hd];
  in.readRawData(data, hd);

  QString str = QString::fromUtf8(data);

  qDebug() << "IMEI: " << str;

To copy to clipboard, switch view to plain text mode 

The device first sends its IMEI number and waits for response codes 00 (that is deny and it stops sending packets) and 01 (server tells the device to proceed with other info). I send this packet to the device using the following:

QByteArray ba;
  QDataStream out(&ba, QIODevice::WriteOnly);
  out << true;
 
  tcpClient->write(ba);

QByteArray ba;
  QDataStream out(&ba, QIODevice::WriteOnly);
  out << true;

  tcpClient->write(ba);

To copy to clipboard, switch view to plain text mode 

Then the first output is like this:

New connection from:  "91.187.105.189" 
Header:  15 
IMEI:  "352848021535568"

New connection from:  "91.187.105.189" 
Header:  15 
IMEI:  "352848021535568"

To copy to clipboard, switch view to plain text mode 

after the server sends the 01 (boolean value in my case) to the device, I get this:

Header:  0 
IMEI:  "x!��"

Header:  0 
IMEI:  "x!��"

To copy to clipboard, switch view to plain text mode 

Which seems to be very strange!

[wysota]

How does the transmission look like on the transmitting end? Mixing serialized and raw data in QDataStream is a bad idea in general, by the way. Also why doesn't the data stream work directly on the socket but instead it operates on some byte array that is then transmitted through the socket? I fail to see the point of using QDataStream in such a situation.

[milot]

I am not able to see the other end because it is a GPS device that transmits and receives data.

So are you suggesting that I should read the packets using only readRawData()?

I cannot find enough documentation on how to treat such situations, if you have something worth sharing I would be more than happy

[wysota]

I am not able to see the other end because it is a GPS device that transmits and receives data.

And it uses QDataStream??????????????????????????? I never heard of a GPS device using QDataStream as its communication protocol... Are you sure of that? Or have I misunderstood your "yes" answer to my question about whether the data was serialized using QDataStream?

[milot]

It uses TCP as communication protocol, for that reason I am trying to read from QTcpSocket using QDataStream!

As far as you can see on my example above, it receives data as expected and sends the data back to the device, because if I don't send data back to the device I am not able to receive any more data. For sending and receiving I data I'm using QDataStream, I pasted code above that I am able to read the header using QDataStream and write to the device using writeRawBytes() function.

[wysota]

QDataStream is not a general purpose stream for reading and writing non-textual data. It is a serialization mechanism with its own protocol. You can't use it to read data from an arbitrary binary stream. Using only readRawBytes() and writeRawBytes() with QDataStream only adds overhead without giving anything in return. If you know how many bytes to read then just read them and interpret them the way the other end of the communication expects you to.

[milot]

Thank you for the clarification.

Now I am avoiding using QDataStream, because in general serialization purposes I use QDataStream very often, and after reading that it has support for readRawData I thought it was the way to go.

Thank you again, whenever I come with a solution I will inform the others in this post as well.

[wysota]

Your problem is probably that you are going out of sync somewhere with your data. You have to remember you can't just read all data that is available in the socket and expect it to contain exactly as many bytes as you would want. Data needs to be buffered and reading needs to be deferred to a time when you are sure the buffer contains all the data you need.