QTcpSocket readyRead strange behavior

hi,

i've made a simple client/server example :

* The client connects to the server
* the server regulary sends data :
** size of buffer (short)
** buffer index (unsigned long)
** buffer, filled with zeros
* the client receives this data
** reads size
** reads buffer
** prints index.

Packet size should be always the same (19998).
But randomly, the client doesn't read the size, but reads 0. It seems there was a problem while reading the stream.

here's the log of my application :
...
onSocketReadyRead bytes= 7504
Bytes= 7504
Packet size= 19998
not enough data in stream
onSocketReadyRead bytes= 15008
Bytes= 15008
Packet size= 19998
not enough data in stream
onSocketReadyRead bytes= 20000
Bytes= 20000
Packet size= 19998
data read, packet index= 6733
onSocketReadyRead bytes= 0
onSocketReadyRead bytes= 3752
Bytes= 3752
Packet size= 0 <-- ERROR!

here's the code of the QThread client :

Code:

---

Client::Client(QObject *parent) : QThread(parent)
{
 
}
 
Client::~Client()
{
 
}
 
void Client::run()
{
	pSocket = new QTcpSocket(this);
 
	connect(pSocket, SIGNAL(readyRead()), this, SLOT(onSocketReadyRead()));
 
	pSocket->connectToHost("192.168.50.77", 1234);
	if (!pSocket->waitForConnected())
	{
		qDebug()<<"Unable to connect";
		return;
	}
 
	exec();
}
 
void Client::onSocketReadyRead()
{
	short sSize;
	char* data;
	unsigned long lPacketIndex;
	qint64 nBytes = pSocket->bytesAvailable();
 
	qDebug()<<"onSocketReadyRead bytes="<<nBytes;
 
	if (nBytes < 2)
		return;
	qDebug()<<"Bytes="<<nBytes;
	pSocket->peek((char*)&sSize, 2);
 
	qDebug()<<"Packet size="<<sSize;
	if (sSize != 19998)
	{
		QMessageBox::critical(NULL, tr("error"), tr("This is it"));
		pSocket->disconnectFromHost();
		return;
	}
 
	if (nBytes < 2 + sSize)
	{
		qDebug()<<"  not enough data in stream";
		return;
	}
 
	data = new char[sSize];
 
	pSocket->read((char*)&sSize, 2);
	pSocket->read(data, sSize);
 
	memcpy((char*)&lPacketIndex, data, 4);
 
	qDebug()<<"  data read, packet index="<<lPacketIndex;
 
	delete data;
}

---

What do i do that i should not do?
Is it a Qt bug?
This problem is random but can be reproduced more easily with a corrupted network (bad wifi connection, etc...)
it MAY be caused by TCP packet retransmission on timeout (i've seen such retransmission with wireshark)

thanks

Quote:

---

Packet size should be always the same (19998).

---

I don't think you have control over the size of the underlying packet. (the ones the server or any servers in between are packing).
What you can do is wait in onSocketReadRead() until the at least 19998 bytes are available before you read.

Quote:

---

Originally Posted by naroin

What do i do that i should not do?

---

You assume that if some data arrives then all data arrives which is not the case. TCP is a stream of bytes without any internal structure. The fact that the sender sends 5 different chunks of data doesn't mean it will also arrive in 5 chunks. It may be one chunk or 99999+ chunks.

thats what i do.

i ready 2 bytes, that should contain the packet size (here always 19998), then i read the buffer.
As a consequence, i should always read 19998 as the size, but i dont.

Quote:

---

Originally Posted by naroin

i ready 2 bytes, that should contain the packet size (here always 19998), then i read the buffer.

---

"packet" is the term reserved for Internel Protocol (IP). TCP works in a concept of "segments". So 19998 is definitely not the packet size. You are sending a stream of 19998 bytes, that's it. You never know across how many packets or segments the data spans.

Quote:

---

As a consequence, i should always read 19998 as the size, but i dont.

---

No, you shouldn't. Eventually you will get 20000 (2+19998) bytes at the receiving side but there is no guarantee they will arrive at the same time.

ok, sorry, my terms are not very clear :)
19998 is my DATA size.
i'm sending a stream of 20000 bytes, the first 2 containing the size of the rest of the data (that is, 19998).

i know my data won't arrive at the same time, and if you look at the code, i verify that i have enough data before reading it.
If i do it wrong, i really don't know how i can do right.

What should i change in my code to make it works?

Quote:

---

What should i change in my code to make it works?

---

Do what I answered you in my previous post!

Well, I would definitely not cast char* to short like that. It is possible the sending side doesn't send it the way you expect it and you get bogus data.

Try something like:

Code:

---

QByteArray buffer; // somewhere
void X::onReadyRead() {
  buffer += socket->readAll(); // read all pending data
 
  while(buffer.size()>=2){
    quint16 size = (buffer.at(0) << 8) + buffer.at(1); // 16bit unsigned value
    if(buffer.size() <= 2+size) return;
    QByteArray data = buffer.mid(2, size);
    processData(data);
    buffer.remove(0,2+size);
  }
}

---

Of course adjust the sending side to send the size in the same manner.

high_flyer i really dont understand, i'm sorry if i am stupid, i DONT ASSUME that all data arrives if SOME data arrives.
Check the code:

Code:

---

qint64 nBytes = pSocket->bytesAvailable();
 
	if (nBytes < 2)
		return;
	pSocket->peek((char*)&sSize, 2);
 
if (nBytes < 2 + sSize)
	{
		qDebug()<<"  not enough data in stream";
		return;
	}

---

---

Added after 32 minutes:

wysota, i've got exactly the same problem.

Sometimes, i read a size of 0 instead of 19998.
i've changed the content of my data
memset(data, 1, 19998);
then the size i read is 257. (0x0101)

as a consequence, there seems to be a shift in stream. Can it be due to TCP retransmission on error?
Windows socket bug? Qt bug?

thanks for your help.

no more help?

I don't understand what help you want.
Can you show your current code?
Do you read until you get the buffer in the size you expect?

yes i read until i get the buffer in the size i expect

the code is in my first post... where is it false??

void Client::onSocketReadyRead()

Your code in the first post does NOT wait for the expected size:

Code:

---

qint64 nBytes = pSocket->bytesAvailable(); //you check bytes available
 
	qDebug()<<"onSocketReadyRead bytes="<<nBytes;
 
	if (nBytes < 2) // it could be you get 0 or 1 bytes, so you get out here. But lets say you got 3, so we go on.
		return;
	qDebug()<<"Bytes="<<nBytes;
	pSocket->peek((char*)&sSize, 2);//why do you use peek and not bytesAvailable? also, why do you peel only 2 bytes ahead? it will alway be at most 2 bytes this the following if will always get out.
 
	qDebug()<<"Packet size="<<sSize;
 
if (sSize != 19998) //so we didn't read 19998, (sSize will be at most 2 because of the peek) which is the size you are waiting for, so we get in to the if.
	{
		QMessageBox::critical(NULL, tr("error"), tr("This is it"));
		pSocket->disconnectFromHost();
		return; //And what do you do? you get out!!
	}
 
	if (nBytes < 2 + sSize)
	{
		qDebug()<<"  not enough data in stream";
		return;
	}

---

EDIT:

Code:

---

if (sSize != 19998)

---

is a tottaly a problem.
Read the peek documentation and be sure to understand it.
sSize should be a buffer, as long as the the data you want to read.
It does not store the size read size, but the read data!
If at all, you probably then use it like this:

Code:

---

int iSize = peel(buff,maySize);
 
if(iSize != iExpectedSize){...}

---

the buffer i want to stream contains:
2 bytes, containing the size of the buffer
rest of the buffer.
in the final program, the size may differ from one packet to another, so i have to put it at the beginning of the send data.
For my tests here, i put it at 19998.

the goal is to read
* the size (in the 2 first bytes)
* then the buffer, corresponding to size just read
then redo the same.

here's my code with comments:

Code:

---

void Client::onSocketReadyRead()
{
	short sSize;
	char* data;
	unsigned long lPacketIndex;
	qint64 nBytes = pSocket->bytesAvailable();
 
	qDebug()<<"onSocketReadyRead bytes="<<nBytes;
 
	if (nBytes < 2) //we haven't read enough data to get the buffer size, return
		return;
	qDebug()<<"Bytes="<<nBytes;
	pSocket->peek((char*)&sSize, 2); //we read 2 bytes, we get the size of the buffer INSIDE these 2 bytes
//here i use peek to get the 2 bytes containing the size, without removing it from the buffer, because i may have not enough data, and i'll have to redo it.
 
	qDebug()<<"Packet size="<<sSize;
/* this IF is only to get easily the error . It's DEBUG code.
    as i KNOW the size must be 19998, if it's not, there was an error.
*/
	if (sSize != 19998) 
	{
//i put a breakpoint here, the code stops.
		QMessageBox::critical(NULL, tr("error"), tr("This is it"));
		pSocket->disconnectFromHost();
		return;
	}
 
	if (nBytes < 2 + sSize) //here we wait to have enough data to read sSize bytes
	{
		qDebug()<<"  not enough data in stream";
		return;
	}
 
	data = new char[sSize];
 
	pSocket->read((char*)&sSize, 2);
	pSocket->read(data, sSize);
 
	memcpy((char*)&lPacketIndex, data, 4);
 
	qDebug()<<"  data read, packet index="<<lPacketIndex;
 
	delete data;
}

---

is it clearer?

wysota gave you the solution, why don't you use it?

I have answrered : i've exactly the same problem with wysota's solution.

As i already said, it works well for a certain time, then it randomly fail. In addition, it does not fail in a "good" network with no TCP retransmission, and often fails in a "bad" network, like low wifi.

well, i'll use directly windows sockets.

Can we see the code for the sender? Also are the sender and the receiver running the same operating system and the same CPU type?

yes, they are both on windows,

here's the server side code :

Code:

---

#define DATA_SIZE 20000
 
Server::Server(QObject *parent)	: QThread(parent)
{
 
}
 
Server::~Server()
{
 
}
 
void Server::run()
{
	pServer = new QTcpServer();
	pMapperReadyRead = new QSignalMapper();
	pMapperDisconnected = new QSignalMapper();
	pMapperError = new QSignalMapper();
 
	connect(pServer, SIGNAL(newConnection()), this, SLOT(onNewConnection()), Qt::DirectConnection);
	connect(pMapperReadyRead, SIGNAL(mapped(QObject*)), this, SLOT(onSocketReadyRead(QObject*)), Qt::DirectConnection);
	connect(pMapperDisconnected, SIGNAL(mapped(QObject*)), this, SLOT(onSocketDisconnected(QObject*)), Qt::DirectConnection);
 
	pServer->listen(QHostAddress::Any, 1234);
 
	//timer d'envoi
	lPacketIndex = 0;
	QTimer* pTimer = new QTimer(this);
	connect(pTimer, SIGNAL(timeout()), this, SLOT(onTimeout()));
 
	pTimer->setSingleShot(false);
	pTimer->start(100);	
 
 
 
	exec();
 
	pServer->close();
	delete pServer;
 
	delete pMapperReadyRead;
	delete pMapperDisconnected;
	delete pMapperError;
}
 
void Server::onNewConnection()
{
	QTcpSocket* pSocket = pServer->nextPendingConnection();
	if (pSocket == NULL)
		return;
 
	qDebug()<<"new connection";
 
	vecClient.push_back(pSocket);
 
	connect(pSocket, SIGNAL(readyRead()), pMapperReadyRead, SLOT(map()), Qt::DirectConnection);
	pMapperReadyRead->setMapping(pSocket, pSocket);
	connect(pSocket, SIGNAL(disconnected()), pMapperDisconnected, SLOT(map()), Qt::DirectConnection);
	pMapperDisconnected->setMapping(pSocket, pSocket);
}
 
void Server::onSocketReadyRead( QObject* pObject )
{
 
}
 
void Server::onSocketDisconnected( QObject* pObject )
{
	int i;
	QTcpSocket* pSocket = (QTcpSocket*)pObject;
 
	qDebug()<<"Socket disconnected";
 
	for (i=0;i<vecClient.size();++i)
	{
		if (pSocket = vecClient[i])
		{
			vecClient.erase(vecClient.begin() + i);
			break;
		}
	}
 
	pSocket->deleteLater();
}
 
void Server::onSocketError( QObject* pObject, QAbstractSocket::SocketError error )
{
	QTcpSocket* pSocket = (QTcpSocket*)pObject;
 
	qDebug()<<"Socket error : "<<pSocket->errorString();
}
 
void Server::onTimeout()
{
	int i;
	char data[DATA_SIZE];
	short sSize = DATA_SIZE - 2;
 
 
	memset(data, 0, DATA_SIZE);
	memcpy(data, &sSize, 2);
	memcpy(data+2, &lPacketIndex, 4);
 
	++lPacketIndex;
 
	for (i=0;i<vecClient.size();++i)
	{	
		vecClient[i]->write(data, DATA_SIZE);
 
	}
}

---

i can also send you VC++ projects, if you need them

This code is not compliant with what I have shown you. You have not implemented what I suggested so do that first and then we can continue.

yeah, that's my server.

the client connects to it, then the server regulary sends data (onTimeout)
In my project, it's the server that pushes bytes in the socket.

as you can see line 108 :

Code:

---

vecClient[i]->write(data, DATA_SIZE);

---