cannot read correct len from socket

[saman_artorious]

I am sending a QByteArray from server to client. the client needs to extract the first four bytes for length of packet and then read the rest of the packet. The unexpected value I get from length of packet though, is a very large value;
what I do is:


int len;
char *data;

client.read((char*)len, 4); // read the first integer for the length
client.read(data, len);



can anyone guess what the problem is?

[Jonny174]

Try client.read((char*)&len, 4);

Added after 4 minutes:



global int blockSize = 0;
...

while (clientConnection->state() != 0)
{
clientConnection->waitForReadyRead(1);

if (clientConnection->bytesAvailable() > 0)
{
if (blockSize == 0)
{
if (clientConnection->bytesAvailable() < sizeof( int ))
continue;

clientConnection->read( (char *)&blockSize, sizeof( int ) );
}

clientConnection->waitForReadyRead(1);

if (clientConnection->bytesAvailable() < blockSize)
continue;

char *msg = new char[ blockSize ];
memset( msg, 0, blockSize );

clientConnection->read( &msg[0], blockSize );
...
blockSize = 0;
}
}

[saman_artorious]

it doesn't help. it gives terrible large numbers.

is the byte array is "47dannie", reading 4 bytes would I think, read 47da, next time it would read nnie, converting them into integers.


int len;
char *data;
client.read((char*)&len, 4);
client.read(&data[0], len);


this is not correct.

[Jonny174]

void MyServer::slotReadClient()
{
QTcpSocket* pClientSocket = (QTcpSocket *)sender();
QDataStream in( pClientSocket );

in.setVersion( QDataStream::Qt_4_7 );

for (;;)
{
if (!m_nNextBlockSize)
{
if (pClientSocket->bytesAvailable() < sizeof( quint16 ))
break;

in >> m_nNextBlockSize;
}

if (pClientSocket->bytesAvailable() < m_nNextBlockSize)
break;

QTime time;
QString str;

in >> time >> str;

QString strMessage = time.toString() + " " + "Client has sended - " + str;
m_ptxt->append( strMessage );

m_nNextBlockSize = 0;

sendToClient( pClientSocket, "Server Response: Received \"" + str + "\"" );
}
}

// ----------------------------------------------------------------------
void MyServer::sendToClient( QTcpSocket *pSocket, const QString &str )
{
QByteArray arrBlock;
QDataStream out( &arrBlock, QIODevice::WriteOnly );

out.setVersion( QDataStream::Qt_4_7 );
out << quint16(0) << QTime::currentTime() << str;

out.device()->seek(0);
out << quint16( arrBlock.size() - sizeof( quint16 ) );

pSocket->write( arrBlock );
}

[saman_artorious]

I am not asking for the way we deal with code in qtcpsocket.
i am asking why the above code which is quite logical in C, does not work out! and how to correct it.

[nix]

What is the value of the big value you get?

[Jonny174]

X3. Maybe not all the data had to come. Shaw all read code.

[saman_artorious]

What is the value of the big value you get?

output:
newConnection
1632843572
879649133
1835094839
926183009
1634558291
1396126830
1851878753
1632843572
879649133
1835094839
926183009

the strange part is when you readLine() and use qbytearray, it contains both the digit and the string "47Saman". Though, in this case, nope

[nix]

Ok, according to the example you gave, your length seems to have a 2 octet size, if the size is fixed try this


QByteArray lenBA(2,'0');
QByteArray data;

client.read(lenBA.data(), 2); // read the first integer for the length
short int length = lenBA.toShort(); // check the conversion is ok here
data.reserve(length);

client.read(data.data(), length );

1632843572 = 0x61533734 and give in ascii : "aS74" look realy close to "47Sa" if you considering byte order issue from brutal conversion from char * to int.

[Lesiok]

the strange part is when you readLine() and use qbytearray, it contains both the digit and the string "47Saman". Though, in this case, nopeAs You see the length of data is transported as characters not as binary int. So You can't read read it as binary data. You must read it as string and convert to int. Another problem : what if data length is bigger then 99 - it is 3 digits.

[saman_artorious]

what if data length is bigger then 99 - it is 3 digits.

Yes, the length of data is random and may vary. I cannot find a way to deal with this.

[Lesiok]

Yes, the length of data is random and may vary. I cannot find a way to deal with this.
This is very simple. Always send fixed number of digits with leading zeros. In example if You want to send 4 digits You should send "0047Saman".
In fact, the problem is the definition of the protocol.

[wysota]

I am sending a QByteArray from server to client. the client needs to extract the first four bytes for length of packet and then read the rest of the packet. The unexpected value I get from length of packet though, is a very large value;
what I do is:


int len;
char *data;

client.read((char*)len, 4); // read the first integer for the length
client.read(data, len);



can anyone guess what the problem is?

How were the first four bytes written to the socket by the server?

[saman_artorious]

How were the first four bytes written to the socket by the server?

By the Client, it was written as such:



void tcpClient::sendCommand(QString com)
{
QByteArray data;

data.append(com); // "47Saman"

if(clientSocket->write (data) == -1)
{
qDebug("sendCommand(QString) Failed");
}
}

[wysota]

If you are writing "47Saman" to the socket, what do you expect to receive as first four bytes on the other end of the communication?

[saman_artorious]

If you are writing "47Saman" to the socket, what do you expect to receive as first four bytes on the other end of the communication?

Wysota, client is sending me a jpeg file. in the server program, I used readAll() to read it all and pixmap it to a label. But, the image does not appear in the label (though the widget background flashes). To solve this, I was told that they would send the size of the jpeg file along with its content. Hence, I need to read the length first, and then realize how many bytes exactly to read.



void MainWindow::readPixmap(QByteArray data)
{
QPixmap pm;

pm.loadFromData(data);

ui->label->setPixmap(pm);
}

[wysota]

Please answer my question.

[saman_artorious]

Please answer my question.

A string of chars.

[wysota]

Then why are you trying to read them as an integer?

[saman_artorious]

Then why are you trying to read them as an integer?

because I have no idea what the length of the integer might be as a character, 2 digits or 5 digits!
that's why I try to read 4 bytes = sizeof(char)

[wysota]

1. sizeof(char) != 4
2. int does not contain decimal digits, it is a binary value ranging from ~ -2^31 to 2^31 (which can hold 10 decimal digits) for a 32 bit integer memory layout
3. reading a value that has not been written cannot end well, can it?
4. you're doing a lot of other errors in your code which completely prove you have no idea what you are doing and just blindly trying to get things working without any understanding what is going on.

Sockets are not magical entities, they do not convert your data, add some artificial data or remove some data -- if you write a sequence of bytes to it, the exact same sequence of bytes will be read by the other end of the communication channel. Don't try to do network programming if you don't understand how it works. Spend two days reading about network programming instead of listening to hints you don't even understand.

[saman_artorious]

Then why are you trying to read them as an integer?

I made a mistake. it is not as chars. it is binary data. So, I need to read the first 4 bytes to get the size of buffer.
The problem is that I cannot simply write an integer to the socket, and read the int 4 bytes in the client.

[wysota]

The problem is that I cannot simply write an integer to the socket, and read the int 4 bytes in the client.

Hmmm... why not?

[saman_artorious]

Then why are you trying to read them as an integer?

I want to read it as binary. appending data length at the beginning with this:


void tcpClient::sendToServer()
{
QByteArray data;

data.append(5);
data.append("Saman");

if(clientSocket->write (data) == -1)
{
qDebug("write() failed");
}
}


in the server, let's read the first 4 bytes, as the size of integer and then read the text.



QByteArray text;

int length;

clientConnection->read((char*)&length, 4);

qDebug() << length;

text.reserve(length);

clientConnection->read(text.data(), length);

clientConnection->flush();


output:
length read is 1835094789 !

what am I missing?

[Lesiok]

QByteArray::append(5) appends one byte with a value of 5 not integer. Do it in such a way :

clientSocket->write (5);//You are sending int(5) and on other side You can read int
clientSocket->write("Saman");

[saman_artorious]

wrong! that would prompt invalid conversion from int to const char*

[wysota]

I want to read it as binary.
You are always reading it as binary, even if it is textual data. It is just a matter of interpreting the data.


what am I missing?
You are missing a couple of hours of learning about computers, their memory and data layout of different data types.

1835094789d = 0x6D615305 = 0x0553616D (swapped byte order) = "\5Sam" interpreted as text. Does that ring a bell?

[ChrisW67]

QByteArray::append(5) appends one byte with a value of 5 not integer. Do it in such a way :

clientSocket->write (5);//You are sending int(5) and on other side You can read int

There's no write(int) function anywhere I can see in the socket/qiodevice interfaces; this won't compile.

[Lesiok]

OK, clientSocket->write(5) is only idea. This code should compile

QByteArray data;
QDataStream m_stream(&data,QIODevice::WriteOnly);
m_stream << 5 << "Saman";
clientSocket->write(data);
saman_artorious read more about QDataStream.

[wysota]

saman_artorious read more about QDataStream.

No, please, don't read about QDataStream. Ignore its existance completely for this project.

[saman_artorious]

I did not know about QDataStream.

Now, it reads the integer fine. However, the amount of data it reads does not depend on how much I allocate according to length.
it always reads the whole thing. How can I control it?



void Server::readState()
{
int len = 0;

QDataStream in(clientConnection);

in >> len;

qDebug() << len;

QByteArray data;

data.reserve(len);

in >> data; // read 20 chars and not all

qDebug() << data.data();

data.clear();

clientConnection->flush();
}


what I sent is of length 50, but only 20 chars would be read, though, it reads all 50 each time:


void tcpClient::sendToServer()
{
QByteArray data;
QDataStream m_stream(&data,QIODevice::WriteOnly);
m_stream << 20 << "xxxxxxxxxx yyyyyyyyyy wwwwwwwwww zzzzzzzzzz aaaaaaaDD"; //text size = 50
clientSocket->write(data);

clientSocket->flush();
}



output:
20
xxxxxxxxxx yyyyyyyyyy wwwwwwwwww zzzzzzzzzz aaaaaaaDD

[yeye_olive]

I did not know about QDataStream.

Now, it reads the integer fine. However, the amount of data it reads does not depend on how much I allocate according to length.
it always reads the whole thing. How can I control it?



void Server::readState()
{
int len = 0;

QDataStream in(clientConnection);

in >> len;

qDebug() << len;

QByteArray data;

data.reserve(len);

in >> data; // read 20 chars and not all

qDebug() << data.data();

data.clear();

clientConnection->flush();
}


what I sent is of length 50, but only 20 chars would be read, though, it reads all 50 each time:


void tcpClient::sendToServer()
{
QByteArray data;
QDataStream m_stream(&data,QIODevice::WriteOnly);
m_stream << 20 << "xxxxxxxxxx yyyyyyyyyy wwwwwwwwww zzzzzzzzzz aaaaaaaDD"; //text size = 50
clientSocket->write(data);

clientSocket->flush();
}



output:
20
xxxxxxxxxx yyyyyyyyyy wwwwwwwwww zzzzzzzzzz aaaaaaaDD

Please stop trying things at random, this is getting out of hand. You went ahead with QDataStream; you should have read its documentation. Do you have any idea what the insertions (<<) and extractions (>>) do?

What you really need is read a bit about serialization and deserialization of basic C/C++ types. Forget about the network for now, just pretend you are designing a platform and architecture-independent binary file format in which to save/from which to load the piece of data you are interested in. This has nothing to do with Qt, by the way.

[ChrisW67]

Now, it reads the integer fine. However, the amount of data it reads does not depend on how much I allocate according to length.
it always reads the whole thing. How can I control it?

Yes, it is doing that because that is the correct behaviour for QDataStream. You don't "control it" but you could at least try to understand it and all that has preceded it in this thread.

If you are serious about understanding then try this experiment: using each approach write the data to a file (substitute a QFile for the QTcpSocket). These files represent exactly what you are sending over the wire. Look at the length of the files. Inspect the files byte by byte (a binary editor or Linux od command). Try changing the numbers and see which bytes change. Look at sizeof(int) how many bytes are you trying to read?

[saman_artorious]

Yes, it is doing that because that is the correct behaviour for QDataStream. You don't "control it" but you could at least try to understand it and all that has preceded it in this thread.

If you are serious about understanding then try this experiment: using each approach write the data to a file (substitute a QFile for the QTcpSocket). These files represent exactly what you are sending over the wire. Look at the length of the files. Inspect the files byte by byte (a binary editor or Linux od command). Try changing the numbers and see which bytes change. Look at sizeof(int) how many bytes are you trying to read?

It is not like I have new to socket programming. I have done it with C many times. The problem when I even try to send and receive data with simple read n write operations it does not give the expected value. I opened a binary file and read the contents, it writes to buffer correctly. the buffer size is also equal to image size. However, writing it to the other side and reading, is not correct.




QString fileName = "/home/saman/Desktop/1.png";
QFile file(fileName);
if (!file.open(QIODevice::ReadOnly)) return;
QByteArray blob = file.readAll();

int len = blob.length();
clientSocket->write(blob, len);


server:


int len;
clientConnection->read((char*)&len, 4);
qDebug() << len;


does not read correctly!

[wysota]

It is not like I have new to socket programming. I have done it with C many times.

Then do it with regular C calls, nobody forces you to use Qt classes for that. With plain C calls you'll be having the exact same "problems" you have now -- you will have to format the data somehow and read it somehow according to all rules that govern asynchronous communication between two autonomous systems.

[saman_artorious]

how to format it? what are those rules?

Added after 1 11 minutes:


Then do it with regular C calls, nobody forces you to use Qt classes for that. With plain C calls you'll be having the exact same "problems" you have now -- you will have to format the data somehow and read it somehow according to all rules that govern asynchronous communication between two autonomous systems.

I think you are wrong. As I tested the program in another system, with standard c++ rcv function, it could receive the image.
So, that means the send or write commands is correct and client program works fine:



clientSocket->write((char*)&size, 4);
clientSocket->write(blob.data(), size);


However, this shows the server program written (the first one) is not correct, which is quite strange. As I debugged it, it received the packet up to 50 KB and then returns 0! On the other, the other one which used std sockets, can read all of it.


int size;
clientConnection->read((char*)&size, 4);
qDebug() << size;

char* image;

qDebug() << size;

image = new char[size];

int size2 = size;
int offset = 0;

while(size2 > 0)
{
int r = clientConnection->read(image+offset, size2);
if(r == 0)
break;
size2 -= r;
offset += r;
}

[wysota]

how to format it? what are those rules?
You should know, it's not the first time you do socket communication.


As I tested the program in another system, with standard c++ rcv function, it could receive the image.
Good for you. Seems the problem is solved, isn't it?

[saman_artorious]

I also tried the receiving buffer code in C. it again only read 49KB and then returns 0, ignoring the rest of file.
I never had such problem in programs, simple send and recv would do it all.
Could I again ask you what I need to do to solve this problem?

[yeye_olive]

I am at a loss. Are you assuming that if you write n bytes using one call to write(), then the other side will read those n bytes in one call to read()? Since you have already programmed with sockets, you must know that TCP is a stream protocol and does not work this way. I recommend checking out http://blog.stephencleary.com/2009/04/tcpip-net-sockets-faq.html for more information on this topic.

I suppose I would do something along these lines to solve your problem:

On the sender's side:
1. Send the length of the byte array as e.g. a 32-bit unsigned integer in big-endian byte order (see quint32 (http://qt-project.org/doc/qt-5.0/qtcore/qtglobal.html#quint32-typedef) and qToBigEndian() (http://qt-project.org/doc/qt-5.0/qtcore/qtendian.html#qToBigEndian-2)).
2. Send the bytes themselves.

On the receiver's side:
1. Store received bytes in a buffer until I have 4 bytes.
2. Interpret those 4 bytes as a 32-bit unsigned integer in big-endian byte order (let n be the value thus read).
3. Store received bytes in a byte array until I have n of them.

This is a quite standard approach to TCP asynchronous communication. You will probably need a few variables to track the current state on the receiver side (e.g. whether you are currently waiting for bytes that are part of the integer or the byte array, how many bytes you are still waiting for, etc.).

[wysota]

I also tried the receiving buffer code in C. it again only read 49KB and then returns 0, ignoring the rest of file.
How unsurprising. I guess if you streamed 16GB of data into the socket, you'd expect to receive that 16GB of data after 2 seconds on the receiving end, right?


I never had such problem in programs, simple send and recv would do it all.
Which means you have no understanding of network programming.


Could I again ask you what I need to do to solve this problem?
You need a good book on networking theory.

[saman_artorious]

If you guys take a look at the code above, you will realize, I am doing the same thing. I think this is bug irrelevant to my code. Besides, with your explanation, I do not think my code is wrong at all.

client:


int size = blob.length();
clientSocket->write((char*)&size, 4);
clientSocket->write(blob.data(), size);


server:



int size;
int y = clientConnection->bytesAvailable();
qDebug() << y;
int x = clientConnection->read((char*)&size, 4);

int size2 = size;
int offset = 0;
char* buffer = new char[size];
while(size2 > 0)
{
int r = clientConnection->read(buffer+offset, size2);
if(r == 0)
break;
size2 -= r;
offset += r;
}
qDebug() << size;

[yeye_olive]

I do not think my code is wrong at all.
And yet it is wrong.



int size = blob.length();
clientSocket->write((char*)&size, 4);

int size;
int x = clientConnection->read((char*)&size, 4);

This is not portable. The size of an int is not necessarily 4 bytes. That is why I mentionned quint32 in my previous post. In addition, this fails if the byte orders are not the same on the server and client machines. That is why I mentioned endianness in my previous post.



int size;
int y = clientConnection->bytesAvailable();
qDebug() << y;
int x = clientConnection->read((char*)&size, 4);

int size2 = size;
int offset = 0;
char* buffer = new char[size];
while(size2 > 0)
{
int r = clientConnection->read(buffer+offset, size2);
if(r == 0)
break;
size2 -= r;
offset += r;
}
qDebug() << size;

QTcpSocket is non blocking by default. When read() returns 0, you must return to the event loop and wait for the readyRead() signal before you can read more data. That is why I mentioned asynchronous-style communication, discussed the storage of the state of your receiver, and provided a link to some explanations on the stream-oriented nature of TCP in my previous post.

Now, are you certain that you have read the material I linked to, and that you have experience in socket programming?

Also I recommend reading the documentation of the Qt classes you use (QIODevice, QAbstractSocket, QTcpSocket, ...) and the various networking examples (http://qt-project.org/doc/qt-5.0/qtnetwork/examples-network.html).

[wysota]

If you guys take a look at the code above, you will realize, I am doing the same thing.
I don't see any code of yours that waits for all the data to arrive.


I think this is bug irrelevant to my code.
Please, don't continue. Just get a book and read it.


Besides, with your explanation, I do not think my code is wrong at all.
Well, it doesn't work so surely it is not correct.



clientSocket->write((char*)&size, 4);
Wrong! What about byte order?



int x = clientConnection->read((char*)&size, 4);
Wrong! What if less than four bytes are available in the socket? What about byte order?



char* buffer = new char[size];
Crashes if size is negative or larger than available memory.



while(size2 > 0)
{
int r = clientConnection->read(buffer+offset, size2);
if(r == 0)
break;
size2 -= r;
offset += r;
}
qDebug() << size;

Wrong! What if less than "size" bytes are available in the socket?