Formatting a packet with different types then send it over TCP

[SIFE]

What does it have to do with anything?

This change the image of how hexadecimal represent it in 32-bit machine.

You are still using QDataStream, which is not what you really want.

Would you suggest a solution fit my need, I made this try. It looks send the formatted packet as I expect but I don't now how to parse it correctly in other side:

void GuiClient::sendMsg()
{
    QByteArray msg;
 
    int n = ui->msgEdit->text().toAscii().trimmed().size();
 
    msg.append(static_cast<unsigned char> (PROTO_VERSION));
 
    for(int i2 = 0; i2 != sizeof(n); ++i2)
     {
        msg.append((char)(n&(0xFF << i2) >>i2));
     }
 
    msg.append(static_cast<unsigned char>(MSG_TYPE));
    msg.append(ui->msgEdit->text().toAscii().trimmed());
 
 
    qDebug("%d", msg.size());
    qDebug("%s", msg.data());
 
    client->write(msg);
 
    QFile f("test0" + QString::number(++i) + ".bin");
    if (f.open(QIODevice::WriteOnly)) {
    f.write(msg);
    f.close();
    }
 
    ui->msgEdit->clear();
    ui->msgEdit->setFocus();
}

void GuiClient::sendMsg()
{
    QByteArray msg;

    int n = ui->msgEdit->text().toAscii().trimmed().size();

    msg.append(static_cast<unsigned char> (PROTO_VERSION));

    for(int i2 = 0; i2 != sizeof(n); ++i2)
     {
        msg.append((char)(n&(0xFF << i2) >>i2));
     }

    msg.append(static_cast<unsigned char>(MSG_TYPE));
    msg.append(ui->msgEdit->text().toAscii().trimmed());


    qDebug("%d", msg.size());
    qDebug("%s", msg.data());

    client->write(msg);

    QFile f("test0" + QString::number(++i) + ".bin");
    if (f.open(QIODevice::WriteOnly)) {
    f.write(msg);
    f.close();
    }

    ui->msgEdit->clear();
    ui->msgEdit->setFocus();
}

To copy to clipboard, switch view to plain text mode 

void GuiServer::readMsg()
 {
    if(client) //user stored member variable, should be 0 if no client is connected
    {
        int blockSize = 0;
 
        while (client->bytesAvailable() < (int)sizeof(int) + 2) {
                    if (!client->waitForReadyRead(2000)) {
                        //emit error(client->error(), client->errorString());
                        return;
                    }
                }
 
        QByteArray array(client->read(6));
 
        QByteArray array2;
 
        array2[0] = array[1];
        array2[1] = array[2];
        array2[2] = array[3];
        array2[3] = array[4];
 
        memcpy(&blockSize, array2, sizeof(int));
 
        int i = 0;
 
        QFile f("test0" + QString::number(++i) + ".bin");
        if (f.open(QIODevice::WriteOnly)) {
        f.write(array);
        f.close();
        }
 
        qDebug("msg %s", array.data());
        qDebug("protocol %s", array.data()[0]);
        qDebug("size %d", blockSize);
        qDebug("message type %s", array.data()[5]);
 
        while (client->bytesAvailable() < blockSize) {
                     if (!client->waitForReadyRead(2000)) {
                         //emit error(socket.error(), socket.errorString());
                         return;
                     }
                 }
 
        QString msg;
 
        ui->historyEdit->append(msg);
    }
 }

void GuiServer::readMsg()
 {
    if(client) //user stored member variable, should be 0 if no client is connected
    {
        int blockSize = 0;

        while (client->bytesAvailable() < (int)sizeof(int) + 2) {
                    if (!client->waitForReadyRead(2000)) {
                        //emit error(client->error(), client->errorString());
                        return;
                    }
                }

        QByteArray array(client->read(6));

        QByteArray array2;

        array2[0] = array[1];
        array2[1] = array[2];
        array2[2] = array[3];
        array2[3] = array[4];

        memcpy(&blockSize, array2, sizeof(int));

        int i = 0;

        QFile f("test0" + QString::number(++i) + ".bin");
        if (f.open(QIODevice::WriteOnly)) {
        f.write(array);
        f.close();
        }

        qDebug("msg %s", array.data());
        qDebug("protocol %s", array.data()[0]);
        qDebug("size %d", blockSize);
        qDebug("message type %s", array.data()[5]);

        while (client->bytesAvailable() < blockSize) {
                     if (!client->waitForReadyRead(2000)) {
                         //emit error(socket.error(), socket.errorString());
                         return;
                     }
                 }

        QString msg;

        ui->historyEdit->append(msg);
    }
 }

To copy to clipboard, switch view to plain text mode 

Hexdump shows:
Client side:
$ hexdump -Cb test01.bin

00000000  a7 05 05 05 05 a8 65 72  74                       |......ert|
0000000 247 005 005 005 005 250 145 162 164                            
0000009

00000000  a7 05 05 05 05 a8 65 72  74                       |......ert|
0000000 247 005 005 005 005 250 145 162 164                            
0000009

To copy to clipboard, switch view to plain text mode 

Server side:

00000000  a7 05 05 05 05 a8                                 |......|
0000000 247 005 005 005 005 250                                        
0000006

00000000  a7 05 05 05 05 a8                                 |......|
0000000 247 005 005 005 005 250                                        
0000006

To copy to clipboard, switch view to plain text mode 

It seems like the server can read the size of packet but it crashes when it tries to read data.
I am in this project since week(trying to implement my protocol), google tired from me, and I am not sleep well because this(its like nightmare).

[wysota]

This change the image of how hexadecimal represent it in 32-bit machine.

No, it doesn't. The datatype states it is a "32 bit integer", it has nothing to do with its hexadecimal representation. You can as well represent it as 32 binary digits. Hex, bin or dec are just interpretations of the actual data stored in the computer memory according to internal representation chosen by the CPU and its compiler.

I made this try.

A very good try. You should have done that in the beginning.

It looks send the formatted packet as I expect but I don't now how to parse it correctly in other side

Exactly the same way you assemble the message on the sending side.

If you send two bytes here, then you need to extract two bytes there, etc.

I am in this project since week(trying to implement my protocol), google tired from me, and I am not sleep well because this(its like nightmare).

Then take a piece of paper and draw a scheme of how your protocol is supposed to work and the bit-by-bit representation of its PDU. If you don't know how to do it, have a look at diagrams of standard binary protocols such as IP or NTP. If you google for "IP packet diagram", something meaningful will pop up. Practise with a raw stream of bytes until you learn to detect a beginning and end of each protocol unit in the raw stream. If you can't do it then probably the protocol lacks some necessary information and you have to adjust it accordingly.

Only when you fully understand how your protocol should work (and why this and not some other way), then you may start writing an implementation. If you expect to talk to computers with different architecture, pay special attention to the byte order of words.

[ChrisW67]

The binary dumps show the size figure as "05 05 05 05" (hex), or 84 215 045 bytes, with a sent text message length of just three bytes. Something is wrong with your sending logic (Hint: shift by bytes not bits). Perhaps you should research the htonl() and ntohl() macros/functions.

This thread is a reminder of why major high-level network protocols (SMTP, HTTP, NNTP, POP3, IMAP) are text-based and not binary. Not as efficient but a lot easier to get right.

[SIFE]

Then take a piece of paper and draw a scheme of how your protocol is supposed to work and the bit-by-bit representation of its PDU. If you don't know how to do it, have a look at diagrams of standard binary protocols such as IP or NTP. If you google for "IP packet diagram", something meaningful will pop up. Practise with a raw stream of bytes until you learn to detect a beginning and end of each protocol unit in the raw stream. If you can't do it then probably the protocol lacks some necessary information and you have to adjust it accordingly.

Only when you fully understand how your protocol should work (and why this and not some other way), then you may start writing an implementation. If you expect to talk to computers with different architecture, pay special attention to the byte order of words.

I already did that, now I am in the implementation, just because i am new to qt network programming.
Now I have a problem in receiving data:

void GuiServer::readMsg()
 {
    if(client) //user stored member variable, should be 0 if no client is connected
    {
        int blockSize = 0;
 
        while (client->bytesAvailable() < (int)sizeof(int) + 2) {
                    if (!client->waitForReadyRead(2000)) {
                        //emit error(client->error(), client->errorString());
                        return;
                    }
                }
 
        QByteArray data(client->read(6));
 
        QByteArray array2;
        array2.reserve(4);
        array2[0] = data[1];
        array2[1] = data[2];
        array2[2] = data[3];
        array2[3] = data[4];
 
        memcpy(&blockSize, array2, sizeof(int));
 
      /*  for (int j = 0; j < 4; ++j)
            blockSize |= ((int)data[j] & 0x000000ff) << (j * 8);*/
 
 
        QFile f("test0" + QString::number(++i) + ".bin");
        if (f.open(QIODevice::WriteOnly)) {
        f.write(data);
        f.close();
        }
 
        qDebug("size %d", blockSize);
        /*qDebug("msg %s", data.data());
        qDebug("protocol %s", data.data()[0]);
        qDebug("message type %s", data.data()[5]);
*/
        while (client->bytesAvailable() < blockSize) {
                     if (!client->waitForReadyRead(2000)) {
                         return;
                     }
                 }
 
        QString msg(data);
 
        ui->historyEdit->append(msg);
    }
 }

void GuiServer::readMsg()
 {
    if(client) //user stored member variable, should be 0 if no client is connected
    {
        int blockSize = 0;

        while (client->bytesAvailable() < (int)sizeof(int) + 2) {
                    if (!client->waitForReadyRead(2000)) {
                        //emit error(client->error(), client->errorString());
                        return;
                    }
                }

        QByteArray data(client->read(6));

        QByteArray array2;
        array2.reserve(4);
        array2[0] = data[1];
        array2[1] = data[2];
        array2[2] = data[3];
        array2[3] = data[4];

        memcpy(&blockSize, array2, sizeof(int));

      /*  for (int j = 0; j < 4; ++j)
            blockSize |= ((int)data[j] & 0x000000ff) << (j * 8);*/


        QFile f("test0" + QString::number(++i) + ".bin");
        if (f.open(QIODevice::WriteOnly)) {
        f.write(data);
        f.close();
        }

        qDebug("size %d", blockSize);
        /*qDebug("msg %s", data.data());
        qDebug("protocol %s", data.data()[0]);
        qDebug("message type %s", data.data()[5]);
*/
        while (client->bytesAvailable() < blockSize) {
                     if (!client->waitForReadyRead(2000)) {
                         return;
                     }
                 }

        QString msg(data);

        ui->historyEdit->append(msg);
    }
 }

To copy to clipboard, switch view to plain text mode 

six-test.JPG
I sent six messages from client, they all sent correctly however in the are side it seems like I receive the length first and if i sent another message from the client I receive the body. The photo from attachment shows how the server receive each message.

[ChrisW67]

So what happens in the second while loop if all your bytes arrived at once and were available when you entered the routine? What about if only some of the message bytes have been received? What if some of the bytes are delayed?

Have you set a breakpoint and single-stepped this?

[SIFE]

The first loop to read 6 bytes(2 bytes + 4 bytes of size), in the second loop I block until I read the whole packet, so this all I get. If you have better solution please post it.

[wysota]

in the second loop I block until I read the whole packet

What if two packets arrive at once?

If you have better solution please post it.

My universal approach is to use signals and slots, buffer incoming data and process the buffer in a while loop as long as there is enough data in it.

[SIFE]

My universal approach is to use signals and slots, buffer incoming data and process the buffer in a while loop as long as there is enough data in it.

Would you say it in code .

[wysota]

If you want code then search the forum. I can give you a short explanation -- connect to the socket's readyRead() signal, append all pending data to your own buffer and process the buffer in a while loop according to the following pseudo-code logic:

buffer += socket.readAll();
while(buffer.size >= expectedBlockSize) {
  data = buffer.take(expectedBlockSize);
  process(data);
}

buffer += socket.readAll();
while(buffer.size >= expectedBlockSize) {
  data = buffer.take(expectedBlockSize);
  process(data);
}

To copy to clipboard, switch view to plain text mode 

Of course expectedBlockSize will depend on what you expect to receive.