Number of bytes receives not the same after each cycle

[AUDI_ENG]

Hi,

I want to receive 9 bytes after each cycle
Im using the QSerialPort to comunicate between a reaspberry pi and an other device (microcontroller)

I write a code

I have perfectly 9 bytes after each cycle but not in the same QbyteArray :

for example :

nb data read = 9 bytes = "011109101809090913"
nb data read = 8 bytes = "0209091018090909"
nb data read = 1 bytes = "13"
nb data read = 8 bytes = "0309091018090908"
nb data read = 1 bytes = "13"
nb data read = 9 bytes = "04110910170909090f"
nb data read = 9 bytes = "050909101809090913"
nb data read = 9 bytes = "061509101809090813"
nb data read = 8 bytes = "0711091018080909"
nb data read = 10 bytes = "1308110910170908090f"

I want to receive exaclty 9 bytes :

If u see the example , you can remark that each byte have an ID(1,2,3....) and 8 other bytes
and when i have 8 bytes, the next byte is the end of the later (8 bytes) ..

i have this code

StopAff->setEnabled(true);
  QByteArray bytes;
  bytes = PortReception->readAll();
  qDebug() <<"nb data read = " << bytes.size() << "   bytes = "<< bytes.toHex();

StopAff->setEnabled(true);
  QByteArray bytes;
  bytes = PortReception->readAll();
  qDebug() <<"nb data read = " << bytes.size() << "   bytes = "<< bytes.toHex();

To copy to clipboard, switch view to plain text mode 

How I can resolve the problem

[Lesiok]

This is normal symptom for each data stream device (serial port, TCP socket etc.). 9 characters in packet is your private assumption. Data stream device do not know about it. You should read stream of data, analyze it and divide for logical packets.
By the way your protocol is very weak. How do you know that for example 01 is the number of the next packet and not its content ?

[anda_skoa]

Work with a local buffer.

All data that comes in is appended to the buffer.
Then you loop over the combined buffer, take chunks of 9 bytes until that is no longer possible.

Cheers,
_

[kuzulis]

Foo::readyRead()
{
    static const int expectedPacketSize = 9; // your 9 bytes
    if (port->bytesAvailable() < expectedPacketSize)
        return; // do nothing
    QByteArray data = port->read(expectedPacketSize);
 
    // do domething processing of your packet
}

Foo::readyRead()
{
    static const int expectedPacketSize = 9; // your 9 bytes
    if (port->bytesAvailable() < expectedPacketSize)
        return; // do nothing
    QByteArray data = port->read(expectedPacketSize);
 
    // do domething processing of your packet
}

To copy to clipboard, switch view to plain text mode 

Of course, I'm skip the parsing and synchronization of incoming packets from an incoming data stream.

[AUDI_ENG]

This is normal symptom for each data stream device (serial port, TCP socket etc.). 9 characters in packet is your private assumption. Data stream device do not know about it. You should read stream of data, analyze it and divide for logical packets.
By the way your protocol is very weak. How do you know that for example 01 is the number of the next packet and not its content ?

Thank you for your reply,

my protocol is :
_ID________ |______ Data______
01_______________8 bytes_____ (always)
02_______________8 bytes_____
03_______________8 bytes_____
04_______________8 bytes_____
05_______________8 bytes_____
06_______________8 bytes_____
..

Work with a local buffer.

All data that comes in is appended to the buffer.
Then you loop over the combined buffer, take chunks of 9 bytes until that is no longer possible.

Cheers,
_

thanks for your reply

what do you mean about local buffer ?

Foo::readyRead()
{
    static const int expectedPacketSize = 9; // your 9 bytes
    if (port->bytesAvailable() < expectedPacketSize)
        return; // do nothing
    QByteArray data = port->read(expectedPacketSize);
 
    // do domething processing of your packet
}

Foo::readyRead()
{
    static const int expectedPacketSize = 9; // your 9 bytes
    if (port->bytesAvailable() < expectedPacketSize)
        return; // do nothing
    QByteArray data = port->read(expectedPacketSize);
 
    // do domething processing of your packet
}

To copy to clipboard, switch view to plain text mode 

Of course, I'm skip the parsing and synchronization of incoming packets from an incoming data stream.

Thanks for your reply

I will try this monday,
with this code, i will lose some packets no ?

[Lesiok]

Thank you for your reply,

my protocol is :
_ID________ |______ Data______
01_______________8 bytes_____ (always)
02_______________8 bytes_____
03_______________8 bytes_____
04_______________8 bytes_____
05_______________8 bytes_____
06_______________8 bytes_____
..

OK, I'm asking once more : How do you know that for example 01 is the packet ID and not its content ?
Is the sender sends the data all the time or only at your request?

[AUDI_ENG]

OK, I'm asking once more : How do you know that for example 01 is the packet ID and not its content ?
Is the sender sends the data all the time or only at your request?

OK, I'm asking once more : How do you know that for example 01 is the packet ID and not its content ?:
__________________________________________________ ___________________________________
because I have 9 bytes : one for ID and 8 for bytes , and I have a point of depart : when I click on start button : I receive data



Is the sender sends the data all the time or only at your request? :
__________________________________________________ ____
I have my application Qt in the raspberry and i receive data from microcontroller

I have 2 buttons : one for start and one for for stop
when i click start, the port serial is openning, and data (packets of 9 bytes) is receiving from µC
I will display this packets on a table ..


You are right, the processing of data in the sender is infinitely, so when i click on the start button, i receive data but i don't know where is the begenning of the packet, !
its an other problem , thank for your tip, but you have any idea about resolving this ?

[kuzulis]

with this code, i will lose some packets no ?

No. All data will be saved (accumulated) in internal buffer of class in background. So, you can read data in any time (even without readyRead() signal, e.g. by a signal from QTimer or by button clicked and so on).

[Lesiok]

You are right, the processing of data in the sender is infinitely, so when i click on the start button, i receive data but i don't know where is the begenning of the packet, !
its an other problem , thank for your tip, but you have any idea about resolving this ?

Finally You saw the problem First question : can you change the controller software ? Second : the data in the package are binary (bytes in the range 00 - FF) ?

[AUDI_ENG]

Finally You saw the problem First question : can you change the controller software ? Second : the data in the package are binary (bytes in the range 00 - FF) ?

Yes, I can change the software in ther sender (Assembly langage) and the data in the package are binary ( 00 - FF) but the data can't be over ~ 400(int) = 190 'H'
I'm thinking to send 2 bytes in the begininning of each package : for exemple A1 - A2 - A3 - A4 .. A16 instead of 01 - 02 - 03 -04 ... 16

---

Added after 10 minutes:

No. All data will be saved (accumulated) in internal buffer of class in background. So, you can read data in any time (even without readyRead() signal, e.g. by a signal from QTimer or by button clicked and so on).

Hi,

I'm testing the code, its work perfectly. So, I have 9 bytes after each cycle, but the beginning of the package is not the ID, it's the last byte :
for example :
byte 8__byte ID__ byte 1__ byte 2__ byte 3__ byte 4__ byte 5__ byte 6__ byte 7
17______01______0D_____ 09_____ 17_____ 08_____ 09_____ 08_____ 08
17______02______0D_____ 09_____ 17_____ 08_____ 09_____ 08_____ 08
17______03______0D_____ 09_____ 17_____ 08_____ 09_____ 08_____ 08
17______04______0D_____ 09_____ 17_____ 08_____ 09_____ 08_____ 08
.................................................. .................................................. .............;

The problem is related to the detection of the beginning of package ?

[Lesiok]

And this is good idea - create a frame with marker.
According to me it would look like this : AA AA id_0 id_1 d_0 d_1 d_2 d_3 d_4 d_5 d_6 d_7 d_8

[AUDI_ENG]

Foo::readyRead()
{
    static const int expectedPacketSize = 9; // your 9 bytes
    if (port->bytesAvailable() < expectedPacketSize)
        return; // do nothing
    QByteArray data = port->read(expectedPacketSize);
 
    // do domething processing of your packet
}

Foo::readyRead()
{
    static const int expectedPacketSize = 9; // your 9 bytes
    if (port->bytesAvailable() < expectedPacketSize)
        return; // do nothing
    QByteArray data = port->read(expectedPacketSize);
 
    // do domething processing of your packet
}

To copy to clipboard, switch view to plain text mode 

Of course, I'm skip the parsing and synchronization of incoming packets from an incoming data stream.

the code works perfectly but I like to understand it deeply, can you explain me how it works, for exemple how it detects 9 bytes without losing data

Thank you

---

Added after 4 minutes:

And this is good idea - create a frame with marker.
According to me it would look like this : AA AA id_0 id_1 d_0 d_1 d_2 d_3 d_4 d_5 d_6 d_7 d_8

Thank you,

but why I want to repeat id_0 id_1 twice ? and you meaning that AA AA , id_0 8bytes , id_1 8bytes , id_2 8bytes...... ??

[Lesiok]

Thank you,
but why I want to repeat id_0 id_1 twice ? and you meaning that AA AA , id_0 8bytes , id_1 8bytes , id_2 8bytes...... ??

If id is one byte send only one of course. I thought that id is 2 bytes.
And NO, I mean : AA AA id_0 8 bytes, AA AA id_1 8 bytes.... Every packet starts with 2 bytes marker AA AA.

[AUDI_ENG]

If id is one byte send only one of course. I thought that id is 2 bytes.
And NO, I mean : AA AA id_0 8 bytes, AA AA id_1 8 bytes.... Every packet starts with 2 bytes marker AA AA.

Ok, i will try this : AA AA id_0 8 bytes , AA AA id_1 8 bytes ...
in Qt application I will change things in the code no when i read data?

[Lesiok]

As we said earlier : collect data into the buffer and then analyze them searching for frame start tag (bytes AA AA). Remember that the frame is 11 bytes no 9.

[AUDI_ENG]

As we said earlier : collect data into the buffer and then analyze them searching for frame start tag (bytes AA AA). Remember that the frame is 11 bytes no 9.

I used this code :

Foo::readyRead(){

static const int expectedPacketSize = 11; // your 11 bytes
if (port->bytesAvailable() < expectedPacketSize)
return; // do nothing

QByteArray data = port->read(expectedPacketSize); // do domething processing of your packet
}

so I have 11 bytes in the data buffer but how I can detect the start of the packet , in the same time when I read data ?
I can't analyse my packet for detecting data after reading, and if I collect all data in the same buffer, its very heavy for application

[Lesiok]

You should declar variable dataBuffer in class Foo and then do something like this :

Foo::readyRead(){
int frameLen = 11;
dataBuffer.append(port->readAll());
if( dataBuffer.size() < frameLen )
    return;// data to short
int indexOfFrame = dataBuffer.indexOf("\xAA\xAA");
if( indexOfFrame < 0)
    return;// frame marker not found
QByteArray oneFrame = dataBuffer.mid(indexOfFrame,frameLen);// we have one frame, do something with this
dataBuffer = dataBuffer.right(indexOfFrame,frameLen);//remove processed data from the buffer
}

Foo::readyRead(){
int frameLen = 11;
dataBuffer.append(port->readAll());
if( dataBuffer.size() < frameLen )
    return;// data to short
int indexOfFrame = dataBuffer.indexOf("\xAA\xAA");
if( indexOfFrame < 0)
    return;// frame marker not found
QByteArray oneFrame = dataBuffer.mid(indexOfFrame,frameLen);// we have one frame, do something with this
dataBuffer = dataBuffer.right(indexOfFrame,frameLen);//remove processed data from the buffer
}

To copy to clipboard, switch view to plain text mode 

[AUDI_ENG]

I'm writing this code before your code, because with yours I'm losing data

if(Port->bytesAvailable() < lenFram){
 
      return; 
}
dataBuffer.append(port->read(lenFram)); // I change your line to this

if(Port->bytesAvailable() < lenFram){

      return; 
}
dataBuffer.append(port->read(lenFram)); // I change your line to this

To copy to clipboard, switch view to plain text mode 

When I'm adding this before your code I have :

bytesreceives = "0aaaaa0c1209080a090908"
index TAG = 1
nb data read = 10 bytes = "aaaa0c1209080a090908"

bytesreceives = "0aaaaa0d0a09090a090909"
index TAG = 1
nb data read = 10 bytes = "aaaa0d0a09090a090909"

bytesreceives = "0aaaaa0e0e09080a080909"
index TAG = 1
nb data read = 10 bytes = "aaaa0e0e09080a080909"

_________________________________________________

with your code (with readAll() ) : I have this :

nb data read = 11 bytes = "aaaa070a09080a0908090a"
nb data read = 11 bytes = "aaaa080a08090a0909080a"
nb data read = 11 bytes = "aaaa0a1709090a0809090a"
nb data read = 11 bytes = "08090aaaaa0d0a08090a09" --> AA is not in the beginning
nb data read = 11 bytes = "09080aaaaa0f1709090a08" --> AA is not in the beginning
nb data read = 11 bytes = "09080aaaaa020a08090a09" --> AA is not in the beginning
nb data read = 11 bytes = "aaaa031209090a0908090a"
nb data read = 11 bytes = "aaaa041209090a0908090a" --> ID = 4
nb data read = 11 bytes = "08090aaaaa070e08090a09" --> AA is not in the beginning (ID = 7) ---> I miss data with ID = 5 and 6
nb data read = 11 bytes = "08090aaaaa091209080a09" --> AA is not in the beginning
nb data read = 11 bytes = "09090aaaaa0b0a09080a09" --> AA is not in the beginning



I have still the problem with the beginning of frames, sometimes, it is AAAA and sometimes other bytes of frames

[Lesiok]

Between lines 8 and 9 of my code should be :

if( (dataBuffer.size() - indexOfFrame) < frameLen )
    return; // frame is not complete

if( (dataBuffer.size() - indexOfFrame) < frameLen )
    return; // frame is not complete

To copy to clipboard, switch view to plain text mode 

Line 10 sholud be :

dataBuffer = dataBuffer.right(indexOfFrame+frameLen);//remove processed data from the buffer

dataBuffer = dataBuffer.right(indexOfFrame+frameLen);//remove processed data from the buffer

To copy to clipboard, switch view to plain text mode 

I have not tested my code. This is just an example of how something should look.

[AUDI_ENG]

Between lines 8 and 9 of my code should be :

if( (dataBuffer.size() - indexOfFrame) < frameLen )
    return; // frame is not complete

if( (dataBuffer.size() - indexOfFrame) < frameLen )
    return; // frame is not complete

To copy to clipboard, switch view to plain text mode 

Line 10 sholud be :

dataBuffer = dataBuffer.right(indexOfFrame+frameLen);//remove processed data from the buffer

dataBuffer = dataBuffer.right(indexOfFrame+frameLen);//remove processed data from the buffer

To copy to clipboard, switch view to plain text mode 

I have not tested my code. This is just an example of how something should look.

I have always 11 bytes and the beginning AAAA (working perfectly)
but I have still the problem about losing data, its normal to losing data (because somtimes the nombre of bytes availables is more then 22 for exemple --> so I lost one frame) ??