QSerialPort High Speed Serial Reading from board and Logging Issue

[franbogaz]

Yes, and now you have to analyze whether the data are complete or not.

This is normal. Serial port knows nothing about packets. Serial port is a stream of bytes. You have to divide the stream into packets and handle them. It should look something like this

void MyObject::toReadThePort()
{
 
    if(serial->isReadable())
    {
 
        ArrayBuffer.append(serial->readAll());
        while(ArrayBuffer.size() >= ExpectedPacketLength)
        {
            QByteArray one_packet = ArrayBuffer.left(ExpectedPacketLength);
            // Here processing of one_packet
            ArrayBuffer.remove(0,ExpectedPacketLength);//delete processed packet
        }
        receiveData();
    }
}

void MyObject::toReadThePort()
{

    if(serial->isReadable())
    {

        ArrayBuffer.append(serial->readAll());
        while(ArrayBuffer.size() >= ExpectedPacketLength)
        {
            QByteArray one_packet = ArrayBuffer.left(ExpectedPacketLength);
            // Here processing of one_packet
            ArrayBuffer.remove(0,ExpectedPacketLength);//delete processed packet
        }
        receiveData();
    }
}

To copy to clipboard, switch view to plain text mode 

Of course I not tried to compile it - it is only a sketch.

I did what you said. And it became more stable. Actually it is working at lower frequency. But i need 2.5kHz. It's working at 500 Hz now.

I changed my code like this :

while(ArrayBuffer.size() >= ExpectedPacketLength)
       {
        indexOfType = ArrayBuffer.indexOf('s');
 
        if(indexOfType >= 1)
        {
            ArrayBuffer.remove(0,indexOfType);
        }
 
        QByteArray one_packet = ArrayBuffer.left(ExpectedPacketLength);
        ArrayBuffer.remove(0,ExpectedPacketLength);
 
       if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
            {
                for(dummyCounter=0;dummyCounter<ExpectedPacketLength;dummyCounter++)
                {
 
                      dummyArray[dummyCounter] = one_packet[dummyCounter];
                      shortArray[dummyCounter] = one_packet[dummyCounter];
                }
 
            CRCValue = crcsum(&dummyArray[0],ExpectedPacketLength,CRC_INIT); // Just Checking the CRCValue. not gonna use anywhere.
            ReceiveIsSuccessful = crcverify(&dummyArray[0],ExpectedPacketLength);
 
            if(ReceiveIsSuccessful == 1) // Checking if CRC is OK or not
            { .. parse and logging process is here..}

while(ArrayBuffer.size() >= ExpectedPacketLength)
       {
        indexOfType = ArrayBuffer.indexOf('s');

        if(indexOfType >= 1)
        {
            ArrayBuffer.remove(0,indexOfType);
        }

        QByteArray one_packet = ArrayBuffer.left(ExpectedPacketLength);
        ArrayBuffer.remove(0,ExpectedPacketLength);

       if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
            {
                for(dummyCounter=0;dummyCounter<ExpectedPacketLength;dummyCounter++)
                {

                      dummyArray[dummyCounter] = one_packet[dummyCounter];
                      shortArray[dummyCounter] = one_packet[dummyCounter];
                }

            CRCValue = crcsum(&dummyArray[0],ExpectedPacketLength,CRC_INIT); // Just Checking the CRCValue. not gonna use anywhere.
            ReceiveIsSuccessful = crcverify(&dummyArray[0],ExpectedPacketLength);

            if(ReceiveIsSuccessful == 1) // Checking if CRC is OK or not
            { .. parse and logging process is here..}

To copy to clipboard, switch view to plain text mode 

Now here is the other problem:

When i'm trying to get data at 2.5kHz terminal goes like this :

Everything is okay to some point. But when any error occurs (like CRC can be wrong) QT cant keep the data instaneously at that time.
https://postimg.org/image/ilb3tg0d5/

So i added to my code this :

if(ArrayBuffer.size() >= 20000){
.. then the code is the same i wrote above..

if(ArrayBuffer.size() >= 20000){
.. then the code is the same i wrote above..

To copy to clipboard, switch view to plain text mode 

Now it's getting better:

https://postimg.org/image/iml1mv26x/

But as we can see suddenly data are escaping from the buffer after its full. When i increment the 20000 to 100000 or much far away its getting better. However board will always send data constantly.

How can i solve this ? I tried to clear the ArrayBuffer after a chunk of data (20000) have been processed. But it didnt fix it. Any idea about this ?

Because i'm thinking QSerialPort cant handle the high frequency communication. Maybe internal read buffer size isn't enough ?

Thanks for help.

[Lesiok]

Can You tell some about protocol (frame construction) ? As I see a frame starts with letter s.

[kuzulis]

> Because i'm thinking QSerialPort cant handle the high frequency communication.

It is only your wrong thinking.

> serial->setReadBufferSize(400);

Don't touch this at all.

By default the internal buffer of QSP is infinite, and QSP does not lose any data.

> serial->clear(...)

Do not use it.

PS: The problem is in your code.

[franbogaz]

Can You tell some about protocol (frame construction) ? As I see a frame starts with letter s.

Yes. Starting with type -> 1byte for 's', then the short value -> which is 2 byte , and the other 2 byte -> is the CRC Value of these 3 bytes('s',short value byte 1, short value byte 2). Total of 5 bytes i'm receiving.

> Because i'm thinking QSerialPort cant handle the high frequency communication.

It is only your wrong thinking.

> serial->setReadBufferSize(400);

Don't touch this at all.

By default the internal buffer of QSP is infinite, and QSP does not lose any data.

> serial->clear(...)

Do not use it.

PS: The problem is in your code.

Hi kuzulis. I read about internal buffer and i removed that code directly. I was just limited the internal buffer without any reason. Yes that was wrong. I forgot to say that at my last post.

I'm basically thinking that now while parsing and logging processes are going on readyRead is not emitted so i'm losing data ? I can't think any other problem. I checked and debugged my code so many times.

Thanks for the help.

Edit: I also removed the serial->clear(); Not using it anywhere in code.

[Lesiok]

OK. Create an else for if in line 13

if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
{
}
else
{
    qDebug() << one_packet;
}

if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
{
}
else
{
    qDebug() << one_packet;
}

To copy to clipboard, switch view to plain text mode 

[franbogaz]

OK. Create an else for if in line 13

if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
{
}
else
{
    qDebug() << one_packet;
}

if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
{
}
else
{
    qDebug() << one_packet;
}

To copy to clipboard, switch view to plain text mode 

Code never fall into there because i always handling with that error with this code.

indexOfType = ArrayBuffer.indexOf('s');
 
        if(indexOfType >= 1)
        {
            ArrayBuffer.remove(0,indexOfType);
        }

indexOfType = ArrayBuffer.indexOf('s');

        if(indexOfType >= 1)
        {
            ArrayBuffer.remove(0,indexOfType);
        }

To copy to clipboard, switch view to plain text mode 

If any errorful package comes starting without 's' it will be removed from package and my loss will be just 1 number. But i'm losing 3000-4000 numbers.

[Lesiok]

After ArrayBuffer.remove(0,indexOfType); ArrayBuffer can be shorter than ExpectedPacketLength.
P.S.
If data in frame are binary then every byte in frame can be 's'.

[franbogaz]

Actually it is not a problem because when ArrayBuffer.size below the expected packet length its leaving the slot and waiting for readyRead signal and i debugged it and see that:

Last 6 bytes to be processed e.g : {value,value,'s',value,value,value}. first two value is removed. and became {'s',value,value,value}. Then readyRead is emitted by the QT and data is appending to buffer again and these 4 bytes of ArrayBuffer which inherited by the last process is appended with the other {value,'s',value, bla bla... } data. And i checked their CRC and values. Everything is ok. But sometimes this buffer gets wrong number. Lets say we counted 1 to 150 without error and our buffer stayed like i exampled. Then its not 151 it became 500 or 1000 directly.

Really hard to explain. I'm trying my best. Thanks for trying to help.

[Lesiok]

It is a problem because after remove You have in buffer only 4 bytes not 5. Try this code :

if(indexOfType >= 1)
{
    ArrayBuffer.remove(0,indexOfType);
    if( ArrayBuffer.size() < ExpectedPacketLength )
        break;//We exit the loop
}

if(indexOfType >= 1)
{
    ArrayBuffer.remove(0,indexOfType);
    if( ArrayBuffer.size() < ExpectedPacketLength )
        break;//We exit the loop
}

To copy to clipboard, switch view to plain text mode 

[franbogaz]

I added this but nothing changed. I will struggle more about this. If i can find the bug will post it to here.

Thanks for all your helps really.

[franbogaz]

Hi again to everyone.

I figure out the problem it's about while loop when readyread signal is emitted in the parsing state. Now if i remove the loops and just emit the readyread signal and directly parse , it became correct but really slow because everytime when data is received parsing will be done.

Have i got any option to receive data and parse it simultaneously. How can i use serial thread and my object simultaneously ? Just need a documentation or an idea. I searched documentation but couldn't understand fully. Thanks a lot.

[Lesiok]

About which loop you speak ? Basically, you do not need an additional thread to handle serial port.