QSerialPort High Speed Serial Reading from board and Logging Issue

[franbogaz]

Hi everyone , i'm newbie to QT. Just struggled with QT about 2 months but i got basic informations about my project. I tried to explain my issue in the best way. Sorry for writing errors. :)

Here is the deal. I'm trying to get sensor values from T.I instruments' Hercules Board which have TMS570LS1227 microcontroller. Main purpose is getting the desired packages from sensor which is connected to board and log it to text file. I managed sending or receiving , logging tasks. But i read so much about QSerialPort losing data and etc. topics. I couldn't figure it out.

For testing the data i'm getting from board i'm only sending 5 byte short number. 1st byte is TypeCheck ( float,decimal and bla bla) for parsing the received package. 2nd and 3rd byte is the short value ( have a union struct no error about it) and the other 2 bytes is CRC(Cyclic Redundancy Check) value of the package.
This was just an information about what i'm trying to do.

I'm sending an counter variable to QT. Sending frequency have to be 10kHz. Means that 10000*5(bytes)*8(bit) = 400000 bit baudrate i should have. I searched forums about QT and it's saying that baudrate 600000 is available in QT. So i set my serial port according to that.

The struggle is that : When i'm getting data from board, first package is not coming like just "5 bytes to read". Incoming bytes in internal serial read buffer showing sometimes 10, sometimes 35, sometimes 500, it depends on how fast i'm doing the job on QT. I managed it with using a Circular Buffer to keep data without losing any of it. E.g: With debugging i saw that; when first package came to QT i'm getting let's say 50 bytes. I'm checking the values of the buffer and it's true.( in this case 50/5=10; last sended number i have to receive is 10). But then second package is coming and i'm checking the received buffer and first 5 byte didn't equal to "11". It can be "45". Then it follows "45,46,47.." .

This means i'm losing data. Why i'm struggling with that ? Is QT doesn't support higher baudrate in QSerialPort ?

I wanted to give the codes as well. You guys can check anything about it. Feel free to critisize my coding as well i wanted my skills to developed :).

ReadyRead Signal is connected to the slot like that.



serial = new QSerialPort(this);
connect(serial,SIGNAL(readyRead()),this,SLOT(toRea dThePort()),Qt::QueuedConnection);


Initializing Serial Port code:



void MyObject::InitSerialPort()
{
/*Circular Buffer Initializing*/

CircularBufferManager_ApiInit(&SerialCircularBufferManager_s,
&SerialBuffer[0],
1,
3000);

//Configure Port Settings;

static bool PortIsOpen;
serial->setPortName("COM7");
serial->setBaudRate(600000); // 460800
serial->setDataBits(QSerialPort::Data8);
serial->setParity(QSerialPort::NoParity);
serial->setStopBits(QSerialPort::OneStop);
serial->setFlowControl(QSerialPort::NoFlowControl);
serial->setReadBufferSize(400);

if(!serial->isOpen())
{
PortIsOpen = serial->open(QIODevice::ReadWrite); // Open the Port;
if(PortIsOpen == 1)
qDebug()<<"Port Has Opened !";
}
serial->clear(QSerialPort::AllDirections);


LogData(GonnaBeWrittenFile,StrBuffer); // Just for writing Header part of the log text.

}

Here is the ReadyRead Signal's connected slot code:



void MyObject::toReadThePort()
{

static uint8 testData[400];
static int counter=0;

NumberOfBytesToRead = serial->bytesAvailable();
if(NumberOfBytesToRead > 0 && serial->isReadable())
{

ArrayBuffer.clear();
ArrayBuffer = serial->readAll();
remain = (NumberOfBytesToRead % ExpectedPacketLength);

memcpy(testData,ArrayBuffer,(NumberOfBytesToRead-remain));

for(counter=0; counter < (NumberOfBytesToRead-remain); counter++)
{
CircularBufferManager_ApiPush(&SerialCircularBufferManager_s,(void*)&testData[counter]);
}
procCounter += (NumberOfBytesToRead-remain);
serial->clear(QSerialPort::AllDirections); // Clear Serial Write and Read Internal Buffers

receiveData();

}

}

For last concern receivedata() function is that just the concerned part :



while(pulledData !='s')
{
CircularBufferManager_ApiPull(&SerialCircularBufferManager_s,&pulledData);
qCnt++;
}


for(qCnt=0; qCnt<(NumberOfBytesToRead-remain); qCnt += ExpectedPacketLength)
// for(qCnt; qCnt<procCounter; qCnt+= ExpectedPacketLength)
{
dummy=0;
if(!(((NumberOfBytesToRead-remain)-qCnt) < ExpectedPacketLength))
{
for(dummyCounter=0;dummyCounter<ExpectedPacketLength;dummyCounter++)
{

dummyArray[dummyCounter] = pulledData;
shortArray[dummyCounter] = pulledData;
CircularBufferManager_ApiPull(&SerialCircularBufferManager_s,&pulledData);

}
}

Note: remain variable is required for circular buffer to don't getting the wrong package data.
Note: ExpectedPacketLength is define variable which can vary but in this code its "5".

[Lesiok]

Why are You clearing serial port buffers in MyObject::toReadThePort() ???? This is the place where you lose data.

[franbogaz]

Why are You clearing serial port buffers in MyObject::toReadThePort() ???? This is the place where you lose data.

Because, i'm reading the data before clearing the serial port. Data is keep coming consistently if i don't clear port there , the other incoming bytes will not going to take a place in buffer.(I thought like that actually.) I think i'm getting all data in buffer then clearing it. If it is wrong where should i clear my serial port for not losing data ?

Thanks for the quick reply by the way.

[marcos.miranda]

Hi, franbogaz.

Verifies that by moving QSerialPort :: Data8 to QSerialPort :: Data5 your project stabilizes.

You can test from QSerialPort :: Data5 to QSerialPort :: Data8.



enum QSerialPort::DataBits

This enum describes the number of data bits used.

Constant Value Description
QSerialPort::Data5 5 The number of data bits in each character is 5. It is used for Baudot code. It generally only makes sense with older equipment such as teleprinters.
QSerialPort::Data6 6 The number of data bits in each character is 6. It is rarely used.
QSerialPort::Data7 7 The number of data bits in each character is 7. It is used for true ASCII. It generally only makes sense with older equipment such as teleprinters.
QSerialPort::Data8 8 The number of data bits in each character is 8. It is used for most kinds of data, as this size matches the size of a byte. It is almost universally used in newer applications.
QSerialPort::UnknownDataBits -1 Unknown number of bits. This value is obsolete. It is provided to keep old source code working. We strongly advise against using it in new code.

[franbogaz]

Hi, franbogaz.

Verifies that by moving QSerialPort :: Data8 to QSerialPort :: Data5 your project stabilizes.

You can test from QSerialPort :: Data5 to QSerialPort :: Data8.



enum QSerialPort::DataBits

This enum describes the number of data bits used.

Constant Value Description
QSerialPort::Data5 5 The number of data bits in each character is 5. It is used for Baudot code. It generally only makes sense with older equipment such as teleprinters.
QSerialPort::Data6 6 The number of data bits in each character is 6. It is rarely used.
QSerialPort::Data7 7 The number of data bits in each character is 7. It is used for true ASCII. It generally only makes sense with older equipment such as teleprinters.
QSerialPort::Data8 8 The number of data bits in each character is 8. It is used for most kinds of data, as this size matches the size of a byte. It is almost universally used in newer applications.
QSerialPort::UnknownDataBits -1 Unknown number of bits. This value is obsolete. It is provided to keep old source code working. We strongly advise against using it in new code.


I tried data5 but serial port is not opening. I'm using QSerialPort::Data8 and each character has 8 bits so its correct i think. Am i right ?
Maybe i didn't understand what you're saying.

Thanks.

[Lesiok]

Because, i'm reading the data before clearing the serial port. Data is keep coming consistently if i don't clear port there , the other incoming bytes will not going to take a place in buffer.(I thought like that actually.) I think i'm getting all data in buffer then clearing it. If it is wrong where should i clear my serial port for not losing data ?

Thanks for the quick reply by the way.
Error in reasoning.
1. During the operation of the data read by serial->readAll buffers appears next portion of the data that you delete in line 22.
2. The data you read in packs of random length (this is the nature of the serial port) and you lose "remain" the last byte.
3. The length of the data read by the serial->readAll () may be greater than serial-> bytesAvailable () read previously.

[marcos.miranda]

Hi, franbogaz.

If I understood your problem well.
Perhaps your problem of loss of information is related to serial cable (manufacturing quality) or connection parameters.

I'm finding it strange that your serial connection configuration.
I've been searching the Internet and I think you should use the following:



1. serial-> setPortName ("COM7");
2. serial-> setBaudRate (19200); // Rate = 19200 * 3.32 * log (4) ... Rate = 38Kbps.
3. serial-> setDataBits (QSerialPort :: Data8);
4. serial-> setParity (QSerialPort :: NoParity);
5. serial-> setStopBits (QSerialPort :: OneStop);
6. serial-> setFlowControl (QSerialPort :: SoftwareControl); // Software flow control (XON / XOFF).
7. serial-> setReadBufferSize (400);


It is more common to use these settings to access equipment via the serial port

[anda_skoa]

Why are you using a QueuedConnection?

Cheers,
_

[franbogaz]

Why are you using a QueuedConnection?

Cheers,
_

Nothing changed when i delete the QueuedConnection. I didn't put it on purpose. Just trying what i see on forums :).

Added after 7 minutes:


Error in reasoning.
1. During the operation of the data read by serial->readAll buffers appears next portion of the data that you delete in line 22.
2. The data you read in packs of random length (this is the nature of the serial port) and you lose "remain" the last byte.
3. The length of the data read by the serial->readAll () may be greater than serial-> bytesAvailable () read previously.

Hi again. I'm aware of losing byte because of "remain" but it is just 1 byte per package. (1 package loss in 500) But i tried your approach. I'm readAll to data to ArrayBuffer and i'm arranging my NumberOfBytesToRead = ArrayBuffer.size();
Actually it became more stable. But i got a problem now like that. First 2500 number is coming without any error then it is going like this : after 2500(number) is recevied -> 3500..3700 -> 7500..7800 -> 11000..12000 -> etc

I'm losing package in somewhere else and it became periodic.



Hi, franbogaz.

If I understood your problem well.
Perhaps your problem of loss of information is related to serial cable (manufacturing quality) or connection parameters.

I'm finding it strange that your serial connection configuration.
I've been searching the Internet and I think you should use the following:



1. serial-> setPortName ("COM7");
2. serial-> setBaudRate (19200); // Rate = 19200 * 3.32 * log (4) ... Rate = 38Kbps.
3. serial-> setDataBits (QSerialPort :: Data8);
4. serial-> setParity (QSerialPort :: NoParity);
5. serial-> setStopBits (QSerialPort :: OneStop);
6. serial-> setFlowControl (QSerialPort :: SoftwareControl); // Software flow control (XON / XOFF).
7. serial-> setReadBufferSize (400);


It is more common to use these settings to access equipment via the serial port


I tried what marcos.miranda said. Baudrate is still 600000 but flow control seems logical. When i tried softwareflowcontrol USB Serial Port stucks and not opening again or closing or receiving or sending data. I'm restarting pc and trying it again and it is same like this. Couldnt understand what's going on about flow control.

[Lesiok]

So make a simple test. Use this :
void MyObject::toReadThePort()
{
qDebug() << serial->readAll();
}
In this way you will find out whether the data are lost in your code or the link.

[anda_skoa]

Nothing changed when i delete the QueuedConnection. I didn't put it on purpose. Just trying what i see on forums :).

Definitely not a good idea.

You want your code to react to the readyRead() signal when it happens, not delayed at some later time.

Cheers,
_

[franbogaz]

So make a simple test. Use this :
void MyObject::toReadThePort()
{
qDebug() << serial->readAll();
}
In this way you will find out whether the data are lost in your code or the link.

Hi man.
I have tried what you said so many times. Here the best result i can get.

https://postimg.org/image/onlcn5mrj/

https://postimg.org/image/mx2blo58f/

I'm counting at qt side simultaneously for testing the COUNTER value which is coming from board. Here you can see lost packages. After that package lost it is happening periodically and always ~3000 data. I checked the sending code ( board side) everything is okay there. I debugged it really carefully. The problem is QT side but i still couldn't figure it out.

list = QSerialPortInfo::standardBaudRates();
I used this code for finding baudrates supported and the maximum i saw is 256000 and i'm using it now. I set the sending frequency to 2.5kHz. Means 2500*5(bytes)*8(bit) = 100000. So 256000 should work with it.


Definitely not a good idea.

You want your code to react to the readyRead() signal when it happens, not delayed at some later time.

Cheers,
_

Now i'm using directConnection. It's better but i still got the problem :/.

Thanks.

[Lesiok]

You did not use methods such as I pointed out - use exactly like mine. The point is you do not do any processing of the received data.

[franbogaz]

Okay but now i see this.

https://postimg.org/image/tocst0rox/

I cant understand from anything here. Every time package is coming in different sizes. Maybe i couldn't understand you fully sorry.

Thanks.

[Lesiok]

Okay but now i see this.

https://postimg.org/image/tocst0rox/

Yes, and now you have to analyze whether the data are complete or not.


I cant understand from anything here. Every time package is coming in different sizes. Maybe i couldn't understand you fully sorry.

Thanks.This is normal. Serial port knows nothing about packets. Serial port is a stream of bytes. You have to divide the stream into packets and handle them. It should look something like this

void MyObject::toReadThePort()
{

if(serial->isReadable())
{

ArrayBuffer.append(serial->readAll());
while(ArrayBuffer.size() >= ExpectedPacketLength)
{
QByteArray one_packet = ArrayBuffer.left(ExpectedPacketLength);
// Here processing of one_packet
ArrayBuffer.remove(0,ExpectedPacketLength);//delete processed packet
}
receiveData();
}
}Of course I not tried to compile it - it is only a sketch.

[franbogaz]

Thanks for quick reply. I will try it and get back to you as soon as possible i can.

[franbogaz]

Yes, and now you have to analyze whether the data are complete or not.

This is normal. Serial port knows nothing about packets. Serial port is a stream of bytes. You have to divide the stream into packets and handle them. It should look something like this

void MyObject::toReadThePort()
{

if(serial->isReadable())
{

ArrayBuffer.append(serial->readAll());
while(ArrayBuffer.size() >= ExpectedPacketLength)
{
QByteArray one_packet = ArrayBuffer.left(ExpectedPacketLength);
// Here processing of one_packet
ArrayBuffer.remove(0,ExpectedPacketLength);//delete processed packet
}
receiveData();
}
}Of course I not tried to compile it - it is only a sketch.

I did what you said. And it became more stable. Actually it is working at lower frequency. But i need 2.5kHz. It's working at 500 Hz now.

I changed my code like this :


while(ArrayBuffer.size() >= ExpectedPacketLength)
{
indexOfType = ArrayBuffer.indexOf('s');

if(indexOfType >= 1)
{
ArrayBuffer.remove(0,indexOfType);
}

QByteArray one_packet = ArrayBuffer.left(ExpectedPacketLength);
ArrayBuffer.remove(0,ExpectedPacketLength);

if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
{
for(dummyCounter=0;dummyCounter<ExpectedPacketLength;dummyCounter++)
{

dummyArray[dummyCounter] = one_packet[dummyCounter];
shortArray[dummyCounter] = one_packet[dummyCounter];
}

CRCValue = crcsum(&dummyArray[0],ExpectedPacketLength,CRC_INIT); // Just Checking the CRCValue. not gonna use anywhere.
ReceiveIsSuccessful = crcverify(&dummyArray[0],ExpectedPacketLength);

if(ReceiveIsSuccessful == 1) // Checking if CRC is OK or not
{ .. parse and logging process is here..}

Now here is the other problem:

When i'm trying to get data at 2.5kHz terminal goes like this :

Everything is okay to some point. But when any error occurs (like CRC can be wrong) QT cant keep the data instaneously at that time.
https://postimg.org/image/ilb3tg0d5/

So i added to my code this :


if(ArrayBuffer.size() >= 20000){
.. then the code is the same i wrote above..

Now it's getting better:

https://postimg.org/image/iml1mv26x/

But as we can see suddenly data are escaping from the buffer after its full. When i increment the 20000 to 100000 or much far away its getting better. However board will always send data constantly.

How can i solve this ? I tried to clear the ArrayBuffer after a chunk of data (20000) have been processed. But it didnt fix it. Any idea about this ?

Because i'm thinking QSerialPort cant handle the high frequency communication. Maybe internal read buffer size isn't enough ?

Thanks for help.

[Lesiok]

Can You tell some about protocol (frame construction) ? As I see a frame starts with letter s.

[kuzulis]

> Because i'm thinking QSerialPort cant handle the high frequency communication.

It is only your wrong thinking.

> serial->setReadBufferSize(400);

Don't touch this at all.

By default the internal buffer of QSP is infinite, and QSP does not lose any data.

> serial->clear(...)

Do not use it.

PS: The problem is in your code.

[franbogaz]

Can You tell some about protocol (frame construction) ? As I see a frame starts with letter s.

Yes. Starting with type -> 1byte for 's', then the short value -> which is 2 byte , and the other 2 byte -> is the CRC Value of these 3 bytes('s',short value byte 1, short value byte 2). Total of 5 bytes i'm receiving.



> Because i'm thinking QSerialPort cant handle the high frequency communication.

It is only your wrong thinking.

> serial->setReadBufferSize(400);

Don't touch this at all.

By default the internal buffer of QSP is infinite, and QSP does not lose any data.

> serial->clear(...)

Do not use it.

PS: The problem is in your code.

Hi kuzulis. I read about internal buffer and i removed that code directly. I was just limited the internal buffer without any reason. Yes that was wrong. I forgot to say that at my last post.

I'm basically thinking that now while parsing and logging processes are going on readyRead is not emitted so i'm losing data ? I can't think any other problem. I checked and debugged my code so many times.

Thanks for the help.

Edit: I also removed the serial->clear(); Not using it anywhere in code.

[Lesiok]

OK. Create an else for if in line 13

if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
{
}
else
{
qDebug() << one_packet;
}

[franbogaz]

OK. Create an else for if in line 13

if(one_packet[0] == 's') // 's' buldur kodu yaz dene!
{
}
else
{
qDebug() << one_packet;
}

Code never fall into there because i always handling with that error with this code.


indexOfType = ArrayBuffer.indexOf('s');

if(indexOfType >= 1)
{
ArrayBuffer.remove(0,indexOfType);
}

If any errorful package comes starting without 's' it will be removed from package and my loss will be just 1 number. But i'm losing 3000-4000 numbers.

[Lesiok]

After ArrayBuffer.remove(0,indexOfType); ArrayBuffer can be shorter than ExpectedPacketLength.
P.S.
If data in frame are binary then every byte in frame can be 's'.

[franbogaz]

Actually it is not a problem because when ArrayBuffer.size below the expected packet length its leaving the slot and waiting for readyRead signal and i debugged it and see that:

Last 6 bytes to be processed e.g : {value,value,'s',value,value,value}. first two value is removed. and became {'s',value,value,value}. Then readyRead is emitted by the QT and data is appending to buffer again and these 4 bytes of ArrayBuffer which inherited by the last process is appended with the other {value,'s',value, bla bla... } data. And i checked their CRC and values. Everything is ok. But sometimes this buffer gets wrong number. Lets say we counted 1 to 150 without error and our buffer stayed like i exampled. Then its not 151 it became 500 or 1000 directly.

Really hard to explain. I'm trying my best. Thanks for trying to help. :o

[Lesiok]

It is a problem because after remove You have in buffer only 4 bytes not 5. Try this code :
if(indexOfType >= 1)
{
ArrayBuffer.remove(0,indexOfType);
if( ArrayBuffer.size() < ExpectedPacketLength )
break;//We exit the loop
}

[franbogaz]

I added this but nothing changed. I will struggle more about this. If i can find the bug will post it to here.

Thanks for all your helps really.

[franbogaz]

Hi again to everyone.

I figure out the problem it's about while loop when readyread signal is emitted in the parsing state. Now if i remove the loops and just emit the readyread signal and directly parse , it became correct but really slow because everytime when data is received parsing will be done.

Have i got any option to receive data and parse it simultaneously. How can i use serial thread and my object simultaneously ? Just need a documentation or an idea. I searched documentation but couldn't understand fully. Thanks a lot.

[Lesiok]

About which loop you speak ? Basically, you do not need an additional thread to handle serial port.

[franbogaz]

About which loop you speak ? Basically, you do not need an additional thread to handle serial port.


while(ArrayBuffer.size() >= ExpectedPacketLength)

here this is the loop i mention about. When parsing is going on readAll can't get the data while loop was going on. I'm thinking creating another thread and when run it arraybuffer will be parsed simultaneously while reading data from serial port is going on.

Am i thinking wrong ?

Thanks.

[Lesiok]

What are You doing in "parsing" ?
If signal emiter and receiver are both in this same thread sending signal is like normal function calling. So that a new signal is issued only after the previous service.