QByteArray container and Timer syncronization

[nagabathula]

Hello every one. i have a problem which i am not able to come out from, been trying to get a solution for it past 2 weeks.. I am actually retrieving data from rs232 port, and doing some bit shifting and aligning, after which i am searching for a Start of the Frame 03b702200000 and then trying to save exactly 2944 chars in every row of the data base.. cause 2944 chars is one master frame in this protocol. Now my problem is after the first 3 rows in the data base either data is repeated from the first row or new data is not saved into the data base. i am calling this function every 30 milli seconds once the Acquire button is clicked on the data base.. Someone pls tell me what is the mistake i am doing here. cause the first 2 rows have the exact data after the 3 rd frame is saved into the db again data from the first row is saved in it , instead of the next new frame data to be saved into the 4th row.. I will be very thank full, i don't have any other help for me other then qt center and forum,

void datareceiver::myexport1()
{
    qint64 numBytes ;  
	qint64 i;
	const int blockLength = 2944;
	char buff[4096];
	QByteArray result;
	quint16 r;
	QString covdat;
 	if(port->bytesAvailable())
	{
		inscnt++;
		numBytes = port->bytesAvailable();
    	  if(numBytes%2 == 0)
				numBytes = numBytes;
			else
				numBytes = numBytes-1;
		if(numBytes > sizeof(buff))
			numBytes = sizeof(buff);
			i = port->read(buff,numBytes);
     QByteArray data = (QByteArray::fromRawData(buff,numBytes));  // retreving data on RS232 terminal
	 quint16 r;
//****************** Bit Wise Shift Left Logic and ORing of two Bytes *********************/
 
	   for (int i = 0; i < data.size(); i+=2)
	     {          
			 // r = ((stfram.at(i) << 5) |(stfram.at(i+1)));
			  r = (((quint16) data.at(i) << 5) |(data.at(i+1)));
			  char *ptrR = (char*) &r;
			  result.append(ptrR[1]); 
			  result.append(ptrR[0]);
    	 }
     	    oldat.append(result.toHex());   // QByteArray oldat,result,stfrm ;
 
 //****************** Logic to search for First frame ID and save to db from there*********************/
		   if(datrow ==0)
 
		{			
		   	QByteArray pattern("03b702200000"); 
			QByteArrayMatcher matcher(pattern); 
		        pos = matcher.indexIn(oldat, pos);		
                        datrow++;
		}
		 for(np = pos; np<oldat.size(); np++)
			 {
                                stfram.append(oldat.at(np));
				pos++;
			 }
 
   //************************* Cheacking for the Start of Frame "03b702200000" if true then insert 2944 chars every row******************//
	if((stfram.startsWith("03b702200000")))
			{
				QSqlQuery qry;
				qry.prepare("insert into Bulkdb (Time_Interval, ChannelData) values (?, ?)");
					while ((stfram.length() > blockLength)) {  
						qry.bindValue(0, rcount++);
						qry.bindValue(1, stfram.left(blockLength));
						qry.exec();
						stfram = stfram.mid(blockLength);
 
			}
				qDebug()<< stfram;
			}
	}
// QByteArray oldat,result,stfrm ; Do i have to clear all these containers everytime this function is called sir, i am not doing it right now , since the data is continuous these containers will have data appended into them till the program runs. 
}

void datareceiver::myexport1()
{
    qint64 numBytes ;  
	qint64 i;
	const int blockLength = 2944;
	char buff[4096];
	QByteArray result;
	quint16 r;
	QString covdat;
 	if(port->bytesAvailable())
	{
		inscnt++;
		numBytes = port->bytesAvailable();
    	  if(numBytes%2 == 0)
				numBytes = numBytes;
			else
				numBytes = numBytes-1;
		if(numBytes > sizeof(buff))
			numBytes = sizeof(buff);
			i = port->read(buff,numBytes);
     QByteArray data = (QByteArray::fromRawData(buff,numBytes));  // retreving data on RS232 terminal
	 quint16 r;
//****************** Bit Wise Shift Left Logic and ORing of two Bytes *********************/
   
	   for (int i = 0; i < data.size(); i+=2)
	     {          
			 // r = ((stfram.at(i) << 5) |(stfram.at(i+1)));
			  r = (((quint16) data.at(i) << 5) |(data.at(i+1)));
			  char *ptrR = (char*) &r;
			  result.append(ptrR[1]); 
			  result.append(ptrR[0]);
    	 }
     	    oldat.append(result.toHex());   // QByteArray oldat,result,stfrm ;

 //****************** Logic to search for First frame ID and save to db from there*********************/
		   if(datrow ==0)

		{			
		   	QByteArray pattern("03b702200000"); 
			QByteArrayMatcher matcher(pattern); 
		        pos = matcher.indexIn(oldat, pos);		
                        datrow++;
		}
		 for(np = pos; np<oldat.size(); np++)
			 {
                                stfram.append(oldat.at(np));
				pos++;
			 }
   
   //************************* Cheacking for the Start of Frame "03b702200000" if true then insert 2944 chars every row******************//
	if((stfram.startsWith("03b702200000")))
			{
				QSqlQuery qry;
				qry.prepare("insert into Bulkdb (Time_Interval, ChannelData) values (?, ?)");
					while ((stfram.length() > blockLength)) {  
						qry.bindValue(0, rcount++);
						qry.bindValue(1, stfram.left(blockLength));
						qry.exec();
						stfram = stfram.mid(blockLength);

			}
				qDebug()<< stfram;
			}
	}
// QByteArray oldat,result,stfrm ; Do i have to clear all these containers everytime this function is called sir, i am not doing it right now , since the data is continuous these containers will have data appended into them till the program runs. 
}

To copy to clipboard, switch view to plain text mode 

Is this a timer syncronization problem or am i not clearing the QByteArray Containers properly..
thank you very much hope to see some one help me.

[wysota]

Your code is very unreadable, try simplifying it first. Lines #13-21 can be rewritten as:

qint64 numBytes = bytesAvailable();
QByteArray data = port->read(numBytes % 2 ? numBytes : numBytes-1);

qint64 numBytes = bytesAvailable();
QByteArray data = port->read(numBytes % 2 ? numBytes : numBytes-1);

To copy to clipboard, switch view to plain text mode 

I don't even know what #23-33 do so I won't try simplifying this.

Lines #36-43 are some attempt to synchronize frames although in my opinion you shouldn't be needing this - is there any data between the end of one frame and the beginning of the next one? I don't even know what oldat is...

Lines #44-48 should be:

strfram.append(oldat.mid(pos));
pos = oldat.size(); // I don't get this one completely

strfram.append(oldat.mid(pos));
pos = oldat.size(); // I don't get this one completely

To copy to clipboard, switch view to plain text mode 

And then there is "something" which looks like dumping some data to a database. What do all the previous calculations have to do with it - I have no idea.

In general your whole code should look more or less like this:

QByteArray buffer; // "global" buffer
void datareceiver::myexport1() {
  buffer.append(port->readAll());
  int pos = 0;
  QByteArray pattern("03b702200000");
  QByteArrayMatcher matcher(pattern); 
  forever {
    int newPos = matcher.indexIn(buffer, pos);
    if(newPos==-1) break; // no more frames detected
    const int frameStart = newPos+pattern.size();
 
    QByteArray frameData = buffer.mid(frameStart, 2944);
    if(frameData.size()!=2944) break; // incomplete frame
    insertIntoDatabase(frameData);
    pos = frameStart+2944;
  }
  // sync buffer:
  if(pos>0) buffer.remove(0, pos); // if a frame was found, remove all the processed frames
}

QByteArray buffer; // "global" buffer
void datareceiver::myexport1() {
  buffer.append(port->readAll());
  int pos = 0;
  QByteArray pattern("03b702200000");
  QByteArrayMatcher matcher(pattern); 
  forever {
    int newPos = matcher.indexIn(buffer, pos);
    if(newPos==-1) break; // no more frames detected
    const int frameStart = newPos+pattern.size();

    QByteArray frameData = buffer.mid(frameStart, 2944);
    if(frameData.size()!=2944) break; // incomplete frame
    insertIntoDatabase(frameData);
    pos = frameStart+2944;
  }
  // sync buffer:
  if(pos>0) buffer.remove(0, pos); // if a frame was found, remove all the processed frames
}

To copy to clipboard, switch view to plain text mode 

[ChrisW67]

Here are a number of ways your data can get broken in your code:

• The first data received does not contain a frame header (the sender is asynchronous after all)
• A frame header falls across a buffer boundary.
• The first byte you receive is an odd byte. This will break your processing at line 28 and mangle the data following. You should look for the sequence of bytes that make up your frame header before you process them: this will allow you to synchronise with sets of 2944 bytes.
• You receive up to 4096 bytes into your buffer (buff), process all of the bytes, potentially save some of them, and then discard the remainder when the routine exits (and buff[] is discarded). The receive buffer needs to persist between calls to this routine.
• If datrow is zero (where is that set) then you look for "03b702200000" and assume it is found. What happens if it is not found? Does datrow ever get reset? Does the header ever appear again?

There are probably many more. You lost me at line 50 with some stuff about covdat.

I can see bits of code cobbled together from the various other times you have asked questions about this problem. You really need to break this problem into individual bits that you understand and can test are working before you try to put them together into a whole and process live data.

Just an afterthought: Once every 30 milliseconds you expect to have 2944 bytes over an RS232 interface. That's quite a bit rate: 2944 bytes * 8 bits/byte / 0.03 secs = 785067 bits/sec. About 7x the typical 115400 bits per sec port. Are you sure about this?


Edit: Look closely at Wysota's great simplification of the problem.

Wysota: There's a long history of questions on this same problem. Most of the code above is poorly understood cut-and-paste jobs from other threads, including the mysterious bit mangling in lines 22 to 33.

[nagabathula]

hello sir thanks for the reply..

In line number 23-33 i am actually bit shifting every second byte by 5 times and ORing it with the byte before it and appending the Bit shifted and ORed data into oldat.

From line number 36 - 48, i am looking for the start of the frame 03b702200000 in oldat and appending the data from there to stfram.

is there any data between the end of one frame and the beginning of the next one

No sir there won't be any data the frame would start with 03b702200000 end with 03b70220001f it will have 640 channels data in each Master frame, After the start of the frame 03b702200000 there is 20 channels of data after which the frame id is incremented 03b702200001and so on till 03b70220001f so i will have 32 frames in one master frame with the total of 640 channels data in it..

From line number 53 - 68
the last part of the code just checking if the data starts with 03b702200000 and if the length is greater then 2944 chars then save every 2944 chars into each row, cause after every 2944 chars a new master frame starts. I thought i will process all the data i.e aligning and capturing data from the start of frame and then save the data into the db.

ChrisW sir tried helping me a lot before with this problem, but i cud't get the final result, all this looks very new to me. I understand what each part of the code does but i think i am going wrong with the flow of the program or something. but i am still programming like how i did for my collage programs.. I need to learn more stuff. I actually tested every part of the code in a different test program and then used it in the main program. I will analyze what you and Chris sir have told me and try getting the result.. I am sorry Chris Sir i have no where else to go,tats why i had to come back with this problem again on the forum. i am working late till one every night no break for Christmas or the new year also working everyday past month.. and still i have no result I am not going in the right way i guess. I will try what wysota sir and you have told me now..

If datrow is zero (where is that set) then you look for "03b702200000" and assume it is found. What happens if it is not found? Does datrow ever get reset? Does the header ever appear again?

Yes sir i set datrow=0; in the constructor of the program, i even did that i tried clearing all the containers and set datrow=0; in the else statement if the start of Frame is not found but when that happened data never showed up again the control stayed in the else itself.


thanks a lot

[wysota]

In line number 23-33 i am actually bit shifting every second byte by 5 times and ORing it with the byte before it and appending the Bit shifted and ORed data into oldat.

Ok but what for? You are only doing it with some of the data. And then you are converting things to text which I something I totally don't understand.

From line number 36 - 48, i am looking for the start of the frame 03b702200000 in oldat and appending the data from there to stfram.

Why?

No sir there won't be any data the frame would start with 03b702200000 end with 03b70220001f it will have 640 channels data in each Master frame, After the start of the frame 03b702200000 there is 20 channels of data after which the frame id is incremented 03b702200001and so on till 03b70220001f so i will have 32 frames in one master frame with the total of 640 channels data in it..

According to me this magic number is a synchronisation sequence. Once you have it, you should never be needing it again unless you lose synchronisation with the data stream.

From line number 53 - 68
the last part of the code just checking if the data starts with 03b702200000 and if the length is greater then 2944 chars then save every 2944 chars into each row,

What if it is not greater than 2944 bytes? What happens then?

ChrisW sir tried helping me a lot before with this problem, but i cud't get the final result, all this looks very new to me. I understand what each part of the code does but i think i am going wrong with the flow of the program or something.

Your main problem is calling this code with a timer. You should be doing that when new data arrives and not periodically.

I need to learn more stuff.

Oh yes, you sure do. And it's high time you do it. It seems the problem is way over your head.


A small suggestion - start by break your design into functions. Make each function do one thing, test it (one function at a time) and then when you know each function behaves correctly, assemble your main method where you will call those functions/methods in sequence to perform the complete task. Furthermore you have completely no error checking, you just assume it will work which is a very bad approach.