how to check data package received in serial port is 5 byte and check each byte ?

[Lesiok]

First of all : in line 47 (serial->readLine()...) You can get any number of bytes in the range of <1..1000>.
Before atof() you need to check whether the buffer contains a complete package.