Receiving raw image data through TCP for display using Qt

[benz6699]

Receiving raw image data through TCP for display using Qt
I am working on a GUI to display image from sensor. The GUI acts as a client while the sensor acts as a server. Once I send a command to the sensor, the sensor will send streaming raw image data (binary data) continuously to the GUI through TCP. The data will be an exact copy of the image memory with size width (1240) x height (1028) coded in U8 (unsigned 8 bit). The size of one incoming data is (1240 x 1028 x 1 byte) + 20 byte (meta header) = 1.3MByte.

My biggest obstacles are:
1.) how can I receive and store the streaming data from sensor continuously for display purpose?
2.) how can I display the raw data?

I have tried the code as below, but QImage return with NULL. No image is saved. Please advise.

"client.h"

#ifndef CLIENT_H
#define CLIENT_H

#include <QObject>
#include <QWidget>
#include <QTcpSocket>

class Client : public QWidget
{
Q_OBJECT
public:
explicit Client(QWidget *parent = 0);
int capture (int mode, int NBRLine);

signals:

public slots:

private:
QTcpSocket* socket;

};

#endif // CLIENT_H


"client.cpp"


#include "client.h"
#include <QHostAddress>
#include "mainwindow.h"
#include <QtGui>
#include <QAbstractSocket>
#include <QImage>

Client::Client(QWidget *parent) :
QWidget(parent)
{
socket = new QTcpSocket(this);
}

int Client::capture(int mode, int NBRLine)
{
if (socket->state() != QTcpSocket::ConnectedState)
{
socket->connectToHost("192.168.0.65", 1096);
}

/* send command to retrieve raw image data from sensor */
if(socket->waitForConnected(5000))
{
QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);
out.setVersion(QDataStream::Qt_4_0);
out.setByteOrder(QDataStream::LittleEndian);
out << qint32(0) << qint32(0) << qint32(0) << qint32(1);
out << qint32(9) << qint32(1) << qint32(0) << mode << qint32(10) << qint32(2) << qint32(0) << NBRLine ;
socket->write(block);
socket->flush();
}
else
{
return false;
}
/************************************************** ********/

/* to get data size of each scan through width and height in the meta header */
QDataStream input(socket);
input.setVersion(QDataStream::Qt_4_0);
input.setByteOrder(QDataStream::LittleEndian);
qint32 buffer, cmdID, counter, metaSize, width, height;
do
{
if (socket->waitForReadyRead(1000))
{
input >> cmdID;
if (cmdID == 101)
{
input >> buffer >> buffer >> buffer >> buffer;
}
}
else
{
socket->disconnectFromHost();
break;

}
} while (cmdID != 1);

input >> counter >> metaSize;
if (metaSize != 8) return false;

input >> width >> height;
quint32 datasize = width * height;

/************************************************** ********/

/* Receiving streaming data which I have problem here !!!! */
while (socket->bytesAvailable() < datasize + 80) {
if (!socket->waitForReadyRead(1000)) {
socket->disconnectFromHost();
break;
}
}

QImage img;
input >> img;

if (img.isNull())
{
return 0;
}
img.save("E:/temp1");
return 1;
}

[anda_skoa]

Usually you would want to stay single threaded as long as possible, meaning that you don't use blocking I/O on the socket but connect to its signals.

Also QDataStream is intended for communication between two Qt programs, it might add information to the data that is written into it, e.g. type information. So make sure that the other side is using the same QDataStream setup.
If the sensor is not using Qt at all, don't use QDataStream on the receiver send either.

If the data you receive is raw image data, then one way to create the image is to write into its buffer.
I.e. you create the QImage with the correct dimensions and depth and when image data is received you copy it into the image's data array. See QImage::bits().

Cheers,
_

[benz6699]

First thanks for your quick response.
Any disadvantage for using the method of blocking I/O on the socket?
One reason I am not using the connect signal method is that I have no idea on how to pass the expected value of Cmd_ID to the function e.g. “connect(socket, SIGNAL(readyRead()),this, SLOT(tcpReady()) )”.
The response data I will receive is like below:
<Cmd_ID (4 byte)> <Counter (4 byte)> <Size_of_Meta_Header (4 byte)> <Width (4 byte)>)> <Height (4 byte)> <Data>
First I need to check the value of Cmd_ID whether it is matched with the expected value.Then I need to calculate the size of data by multiplying Width and Height. This is the reason I use QDataStream.
The sensor is not using Qt, what should I use instead?
I am a newbie in Qt. Can you show me an example to write the raw image data into its buffer and copy it into the image's data array? Thanks

[anda_skoa]

Any disadvantage for using the method of blocking I/O on the socket?

The disadvantage is that you need a second thread if you don't want to block the UI.
A multithreaded program is usually much harder to get right.



One reason I am not using the connect signal method is that I have no idea on how to pass the expected value of Cmd_ID to the function e.g. “connect(socket, SIGNAL(readyRead()),this, SLOT(tcpReady()) )”.

The command ID is part of the data, no?
So it can be read from the socket independent of whether you wait for data blockingly or asynchronously.



First I need to check the value of Cmd_ID whether it is matched with the expected value.Then I need to calculate the size of data by multiplying Width and Height. This is the reason I use QDataStream.
The sensor is not using Qt, what should I use instead?

Well, you can use the data stream to interpret multibyte integers for you, but don't expect it to be of any value for any other type of data.



I am a newbie in Qt. Can you show me an example to write the raw image data into its buffer and copy it into the image's data array? Thanks

You would create the image when you know about its size


m_image = QImage(width, height, QImage::Format_Indexed8);

you need to have either a pointer to the current byte or an index that tells you which byte in the image you are at


m_index = 0;

then, when you receive image data, you copy the data into the image


// loop through incoming data
for (int i = 0; i < data.count(); ++i, ++m_index) {
m_image.bits()[m_index] = data[i];
}

That can probably be optimized using memcpy() instead of the loop.

Cheers,
_

[yeye_olive]

The kind of problem you are facing has been extensively covered multiple times on this forum. Searching the forum will give you lots of useful advice.

Reading structured data (such as an image) asynchronously is not entirely trivial because you need to explicitly represent the state of a decoder that may be stopped at any point. By contrast, with synchronous I/O the state is implicitly represented in the context of the decoding thread.

In your case, you will need:

an enum value State representing the piece of data you are currently receiving: command ID, counter, size of meta header, width, height, image data, or meta header;
an integer I representing how many bytes are missing from the piece of data currently received;
variables representing data currently being received or already received (e.g. a quint32 for each of the command ID, counter, size of meta header, width, and height, a QImage, and something for the meta header -- maybe a QByteArray -- should do it).


Initialize State with "currently receiving the command ID", and I with 4 (since the command ID is encoded on 4 bytes). When you receive readyRead(), do the following:

Start a loop.
If there are 0 bytes to read, then exit the loop.
Switch on S.
Read up to I bytes straight to the target buffer (e.g. I bytes from the end of the appropriate quint32, QImage raw data, or QByteArray), and subtract from I the number of bytes actually read.
If I becomes 0, then you finished reading the current piece of data; do some postprocessing (such as endianness conversion, checks that values are withing acceptable bounds, etc.), update S and I and the other variables (for example, if you are then preparing to receive the QImage's raw data, do not forget to set its size). If you just received the last piece of data, notify the user by emitting a signal.
Loop.

[benz6699]

The disadvantage is that you need a second thread if you don't want to block the UI.
A multithreaded program is usually much harder to get right.
Understood.


The command ID is part of the data, no?
So it can be read from the socket independent of whether you wait for data blockingly or asynchronously.
The command ID is part of the receiving data. It tells me which data format will I receive. But it is not part of the image data. The real useful image data comes after <Height (4 bytes)>. The image data is an exact copy of the image memory with size <width> * <height> coded in U8.



Well, you can use the data stream to interpret multibyte integers for you, but don't expect it to be of any value for any other type of data.
After I read the width and height using data stream. How can I read and save the image data so that I can use your method below to build the QImage?



You would create the image when you know about its size


m_image = QImage(width, height, QImage::Format_Indexed8);

you need to have either a pointer to the current byte or an index that tells you which byte in the image you are at


m_index = 0;

then, when you receive image data, you copy the data into the image


// loop through incoming data
for (int i = 0; i < data.count(); ++i, ++m_index) {
m_image.bits()[m_index] = data[i];
}

That can probably be optimized using memcpy() instead of the loop.

Cheers,

[anda_skoa]

After I read the width and height using data stream. How can I read and save the image data so that I can use your method below to build the QImage?

QTcpSocket is a QIODevice subclass, you can use its read methods, e.g. readAll() to read all currently buffered data.

Cheers,
_

[benz6699]

QTcpSocket is a QIODevice subclass, you can use its read methods, e.g. readAll() to read all currently buffered data.

Cheers,
_

Thanks for your advice. I have tried your method below to read and construct the image (gray level image). In the debugging mode I have checked that all the value stored in the variable "buffer" are zero (Refer attachment "BufferAllZero.jpg"). However the image I get is not totally black (It should be all black as I am not providing any light to the sensor). There are always some pixels with different color at the right bottom of the image (Refer attachment "temp1.png"). Can it be the noise? Or the image format I use is not suitable?


pixel = width * height;
char *temp = new char[pixel];
//char *temp = 0;
input.readRawData(temp, pixel);
buffer.append(temp, pixel);


QImage image = QImage(8, 10, QImage::Format_Indexed8);

int m_index = 0;
for (int i = 0; i < 79; ++i, ++m_index)
{
image.bits()[m_index] = buffer[i];

}
bool gray = image.isGrayscale();

if( !image.isNull() )
{
int result = image.save("E:/temp1.png", "PNG",1);
}

[bibhukalyana]

Try to read from socket.



int sizeReceived = 0;
int sizeAvailable = 0;

while(socket->waitForReadyRead(time) && socket->bytesAvailable())
{
sizeAvailable = socket->bytesAvailable();
if(totalSize < sizeReceived + sizeAvailable)
buffer += socket->read(sizeAvailable);
else
{
buffer += socket->read(totalSize - sizeReceived);
break;
}
sizeReceived += sizeAvailable ;
}


(Please ignore if there is some syntax mistake. This is just a dummy code.)

then user buffer to construct QImage.

[anda_skoa]

However the image I get is not totally black (It should be all black as I am not providing any light to the sensor). There are always some pixels with different color at the right bottom of the image (Refer attachment "temp1.png"). Can it be the noise?

Have you checked the return value of readRawData()? Is it the number of bytes you expect?

Cheers,
_

[yeye_olive]

You cannnot expect to read all the image data from the socket in one go. As already said in this thread, you must keep track of how much data you have received. My previous post outlines a solution to your problem that does not assume any single piece of data (including the 32-bit integers) to be read in a single execution of the slot connected to readyRead().

[benz6699]

Have you checked the return value of readRawData()? Is it the number of bytes you expect?

Cheers,
_

Yes, I have checked the return value of readRawData(). It is the number of bytes I expect. For this testing I have set the camera so that it deliver only 8 x 10 pixel image. And all the values in the buffer are zero from buffer[0] to buffer[79]. So I should get a total black image right? However I always get an image with pixels with other colour at the bottom right corner. I cannot identify where is my mistake here. Please advice.

[yeye_olive]

What value does QImage::byteCount() return? Scanlines must be 32-bit aligned, which generally introduces some padding; though there shouldn't be any if your image is 8 pixels wide and 8 bits deep.

By the way, you do not need to go through a temporary buffer. Just read into QImage::bits() directly.

[benz6699]

What value does QImage::byteCount() return? Scanlines must be 32-bit aligned, which generally introduces some padding; though there shouldn't be any if your image is 8 pixels wide and 8 bits deep.

By the way, you do not need to go through a temporary buffer. Just read into QImage::bits() directly.

QImage::byteCount() return the value I expect which is 80 (I "ask" the sensor to send me 8 x 10 x1 byte image). So what else can I check more?

Can you show me the code of how to read data directly into QImage::bits()? Thanks.

[yeye_olive]

QImage::byteCount() return the value I expect which is 80 (I "ask" the sensor to send me 8 x 10 x1 byte image). So what else can I check more?
Could you post a minimal complete example reproducing the problem?

Can you show me the code of how to read data directly into QImage::bits()? Thanks.
Sure, something like: result = my_qiodevice_ptr->read(my_qimage.bits() + number_of_bytes_already_read, my_qimage.byteCount() - number_of_bytes_already_read);

[benz6699]

Could you post a minimal complete example reproducing the problem?


#include "client.h"
#include <QHostAddress>
#include "mainwindow.h"
#include <QtGui>
#include <QAbstractSocket>
#include <QImage>

Client::Client(QWidget *parent) :
QWidget(parent)
{
socket = new QTcpSocket(this);
connect(socket, SIGNAL(readyRead()),this, SLOT(tcpReady()) );
}

int Client::capture(int mode, int NBRLine)
{
if (socket->state() != QTcpSocket::ConnectedState)
{
socket->connectToHost("192.168.0.65", 1096);
}
/* send command to retrieve raw image data from sensor */
if(socket->waitForConnected(5000))
{
QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);
out.setVersion(QDataStream::Qt_4_0);
out.setByteOrder(QDataStream::LittleEndian);
//out << qint32(0) << qint32(0) << qint32(0) << qint32(1);
out << qint32(9) << qint32(1) << qint32(0) << mode << qint32(10) << qint32(2) << qint32(0) << NBRLine ;
socket->write(block);
socket->flush();
}
else
{
return false;
}
}


int Client::tcpReady()
{
QDataStream input(socket);
input.setVersion(QDataStream::Qt_4_0);
input.setByteOrder(QDataStream::LittleEndian);
qint32 cmdID, counter, metaSize, CmdNbr, min, max, value, error, width, height;
QByteArray buffer;

int sizereceived = 0;
int sizeAvailable = 0;

forever{
if (socket->bytesAvailable() < sizeof(qint32))
{
return 0;
}
input >> cmdID;
// We get the data response with cmdID = 100 when we send a command to get parameter value
if (cmdID == 100)
{
input >> counter >> metaSize >> CmdNbr >> min >> max >> value;
}
// We get the data response with cmdID = 101 when we send a command to set parameter value
else if (cmdID == 101)
{
input >> counter >> metaSize >> CmdNbr >> error;
}
// Here we get a streaming raw image data
else if (cmdID == 1)
{
break;
}
else
{
return 0;
}
}

// Here we read the data and construct image
input >> counter >> metaSize >> width >> height;
int pixel = width * height;
char *temp = new char[pixel];
//char *temp = 0;
int read = input.readRawData(temp, pixel);
buffer.append(temp, pixel);
QImage image = QImage(width, height, QImage::Format_Indexed8);
delete [] temp;

int m_index = 0;
for (int i = 0; i < (pixel-1); ++i, ++m_index)
{
image.bits()[m_index] = buffer[i];
}
int cnt = image.byteCount();
bool gray = image.isGrayscale();

if( !image.isNull() )
{
int result = image.save("E:/temp1.png", "PNG",1);
}
return 1;
}



Sure, something like: result = my_qiodevice_ptr->read(my_qimage.bits() + number_of_bytes_already_read, my_qimage.byteCount() - number_of_bytes_already_read);

I tried the following code, but it give me the error message" :-1: error: C2664: 'qint64 QIODevice::read(char *,qint64)' : cannot convert parameter 1 from 'uchar *' to 'char *' Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast."


input >> counter >> metaSize >> width >> height;
int pixel = width * height;

QImage image = QImage(width, height, QImage::Format_Indexed8);
socket->read(image.bits(), pixel);

[benz6699]

Try to read from socket.



int sizeReceived = 0;
int sizeAvailable = 0;

while(socket->waitForReadyRead(time) && socket->bytesAvailable())
{
sizeAvailable = socket->bytesAvailable();
if(totalSize < sizeReceived + sizeAvailable)
buffer += socket->read(sizeAvailable);
else
{
buffer += socket->read(totalSize - sizeReceived);
break;
}
sizeReceived += sizeAvailable ;
}


(Please ignore if there is some syntax mistake. This is just a dummy code.)

then user buffer to construct QImage.


I have tried your method to read the big file. It is working to combine the small packet to become one. However the image I get is different from the original image (refer attachment ori.jpg with only one line and temp1.png with 4 line). Any idea?



input >> counter >> metaSize >> width >> height;
int totalSize = width * height;
int sizeReceived = 0;
int sizeAvailable = 0;
char *temp = new char[totalSize];

while(socket->waitForReadyRead(1000) && socket->bytesAvailable())
{
sizeAvailable = socket->bytesAvailable();
if (totalSize > (sizeReceived + sizeAvailable))
{
input.readRawData(temp, sizeAvailable);
buffer.append(temp, sizeAvailable);
}
else
{
input.readRawData(temp, (totalSize - sizeReceived));
buffer.append(temp, (totalSize - sizeReceived));
break;
}
sizeReceived += sizeAvailable;
}

QImage image = QImage(width, height, QImage::Format_Indexed8);
delete [] temp;

int m_index = 0;
for (int i = 0; i < (totalSize-1); ++i, ++m_index)
{
image.bits()[m_index] = buffer[i];
}
int cnt = image.byteCount();
bool gray = image.isGrayscale();

if( !image.isNull() )
{
int result = image.save("E:/temp1.png", "PNG",1);
}





1050310504

[yeye_olive]

I dug this up a bit, and here are my findings.

It seems that QImage::save() produces garbage unless you set the color map. The following example demonstrates this:


#include <QImage>

int main(int argc, char *argv[])
{
QImage image(8, 10, QImage::Format_Indexed8);
image.setColorTable(QVector<QRgb>() << 0xff0000ff); // This line is crucial
image.fill(0);
image.save("foo.png");
return 0;
}

This produces a blue image as expected, and garbage if you omit line 6. Do not forget to initialize the color table, then. In your case it may mean streaming it along with the image data, depending on whether or not the color table is always the same.

Secondly, avoid going through all these useless intermediate buffers (temp and buffer). If you are going to use QDataStream::readRawData(), just do my_qdatastream.readRawData(reinterpret_cast<char *>(my_qimage.bits()), my_qimage.byteCount()), provided you are certain you have enough bytes in the stream, of course.

Finally, your current approach to reading from the socket in one go is completely broken. As I have already warned you twice about it, I will stop here. Just do not be surprised if this stops working as soon as you test your program over a real network.

[benz6699]

This produces a blue image as expected, and garbage if you omit line 6. Do not forget to initialize the color table, then. In your case it may mean streaming it along with the image data, depending on whether or not the color table is always the same.

Where should I put the following code:


image.setColorTable(QVector<QRgb>() << 0xff0000ff); // This line is crucial
image.fill(0);?

before or after input.readRawData(reinterpret_cast<char *>(image.bits()), sizeAvailable);?

My code is as the following:


int Client::tcpReady()
{
QDataStream input(socket);
input.setVersion(QDataStream::Qt_4_0);
input.setByteOrder(QDataStream::LittleEndian);
qint32 cmdID, counter, metaSize, CmdNbr, min, max, value, error, width, height;
QByteArray buffer;

forever{
if (socket->bytesAvailable() < sizeof(qint32))
{
return 0;
}
input >> cmdID;
// We get the data response with cmdID = 100 when we send a command to get parameter value
if (cmdID == 100)
{
input >> counter >> metaSize >> CmdNbr >> min >> max >> value;
}
// We get the data response with cmdID = 101 when we send a command to set parameter value
else if (cmdID == 101)
{
input >> counter >> metaSize >> CmdNbr >> error;
}
// Here we get a streaming raw image data
else if (cmdID == 1)
{
break;
}
else
{
return 0;
}
}
input >> counter >> metaSize >> width >> height;
int totalSize = width * height;
int sizeReceived = 0;
int sizeAvailable = 0;


QImage image = QImage(width, height, QImage::Format_Indexed8);
// image.setColorTable(QVector<QRgb>() << 0xff0000ff); // This line is crucial
// image.fill(0);

while(socket->waitForReadyRead(1000) && socket->bytesAvailable())
{
sizeAvailable = socket->bytesAvailable();
if (totalSize > (sizeReceived + sizeAvailable))
{
input.readRawData(reinterpret_cast<char *>(image.bits()), sizeAvailable);
}
else
{
input.readRawData(reinterpret_cast<char *>(image.bits()), (totalSize - sizeReceived));
break;
}
sizeReceived += sizeAvailable;
}


if( !image.isNull() )
{
int result = image.save("E:/temp1.png", "PNG",1);
}
return 1;
}


Secondly, avoid going through all these useless intermediate buffers (temp and buffer). If you are going to use QDataStream::readRawData(), just do my_qdatastream.readRawData(reinterpret_cast<char *>(my_qimage.bits()), my_qimage.byteCount()), provided you are certain you have enough bytes in the stream, of course.

I have used this method in my code. But it give me a blank image. Did I apply your method incorrectly?


Finally, your current approach to reading from the socket in one go is completely broken. As I have already warned you twice about it, I will stop here. Just do not be surprised if this stops working as soon as you test your program over a real network.

Thanks for your precious advice. I always keep that in my mind. I just want to deal the image issue before this one. I handle this issue in the latest code, but I found out that if the sensor send me the image non-stop, my software will be hanged. I cannot do anything on my software ( even I tried to move the position of my software). Any mistake I made?