QImage data via FFmpeg

[Degenko]

I'm working on code, that would simply play video frames using ffmpeg in QWidget. All of code is based on this tutorial: http://dranger.com/ffmpeg/tutorial01.c

I'll show only the part where I paint the image (since the rest is the same as in the tutorial I linked):

void MagicCube::PaintFrame(AVFrame *pFrame, int width, int height)
{
    QImage image(width, height, QImage::Format_RGB32);
    int x, y;
    int *src = (int*)pFrame->data[0];
 
    for (y = 0; y < height; y++)
    {
        for (x = 0; x < width; x++)
        {
            image.setPixel(x, y, src[x] & 0x00ffffff);
        }
        src += width;
    }
 
    QPainter painter(this);
    painter.drawImage(QRect(10, 37, width, height), image);
}

void MagicCube::PaintFrame(AVFrame *pFrame, int width, int height)
{
    QImage image(width, height, QImage::Format_RGB32);
    int x, y;
    int *src = (int*)pFrame->data[0];

    for (y = 0; y < height; y++)
    {
        for (x = 0; x < width; x++)
        {
            image.setPixel(x, y, src[x] & 0x00ffffff);
        }
        src += width;
    }

    QPainter painter(this);
    painter.drawImage(QRect(10, 37, width, height), image);
}

To copy to clipboard, switch view to plain text mode 

The current code is working, but it has its problems - it's using 40+% of CPU. So, is there a way to avoid all the loops and just load the data into image (I tried a lot of combinations but with no success) or any other method?

Thanks.

[jpn]

Perhaps you could copy the data directly to QImage::bits()?

[pherthyl]

I don't have a good solution for the data copying part, but I'm pretty sure setPixel will be very slow. I had the same problem with our eye tracker, where the opencv image data was RGB, but not aligned to int, so I couldn't do a straight copy. Here is the code I used, which is not ideal, but reasonably fast.

in the function that updates the image data from the source

if(!currImg) {
			imgWidth = state->image.width;
			imgHeight = state->image.height;
			currImg = new unsigned char[4*imgWidth*imgHeight];
			for(int i = 0; i < imgWidth*imgHeight; i++) {
				currImg[i*4+3] = 0xFF;
			}
		}
 
		for(int i = 0; i < imgWidth*imgHeight; i++) {
			memcpy(currImg+i*4, state->bufferImg.data+i*3, 3);
		}

if(!currImg) {
			imgWidth = state->image.width;
			imgHeight = state->image.height;
			currImg = new unsigned char[4*imgWidth*imgHeight];
			for(int i = 0; i < imgWidth*imgHeight; i++) {
				currImg[i*4+3] = 0xFF;
			}
		}

		for(int i = 0; i < imgWidth*imgHeight; i++) {
			memcpy(currImg+i*4, state->bufferImg.data+i*3, 3);
		}

To copy to clipboard, switch view to plain text mode 

then in the paintevent

if(currImg) {	
		QImage tImg(currImg, imgWidth, imgHeight, QImage::Format_RGB32);
 
		QPainter painter(this);
		painter.drawImage(0, 0, tImg);
	}

if(currImg) {	
		QImage tImg(currImg, imgWidth, imgHeight, QImage::Format_RGB32);

		QPainter painter(this);
		painter.drawImage(0, 0, tImg);
	}

To copy to clipboard, switch view to plain text mode 

I'd love to hear a better solution though

[jpn]

I bumped into this by accident: http://qtnode.net/pastebin/4666

[pherthyl]

Interesting, thanks. I'll have to test to see if this is any faster than my version. I assume it will be because of the lack of function call overhead.

[Degenko]

It's working much faster now, thanks for posting.
Also, the frame coding should be changed from PIX_FMT_RGB32 to PIX_FMT_BGR24.

[pherthyl]

So I'm writing another program that capture video from an opencv camera and needed this again. Well I tried the linked code and it was still seemingly taking quite a long time to copy the frame over. So I benchmarked my original implementation with the for loop compared to that one.

All results were running on my Athlon 64 3200, scaled down to 1Ghz, and are the time to copy one 640x480 3 channel frame from the camera.
Results:
memcpy in a for loop: 14.75ms
code from the pastebin: 13.42ms
new implementation: 11.80ms

So I found a better implementation that's about 12% faster than the one linked above. Here it is:

void RenderWidget::updateData(const IplImage* frame) {
	if(!imageData) {
		imageWidth = frame->width;
		imageHeight = frame->height;
		imageData = new unsigned char[4*imageWidth*imageHeight];
		for(int i = 0; i < imageWidth*imageHeight; i++) {
			imageData[i*4+3] = 0xFF;
		}
	}
 
	int pixels = imageWidth * imageHeight;
	uchar* src = (uchar*)(frame->imageData);
        uchar* srcEnd = src + (3*pixels);
        uchar* dest = imageData;
 
    do {
		memcpy(dest, src, 3);
		dest += 4;
		src += 3;
    } while(src < srcEnd);
}

void RenderWidget::updateData(const IplImage* frame) {
	if(!imageData) {
		imageWidth = frame->width;
		imageHeight = frame->height;
		imageData = new unsigned char[4*imageWidth*imageHeight];
		for(int i = 0; i < imageWidth*imageHeight; i++) {
			imageData[i*4+3] = 0xFF;
		}
	}

	int pixels = imageWidth * imageHeight;
	uchar* src = (uchar*)(frame->imageData);
        uchar* srcEnd = src + (3*pixels);
        uchar* dest = imageData;

    do {
		memcpy(dest, src, 3);
		dest += 4;
		src += 3;
    } while(src < srcEnd);
}

To copy to clipboard, switch view to plain text mode 

So memcpy is faster than a bunch of pointer arithmetic. Hope it's useful to someone.

I'm still not happy with this solution though. Once I copy the data, I have to construct a QImage with it and then convert it to a QPixmap to render it to the screen. I wish there was a way to construct a QPixmap from raw data. The QImage -> QPixmap conversion takes about as long as the data copying.

[seveninches]

So memcpy is faster than a bunch of pointer arithmetic.

This is because memcpy copies the data using pointers to 32-bit data (i.e. 4 bytes a time).

The QImage -> QPixmap conversion takes about as long as the data copying.

This is true for X11, because the QImage is on the client side (in your code), and the QPixmap is on the X server side (in a separate process). It should take less under Windows.

[pherthyl]

This is because memcpy copies the data using pointers to 32-bit data (i.e. 4 bytes a time).

But I'm only copying 3 bytes, so that shouldn't make a difference.

This is true for X11, because the QImage is on the client side (in your code), and the QPixmap is on the X server side (in a separate process). It should take less under Windows.

Yeah, no way around that on X11 I guess. Thanks.

[mikez]

From the latest documentation for setPixel:

Warning: This function is expensive due to the call of the internal detach() function called within; if performance is a concern, we recommend the use of scanLine() to access pixel data directly.

For those still looking here is a routine to convert AVFrame to QImage. QImage must be allocated prior to the call with something like the following:

imageQt = new QImage(w,h,QImage::Format_RGB32);

imageQt = new QImage(w,h,QImage::Format_RGB32);

To copy to clipboard, switch view to plain text mode 

// Assumes AVFrame is in RGB24 format
void AVFrame2QImage(AVFrame *ffFrame, QImage *imageQt, int w, int h)
{
        unsigned char *src = (unsigned char *)ffFrame->data[0];
	for (int y = 0; y < h; y++)
	{
		QRgb *scanLine = (QRgb *)imageQt->scanLine(y);
		for (int x = 0; x < w; x++)
		{
                    scanLine[x] = qRgb(src[3*x], src[3*x+1], src[3*x+2]);
		}
                src += ffFrame->linesize[0];
	}
 
	return;
}

// Assumes AVFrame is in RGB24 format
void AVFrame2QImage(AVFrame *ffFrame, QImage *imageQt, int w, int h)
{
        unsigned char *src = (unsigned char *)ffFrame->data[0];
	for (int y = 0; y < h; y++)
	{
		QRgb *scanLine = (QRgb *)imageQt->scanLine(y);
		for (int x = 0; x < w; x++)
		{
                    scanLine[x] = qRgb(src[3*x], src[3*x+1], src[3*x+2]);
		}
                src += ffFrame->linesize[0];
	}

	return;
}

To copy to clipboard, switch view to plain text mode