QImage as average of 10 sample video frame

[astodolski]

What would be the fastest way to get an average of 10 QImage samples from my video device? I want to avoid doing things pixel by pixel as that would be a lot slower.

My application buffers 10 QImages which can then be used as QByteArray. The images are without detail. That is, they are imaging a black subject and I want to see which pixels have higher noise or even what is known as "dead pixel". The resulting image for analysis is an average of 10 samples. QByteArray has the append method but that only grows the array size. The array size should be always fixed at 921,600 (1280x720).

[wysota]

I would assume most CPUs have a SIMD instruction (e.g. PADDW for MMX) which allows to add two arrays together efficiently. From a quick google search it seems that if you write code properly and enable mmx support in the compilation, the compiler should be able to optimize your loops using SIMD instructions.

You can start from here: http://stackoverflow.com/questions/586609/using-sse-instructions

[astodolski]

I would assume most CPUs have a SIMD instruction (e.g. PADDW for MMX) which allows to add two arrays together efficiently. From a quick google search it seems that if you write code properly and enable mmx support in the compilation, the compiler should be able to optimize your loops using SIMD instructions.

You can start from here: http://stackoverflow.com/questions/586609/using-sse-instructions

Thanks for that. It looks like then it's using SSE, MMX or that along with STL. I thought I could do it within Qt.

[ChrisW67]

Would this approach work?


Start with a new Qimage of the same size and start a QPainter on it.
Fill it in black
Set the painter opacity to 0.1
Call drawImage() once for each image


A black frame will not show a totally dead pixel, but a white frame might.

[astodolski]

Would this approach work?


Start with a new Qimage of the same size and start a QPainter on it.
Fill it in black
Set the painter opacity to 0.1
Call drawImage() once for each image



A black frame will not show a totally dead pixel, but a white frame might.

Thanks for the idea. However, I need to sample frames of real video. The sensor is made totally opaque. If you snap an image and blow it up you can discern one or more pixels that look distorted than the rest of the blank image (MJPEG promotes that). I want to be able to sample one frame which is an average of 10. Wysota suggested something that I wasn't aware of and will be worth investigating. I thought you could do it though with the Qt libs.

[wysota]

I thought I could do it within Qt.

You can do it with QtConcurrent but I'm guessing it'd be slower than using your CPU directly. Of course you can probably (depending on how SSE/MMX instructions work, I have never used them) take a hybrid approach -- divide the whole image into chunks and process each chunk in a separate thread using SIMD instructions. Note that the performance might vary significantly depending on the size of the image and size of data cache in the cpu. If you're unlucky enough threads will compete for the data cache effectively degrading performance.

Edit: Also if your GPU supports it (OpenGL >= 4.3) and you have fresh enough Qt, you can probably use a compute shader to do what you want on the GPU. However I don't know how accurate results, you'd get.

[astodolski]

You can do it with QtConcurrent but I'm guessing it'd be slower than using your CPU directly. Of course you can probably (depending on how SSE/MMX instructions work, I have never used them) take a hybrid approach -- divide the whole image into chunks and process each chunk in a separate thread using SIMD instructions. Note that the performance might vary significantly depending on the size of the image and size of data cache in the cpu. If you're unlucky enough threads will compete for the data cache effectively degrading performance.

Edit: Also if your GPU supports it (OpenGL >= 4.3) and you have fresh enough Qt, you can probably use a compute shader to do what you want on the GPU. However I don't know how accurate results, you'd get.

A shader wont work for me because I am imaging an opaque surface. Due to MJPEG, a dead pixel shows up differently among the rest. It will have a higher value than normal pixels. Perhaps averaging using the STL such as:

v-Avg = v0 + v1 + v2 + v3...v10 * 0.1F;

could be possible. I'm trying that for now.

[wysota]

A shader wont work for me because I am imaging an opaque surface.
I don't see how that matters. Mathematics doesn't care what the numbers represent, they are just values.


Perhaps averaging using the STL such as:

v-Avg = v0 + v1 + v2 + v3...v10 * 0.1F;

could be possible. I'm trying that for now.

You can put that in the compute shader kernel, pass it the 10 images and have the GPU parallelize computation among all GPU cores. It will definitely be much faster than doing the same on a multi-core CPU.

[astodolski]

I don't see how that matters. Mathematics doesn't care what the numbers represent, they are just values.



You can put that in the compute shader kernel, pass it the 10 images and have the GPU parallelize computation among all GPU cores. It will definitely be much faster than doing the same on a multi-core CPU.

Very handy though I don't know how much of this is useful within Qt. It sounds more like a job using Cuda (http://www.nvidia.com/object/cuda_home_new.html). That's on me though but thanks for the insight towards using a shader kernel - I wasn't aware of it till now.

[wysota]

Very handy though I don't know how much of this is useful within Qt.

Hmm... QOpenGLShader is a Qt class with a couple of friends available.

[astodolski]

Hmm... QOpenGLShader is a Qt class with a couple of friends available.

Yes it is. Full disclosure, I am not using OpenGL.

[wysota]

Yes it is. Full disclosure, I am not using OpenGL.

Then obviously you have to implement everything in a much slower CPU :) Which in turn will not get you where you wanted to be in your first post:


What would be the fastest way (...)

The fastest way would be to use the GPU. You don't even need compute shaders, regular GL/ES pipeline will do as well -- you can use a fragment shader that will average the 10 images for you, place it in a frame buffer object and make a QImage from it.

So consider adding QT+=opengl to your Qt4 project and use the GPU in this case.