I have been suggested to use Memory Mapping instead
Unless you have 100 GB of RAM, memory mapping probably is a waste of time if you are streaming through the video. The underlying operating system and disk controller will take care of buffering incoming data. Memory mapping is only useful when you have a very large chunk of RAM available and you can put a significant part of the file into it. With streaming, you'd be swapping out of the memory map as fast as you loaded into it.

What kind of canvas should I display the video on?
QLabel is typically used to display images.

How do I ensure, that I can see the video, while processing it in the background?
To ensure your GUI stays responsive, you could set the priority of the background thread lower than normal, unless you mean that the background thread is responsible for reading the data from disk and converting it to images to be displayed. If you mean that the processing has to keep up with normal video display rates, I'm not sure what you can do here.

Why can't you use Phonon? It seems that nearly all the functionality needed is in there; you will need to roll your own back end for serving video frames, but you'll be doing that anyway in your own implementation.