Problem with playing color-keyed AxWidget (Shockwave Flash)

[indiocolifa]

I'm trying to play a transparent flash movie using layered windows and Qt. I'm not going for per pixel alpha and windowless OLE controls, I want to make the background color (black) to become transparent using the UpdateLayeredWindow call. The problem is that I'm getting a black background (no color-key) on Windows XP altough it works on Vista and 7 (related to DWM composition, surely).

Well, the window to be transparent is QWidget subclass. In the constructor I setup the (I think) the proper flags and the layered window style :

setWindowFlags(Qt::FramelessWindowHint | Qt::WindowStaysOnTopHint);
LONG_PTR exstyle = ::GetWindowLongPtr(winId(), GWL_EXSTYLE);
::SetWindowLongPtr(winId(), GWL_EXSTYLE, exstyle | WS_EX_LAYERED | WS_EX_TOPMOST );
 
 _axWidget = new FlashAxWidget(this, winId());

setWindowFlags(Qt::FramelessWindowHint | Qt::WindowStaysOnTopHint);
LONG_PTR exstyle = ::GetWindowLongPtr(winId(), GWL_EXSTYLE);
::SetWindowLongPtr(winId(), GWL_EXSTYLE, exstyle | WS_EX_LAYERED | WS_EX_TOPMOST );

 _axWidget = new FlashAxWidget(this, winId());

To copy to clipboard, switch view to plain text mode 

FlashAxWidget is a QAxWidget subclass where I simply put a window subclass,etc (This does not change anything since I've tried without subclassing and SetLayeredWindowAttributes and fails also).

Note that as I've told, I update the layered window manually on every flash frame. So I must subclass the AX control to obtain the WM_PAINT message:

case WM_PAINT:
        PAINTSTRUCT ps;
        HDC hdc = BeginPaint(hwnd, &ps);
        g_pAxWInstance->DispatchPaint(hwnd, hdc);
        EndPaint(hwnd, &ps);
        return 0L;
        break;

case WM_PAINT:
        PAINTSTRUCT ps;
        HDC hdc = BeginPaint(hwnd, &ps);
        g_pAxWInstance->DispatchPaint(hwnd, hdc);
        EndPaint(hwnd, &ps);
        return 0L;
        break;

To copy to clipboard, switch view to plain text mode 

This is working well, I catch the paint messages properly.

Now, the painting code where the main window layering is updated:

void FlashAxWidget::DispatchPaint(HWND hwnd, HDC hdc)
{
    RECT r;
    ::GetClientRect(hwnd, &r);
 
    POINT p =  {r.left, r.top};
    POINT p0 = {0,0};
    SIZE sz = {r.right - r.left, r.bottom - r.top};
 
    HDC hdcScr = ::GetDC(0);
    HDC _dcBackBuf = ::CreateCompatibleDC(hdcScr);
 
    BYTE * pBits;
 
    BITMAPINFOHEADER bmi = {0};
    bmi.biSize = sizeof(BITMAPINFOHEADER);
    bmi.biBitCount = 32;
    bmi.biCompression = BI_RGB;
    bmi.biPlanes = 1;
    bmi.biWidth = sz.cx;
    bmi.biHeight = -(sz.cy);
 
    HBITMAP _hbmpBackBuf = ::CreateDIBSection(hdc, (BITMAPINFO*)&bmi, DIB_RGB_COLORS,
                                              (void**)&pBits, 0, 0);
 
    ::SelectObject(_dcBackBuf, _hbmpBackBuf);
 
    unsigned long cDIBits = sz.cx*sz.cy*4;
 
    ::RtlZeroMemory(pBits, cDIBits);
    ::OleDraw(_pIVO, DVASPECT_TRANSPARENT, _dcBackBuf, &r);
 
     ::UpdateLayeredWindow(parentWidget()->winId(),  NULL, &p,
                          &sz, _dcBackBuf, &p0, 0, NULL, ULW_COLORKEY);
    ::DeleteDC(_dcBackBuf);
    ::DeleteObject(_hbmpBackBuf);
    ::ReleaseDC(GetDesktopWindow(), hdcScr);
}

void FlashAxWidget::DispatchPaint(HWND hwnd, HDC hdc)
{
    RECT r;
    ::GetClientRect(hwnd, &r);

    POINT p =  {r.left, r.top};
    POINT p0 = {0,0};
    SIZE sz = {r.right - r.left, r.bottom - r.top};

    HDC hdcScr = ::GetDC(0);
    HDC _dcBackBuf = ::CreateCompatibleDC(hdcScr);

    BYTE * pBits;

    BITMAPINFOHEADER bmi = {0};
    bmi.biSize = sizeof(BITMAPINFOHEADER);
    bmi.biBitCount = 32;
    bmi.biCompression = BI_RGB;
    bmi.biPlanes = 1;
    bmi.biWidth = sz.cx;
    bmi.biHeight = -(sz.cy);

    HBITMAP _hbmpBackBuf = ::CreateDIBSection(hdc, (BITMAPINFO*)&bmi, DIB_RGB_COLORS,
                                              (void**)&pBits, 0, 0);

    ::SelectObject(_dcBackBuf, _hbmpBackBuf);

    unsigned long cDIBits = sz.cx*sz.cy*4;

    ::RtlZeroMemory(pBits, cDIBits);
    ::OleDraw(_pIVO, DVASPECT_TRANSPARENT, _dcBackBuf, &r);

     ::UpdateLayeredWindow(parentWidget()->winId(),  NULL, &p,
                          &sz, _dcBackBuf, &p0, 0, NULL, ULW_COLORKEY);
    ::DeleteDC(_dcBackBuf);
    ::DeleteObject(_hbmpBackBuf);
    ::ReleaseDC(GetDesktopWindow(), hdcScr);
}

To copy to clipboard, switch view to plain text mode 

The rationale is that I'm creating a 32-bit DIB section, I blit the flash contents with OleDraw (this works since I'm tried putting it with BitBlt to screen DC) and make the black background transparent with the ::UpdateLayeredWindow call.

Well, it does not work.

I'm getting strange results. For example, if I remove the window flags (Qt::FramelessWindowHint,etc) I can resize the window. When I resize the window the flash movie stops or gets clipped in incorrect positions.

Anyone with experience on layered windows and Qt? Consider this is my first serious Qt project.

Thank you very much,

[indiocolifa]

I've fixed some code in the DispatchPaint code above but the background it's still black and no UpdateLayeredWindow effect.

I'm thinking that i'm getting some painting order problem in Qt, e.g: UpdateLayered --> and immediately I get the AX control painted with the black background (altough i'm subclassing WM_PAINT and not calling the original procedure from the AX Control for painting).

[indiocolifa]

I've discovered that it's actually working when I set the display mode to 16-bit, but not in 32-bit, at least on my Intel 965-based notebook.

I'm totally confused now....

[indiocolifa]

Solved: Seems that this is not documented , but for ULW_COLORKEY to work in some configurations (WinXP+Intel Express chipsets) I must supply a 24-bit DIB, not a 32-bit one.