setMask (smooth)

How i can set the mask look like the right picture (frmae is smooth)?
http://file.oboz.ua/files/vf46476dce...0751322586.jpg

My result in the left.

The mask is a bitmap (opaque/transparent) so if you need to include antialiasing, the mask should return opaque (1) for parts that are to be antialiased.

It depends on what you want to do. If you want to display a QPixmap in your application you should use a transparent png or use setAlphaCannel() instead of setMask() and the background will be propagated.
On the other hand if you want to set the shape of a window(like the shaped clock example) you're out of luck. There is no easy platform independent way to do this. See this thread.