The magic is likely some element(s) of the style's rendering of the widgets that you are not accounting for. You may be able to work the amount of "magic" required using the QStyle::pixelMetric() function and constants
http://doc.qt.io/qt-4.8/qstyle.html#PixelMetric-enum

The flickering is probably because the automatic scrollbar is being added and removed repeatedly because the size of the contained widget is changing around the scroll area's critical size. You need to be careful that the appearance of the scroll bar, which narrows the viewport, does not trigger a shrinking of the layout that causes the viewport to grow... Circular updates.