I doubt it has anything to do with the property animation, but you can verify that easily by resizing without animation.
Yes, behavior is the same ( wrong) for simple resize without animation, but only for dialog window, for main window it works properly.
But I created similar project in gtk+ and it works properly for main and also dialog window so this can be a bug inside Qt? Should I report this to Qt?

Thanks,