I don't have any suggestions to resolve the issue unfortunately, but I can confirm that I experienced the exact same behavior you described on my Mac using Qt 5.7.0. Seems like it should work. Interestingly enough, setMaximumSize seems to be honored, so my best guess is it's a bug. I'd suggest that you report it as such.
Good luck.
Bookmarks