I wonder if u really searched
Even with the same heading as your thread - "Custom shaped button" I could find this.

Also if u had seen shaped clock example in Qt Demo, it will give you a good idea.