One simple approach could be to create a more efficient algorithm for you font decrease. If it is too big use as a new font size the half of the last one. E.g. you start with 100, use 50. If then it is too small increase with the half of your last step, which would be 25. etc. With that you can speed up things a lot.
Also don't create a new QFont each time. Better alter the old object by using setPointSize().