If all communication with the port is done asynchronously then you only need one thread for everything. You don't need to loop over ports all the time to query them for data, both serial port implementations for Qt should provide the readyRead() signal for the serial device so you will get notified when there is anything to read.
Bookmarks