For that to work you need to keep track of selection as it happens.
I.e. you need to follow what is added or removed, not just whether something is selected or not.

For that you will have to handle the selection signals appropriately, e.g. using a QListView and connecting to the view's item selection model.
Once you have that it should be easy to put into a dialog that can then serve as your file dialog.

Cheers,
_