Custom QSortFilterProxyModel for filtering large data

[ednakram]

I have a model sub-classed from QAbstractTableModel, which shows around 1.5 million rows. There is a QSortFilterProxyModel used to filter over this original model. Two problems are that filtering is real slow and it hangs the GUI.
I am using Qt4.6 and I did explore Qt5 but I doesnt seem to have significant improvements to the performance of QSortFilterProxyModel.

I have thought of an approach (still theoretical) - with two goals
A) to give the user whatever results are obtained as they are available (i.e. row accepted by filterAcceptsRow)
B) to not let the GUI freeze and have user abort the operation to see partial results of filtering only.

To achieve this the plan is (theoretical still) is as follows:
1) Split the original model into smaller models e.g. 100 models of 15K rows each (have to think of way to do this without duplicating the entire data stored in original model).
2) Perform the filtering on first model - results should be back within 1 second.
3) Show the obtained results in a view (using a temp model that can be appended to?)
4) If a "Stop" button has been pressed stop here - indicating this is partial result of filtering only (my users will be Ok with that).
5) Schedule the step 2 using a zero timeout QTimer (i.e. process will start after events in QT event loop are serviced) for the "next" model.

Would like to hear about advice regarding this approach, will it really solve the two goals in mind?
What details should I care about to avoid too much book-keeping costs in terms of memory and run-time?

[anda_skoa]

My preference for advanced sorting and/or filtering is to do it in the model itself.

A proxy model is nicely generic but that also implies it can use knowledge about the data and the structures inside the main model.

The main model can easily apply filtering iteratively, progressively, etc.

Cheers,
_