Results 1 to 8 of 8

Thread: Custom QSortFilterProxyModel for filtering large data

  1. #1
    Join Date
    Dec 2016
    Posts
    16
    Thanks
    2
    Thanked 1 Time in 1 Post
    Qt products
    Qt4

    Default Custom QSortFilterProxyModel for filtering large data

    I have a model sub-classed from QAbstractTableModel, which shows around 1.5 million rows. There is a QSortFilterProxyModel used to filter over this original model. Two problems are that filtering is real slow and it hangs the GUI.
    I am using Qt4.6 and I did explore Qt5 but I doesnt seem to have significant improvements to the performance of QSortFilterProxyModel.

    I have thought of an approach (still theoretical) - with two goals
    A) to give the user whatever results are obtained as they are available (i.e. row accepted by filterAcceptsRow)
    B) to not let the GUI freeze and have user abort the operation to see partial results of filtering only.

    To achieve this the plan is (theoretical still) is as follows:
    1) Split the original model into smaller models e.g. 100 models of 15K rows each (have to think of way to do this without duplicating the entire data stored in original model).
    2) Perform the filtering on first model - results should be back within 1 second.
    3) Show the obtained results in a view (using a temp model that can be appended to?)
    4) If a "Stop" button has been pressed stop here - indicating this is partial result of filtering only (my users will be Ok with that).
    5) Schedule the step 2 using a zero timeout QTimer (i.e. process will start after events in QT event loop are serviced) for the "next" model.

    Would like to hear about advice regarding this approach, will it really solve the two goals in mind?
    What details should I care about to avoid too much book-keeping costs in terms of memory and run-time?

  2. #2
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Custom QSortFilterProxyModel for filtering large data

    My preference for advanced sorting and/or filtering is to do it in the model itself.

    A proxy model is nicely generic but that also implies it can use knowledge about the data and the structures inside the main model.

    The main model can easily apply filtering iteratively, progressively, etc.

    Cheers,
    _

  3. #3
    Join Date
    Dec 2016
    Posts
    16
    Thanks
    2
    Thanked 1 Time in 1 Post
    Qt products
    Qt4

    Default Re: Custom QSortFilterProxyModel for filtering large data

    thanks anda_skoa.

    I think you missed adding a "NOT", right?

    A proxy model is nicely generic but that also implies it can NOT use knowledge about the data and the structures inside the main model.

    Do note that I have multiple views using the same underlying model - some of them may have the sorting/filtering applied and some may not.
    Implementing this all in the "main model" means that at a time only one sort/filter criteria can be applied at a time, right?

    What would you suggest if goal is reduced - "to not let the GUI freeze" and give some indication that "filtering" is going on?
    Letting the user interrupt the filtering process would be a bonus.
    Last edited by ednakram; 4th January 2017 at 22:26.

  4. #4
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Custom QSortFilterProxyModel for filtering large data

    Quote Originally Posted by ednakram View Post
    I think you missed adding a "NOT", right?
    Right, of course, sorry.

    Quote Originally Posted by ednakram View Post
    Do note that I have multiple views using the same underlying model - some of them may have the sorting/filtering applied and some may not.
    Implementing this all in the "main model" means that at a time only one sort/filter criteria can be applied at a time, right?
    Yes, but can't you simply use multiple instances of the same model?
    That's what I usually do, haven't had a use case yet where I needed to be able to have the same QModelIndex across views.

    Quote Originally Posted by ednakram View Post
    What would you suggest if goal is reduced - "to not let the GUI freeze" and give some indication that "filtering" is going on?
    Letting the user interrupt the filtering process would be a bonus.
    The model would need a slot that is called by a timer or QMetaObject::invokeMethod() to make each invocation return to the event loop.
    The slot would either based on number of items processed or time spent decide when to return, leaving the rest of the work to further invocations.

    As the slot processes data against the filter criteria it would do beginRemoveRows()/endRemoveRows() on those that no longer match.

    Cheers,
    _

  5. #5
    Join Date
    Dec 2016
    Posts
    16
    Thanks
    2
    Thanked 1 Time in 1 Post
    Qt products
    Qt4

    Default Re: Custom QSortFilterProxyModel for filtering large data

    Thanks for the reply anda_skoa, sorry I lost this topic for a while.

    Quote Originally Posted by anda_skoa View Post
    Yes, but can't you simply use multiple instances of the same model?
    That's what I usually do, haven't had a use case yet where I needed to be able to have the same QModelIndex across views.
    _
    Having multiple instances of the "main model" will mean replicating the data it is storing. But perhaps that can be worked around by having the data, in this case a QList<Foo> to be created globally and passed to all these model instances.


    Quote Originally Posted by anda_skoa View Post
    The model would need a slot that is called by a timer or QMetaObject::invokeMethod() to make each invocation return to the event loop.
    The slot would either based on number of items processed or time spent decide when to return, leaving the rest of the work to further invocations.
    As the slot processes data against the filter criteria it would do beginRemoveRows()/endRemoveRows() on those that no longer match.
    _
    By model here you mean a model derived directly from QAbstractItemModel (or QAbstractTableModel) right?
    Also with this suggestion the user would see the GUI as responsive, but will slowly see the "un-matched" rows disappearing. So if i "want to stop" the process in the middle what I see in the "view" will be a mix of "matched" and "un-matched" rows, which can be very mis-leading - specifically it is worse than an approach wherein - when filtering starts, everything is blank and then you slowly start seeing matches, if you stop in the middle you still see the matches, only problem is that they are "incomplete".


    Keeping all three goals in mind i.e. keeping GUI alive, allowing stopping in middle and seeing results incrementally, I think a hybrid approach has to be applied.
    Here are the details, please suggest/correct.

    1. QList<Foo> fooList; is the actual list of objects displayed in all of the tables, move this to global scope and remove from model.
    2. FooItemModel is the model derived from QAbstractTableModel that uses the fooList.
    3. A new slot is added to FooItemModel i.e. int FooItemModel::startFilter(QRegExp searchExp, int index), which accept an index where to start the search and return index till where it made the search. Indexes refer to places in fooList.
    4. When user enters a filter request int FooItemModel::startFilter(searchExp, 0) is called to begin search from 0th index of fooList. At the start this function calls beginModelReset() and indicate that model has no entries.
    5. The startFilter function goes through till 2000 entries in fooList and adds results i.e. indexes in fooList that match - to another list QList<int> fooListFilteredIndexes; Let u say out of 2000 - 200 items have matches for fooListFilteredIndexes has 200 entries.
    6. The startFilter function calls endModelReset and control is returned to event loop, and return index is 2000 (i.e. index till which search was done).
    7. When the view queries for "rowCount()" it will get 200 i.e. size of fooListFilteredIndexes and when it requests data for these 200 items it will be retrieved by first getting the indexes from fooListFilteredIndexes and then using that index to get actual data from fooList.
    8. If a "Stop" button was pressed - the process stops and user is told that filtering results are "partial", if not a 0 interval timer's timeout function will fire the slot startFilter again but with new arguments i.e. FooItemModel::startFilter(searchExp, 2000) so that search now starts from 2000 onwards. And these steps i.e. 5-8 are repeated till the complete fooList is covered.

    Do note I have not considered sorting at all here. I think approach for that should be that when incremental filtering is going on sorting can be "disabled" ! Once the partial or complete results are available then sorting can be done only on the "results" and not the complete fooList.

    What do you think about this approach?

  6. #6
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Custom QSortFilterProxyModel for filtering large data

    Quote Originally Posted by ednakram View Post
    Having multiple instances of the "main model" will mean replicating the data it is storing.
    Why?

    Quote Originally Posted by ednakram View Post
    But perhaps that can be worked around by having the data, in this case a QList<Foo> to be created globally and passed to all these model instances.
    Exactly.
    A model is usually not the place where the data is stored, it is the interface to the data.

    Quote Originally Posted by ednakram View Post
    By model here you mean a model derived directly from QAbstractItemModel (or QAbstractTableModel) right?
    Yes.

    Quote Originally Posted by ednakram View Post
    Also with this suggestion the user would see the GUI as responsive, but will slowly see the "un-matched" rows disappearing. So if i "want to stop" the process in the middle what I see in the "view" will be a mix of "matched" and "un-matched" rows, which can be very mis-leading - specifically it is worse than an approach wherein - when filtering starts, everything is blank and then you slowly start seeing matches, if you stop in the middle you still see the matches, only problem is that they are "incomplete".
    There are multiple options of course, which ever behavior you need is the one to implement.
    If you incrementally remove rows from the full table or incrementally add rows to an empty table is really up to you.

    Quote Originally Posted by ednakram View Post
    1. QList<Foo> fooList; is the actual list of objects displayed in all of the tables, move this to global scope and remove from model.
    Well, if you like global, but you could also pass in as reference or pointer.
    You probably will want to wrap that in a class anyway, so that you have an object that can notify about data modifications.

    Quote Originally Posted by ednakram View Post
    4. When user enters a filter request int FooItemModel::startFilter(searchExp, 0) is called to begin search from 0th index of fooList. At the start this function calls beginModelReset() and indicate that model has no entries.
    Alternatively reset only at the first time and then add rows later on.
    Depends on whether you want the list to be replaced every time or grow.

    Quote Originally Posted by ednakram View Post
    8. If a "Stop" button was pressed - the process stops and user is told that filtering results are "partial", if not a 0 interval timer's timeout function will fire the slot startFilter again but with new arguments i.e. FooItemModel::startFilter(searchExp, 2000) so that search now starts from 2000 onwards. And these steps i.e. 5-8 are repeated till the complete fooList is covered.
    Right

    Quote Originally Posted by ednakram View Post
    Do note I have not considered sorting at all here. I think approach for that should be that when incremental filtering is going on sorting can be "disabled" ! Once the partial or complete results are available then sorting can be done only on the "results" and not the complete fooList.
    Or you insert new rows so that fooListFilteredIndexes stays sorted.

    Quote Originally Posted by ednakram View Post
    What do you think about this approach?
    That's basically what I proposed, so I am biased

    Cheers,
    _

  7. #7
    Join Date
    Dec 2016
    Posts
    16
    Thanks
    2
    Thanked 1 Time in 1 Post
    Qt products
    Qt4

    Default Re: Custom QSortFilterProxyModel for filtering large data

    Thanks @anda_skoa.

    I think this looks like a good approach. I think the main benefit here is that you don't have to maintain a source-proxy index book-keeping - the kind that is done in QSortFilterProxyModel - infact the real slowdown I see is in when the insertions happens to QVector used for that book-keeping.

    But what are the cons here? At the top of my head - can this be really made into a generic class which can be used for any kind of model? For example can it be plugged just as easily in any situation to make the sort/filter incremental feature work? e.g. Take a situation where there are chains of models which are nesting the real QList<Foo> deep down. Or perhaps those chains were "merging" data from multiple QList<foo> into one. Can this work in that situation without modifying the chains in between?

    If yes, have latest versions of QT thought about it - but maybe didnt go ahead because the problem was not reported by enough people? What the other things that will have to be take care of to make this work?

  8. #8
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Custom QSortFilterProxyModel for filtering large data

    It might be possible to do this generically but it would be a heck lot of work.

    These kind of advanced filtering/sorting techniques are usually best applied as close to the source data as possible, to be able to use knowledge about the data itself and how and when it might change.

    Cheers,
    _

  9. The following user says thank you to anda_skoa for this useful post:

    ednakram (24th January 2017)

Similar Threads

  1. QSortFilterProxyModel "inverse" filtering
    By aguayro in forum Newbie
    Replies: 1
    Last Post: 27th September 2014, 14:36
  2. Find out when filtering is doing in QSortFilterProxyModel
    By alizadeh91 in forum Qt Programming
    Replies: 5
    Last Post: 26th February 2013, 22:00
  3. Replies: 3
    Last Post: 6th June 2011, 13:34
  4. QSortFilterProxyModel not filtering properly
    By freemind in forum Qt Programming
    Replies: 9
    Last Post: 8th August 2010, 01:23
  5. Replies: 1
    Last Post: 10th January 2008, 14:38

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.