PDA

View Full Version : Rapid updating of QSortFilterProxyModel



jkrienert
29th December 2014, 15:20
Having a bit of trouble updating a proxyFilter for display in QtableView with 'rapidly' changing user selected ranges.

Currently the workflow is setup as:

[QAbstractTableModel]<--->[QSortFilterProxyModel]<--->[QtableView]<--->[GIS interface]

When the user selects a range of objects(in this case, grid cells) via the GIS interface, I push a string of 'ID' values associated with this selection to the proxyFilter; allowing their display in the QtableView.

Although this works for some selections in the upper section of the grid, it oddly doesn't elsewhere - and I'm a bit perplexed as to why.

And the proxy-code bits:
Within the __init__ function of the QtableView Class-


self.tableData = CSVModel(fileName)
self.proxyModel = QtGui.QSortFilterProxyModel()
self.proxyModel.setSourceModel(self.tableData)
self.proxyModel.setFilterKeyColumn(0)
self.tableView.setModel(self.proxyModel)

...

self.show()
self.selectedObjects()
self.iface.mapCanvas().selectionChanged.connect(se lf.selectedObjects)

The called function 'selectedObjects()'-


def selectedObjects(self):
## self.proxyModel.reset()
## self.proxyModel.dynamicSortFilter(True)
selectListIDs = []
editLayer = self.iface.activeLayer().name()
editLayer_Atrib = ftools_utils.getMapLayerByName(unicode("%s") % (editLayer))
totalFeatures = range(editLayer_Atrib.featureCount())
features = editLayer_Atrib.selectedFeatures()
for feature in features:
selectListIDs.append(list(feature.attributes())[0])
revisedSelection = "|".join([str(cellID) for cellID in selectListIDs])
print revisedSelection
self.proxyModel.setFilterRegExp('^('+str(revisedSe lection)+')$')

I have tried dynamicSortFilter(), and reset() to no effects.
Is there a fundamental bit for refreseh or something as such I am missing?

Added after 1 15 minutes:

...Alright, after spending some time experimenting it was reviealed that either using the mouse scroll wheel or resizing the QtableView window populated the table with the properly selected range(s).

Is there anyway to assure updates occur 'automatically', thus avoiding a initially blank display?

anda_skoa
29th December 2014, 19:25
The filter should have emitted respective signals and the view should have reacted to that.
E.g. rowsRemoved() if the new filter reduces the set

Can you check if the proxy is emitting any of these?

In general, if your model is not used with a different proxy in a second view, it might be worth considering implementing the filter in the model itself.

Usually the model can do sorting/filtering more efficiently since it "knows" the data.
E.g. if you add a filter criterium it could only have to test the currently accepted rows and remove those that no longer are instead of having to rebuild the whole set (which is what the QSortFilterProxyModel would have to do).

Cheers,
_

jkrienert
30th December 2014, 01:23
It seems I might be making use of QSortFilterProxyModel incorrectly. Is it proper to distinguish the proxy in its own class or alright to implement within the TableView class?
(Current needs only resort to a single TableView for this model)

Sorting out the rows within the TableModel makes more sence - but this wont require a reset of the Model upon every new ID range would it?
The documentation for filtration within the QAbstractTableModel seems to be limited, or perhaps I'm searching for the wrong aspects.
Would this be accomplished by way of flags or ...?



Also... Attached below is a visual reference to the current workings at the Ui level:

Key for the video-link below:
-Main window on upper right (with cursor) displays the GIS interface selection
-Python console below main window displays string of 'IDs' (x) sent to setFilterRegExp(x)
-Upper left window displays the updates to the QtableView care of the filterProxy


http://www.instructables.com/files/orig/F76/EUNF/I49RMVOI/F76EUNFI49RMVOI.mp4

Added after 16 minutes:

Short addition (for clarities sake)

Images with proxy.rowcount() printed in lower right corner [python console] as selected from yellow squares in main window upper right [GIS program] ...[and (lagged) display of TableView upper left]~

Partial initial display (notice proxy.rowCount() = 0):
10853

Full resultant display (after jostling the horizontal scroll bar left to right frequently):
10854

Nominal full display (initial, with proper proxy.rowCount()):
10855

...and some code:

self.iface.mapCanvas().selectionChanged.connect(se lf.selectedObjects)
def selectedObjects(self):
selectListIDs = []
editLayer = self.iface.activeLayer().name()
editLayer_Atrib = ftools_utils.getMapLayerByName(unicode("%s") % (editLayer))
totalFeatures = range(editLayer_Atrib.featureCount())
features = editLayer_Atrib.selectedFeatures()
for feature in features:
selectListIDs.append(list(feature.attributes())[0])
revisedSelection = "|".join([str(cellID) for cellID in selectListIDs])
print revisedSelection
self.proxyModel.setFilterRegExp('^('+str(revisedSe lection)+')$')
print self.proxyModel.rowCount()
del revisedSelection

anda_skoa
30th December 2014, 12:29
It seems I might be making use of QSortFilterProxyModel incorrectly. Is it proper to distinguish the proxy in its own class or alright to implement within the TableView class?

I am not sure what you mean. Do you have a subclass of QSortFilterProxyModel?
Or are you asking if it is OK to have the proxy model a member of the view?


I
Sorting out the rows within the TableModel makes more sence - but this wont require a reset of the Model upon every new ID range would it?

Not necessarily.
A reset usually indicates when a model's content changes in its whole, filtering differently could also be a combination of row insertions and removals.


I
The documentation for filtration within the QAbstractTableModel seems to be limited, or perhaps I'm searching for the wrong aspects.
Would this be accomplished by way of flags or ...?

It is not explicitly documented because it is not so much "filtering" as it is presenting a different view on the data.

Lets for example assume a simple filtering criteria: only show every second row.
The model could do that by returning half the number of the internal data as row count and multiply every incoming row number by two.
Or it could create a lookup table. Or even reduce the internal data set and reload from source if the filter is changed.

Anyway, the QSortFilterProxyModel should work, the other approach is mostly for potential optimizations

Cheers,
_

jkrienert
30th December 2014, 13:45
Very helpful thoughts. Thank you.

Is there any reason (of error on my part) for the 'print self.proxyModel.rowCount()' returning a zero? Even whilst when fiddling with the tableView window scale - the selected values eventually get displayed?


...is OK to have the proxy model a member of the view?
Currently my proxyModel is a member of the TableView.
It sounds like any optimization of TableView update delays might require moving filtering to the dataModel itself?
(would there be any sence in having a signal sent to the setData() function in my AbstractModel everytime the RegExp(list) is updated, causing the Model to push a dataChanged() callout to the TableView as a sort of adhoc refresh?)

The lookup-table sounds like it might be route to check out, and have not come across such a topic in documentation.
Would this involve incorporating a function within the dataModel that receives a limiting-list of 'IDs' to populate tableView, everytime the user selects a new range?

anda_skoa
30th December 2014, 14:15
Is there any reason (of error on my part) for the 'print self.proxyModel.rowCount()' returning a zero? Even whilst when fiddling with the tableView window scale - the selected values eventually get displayed?

No, the API of the proxy should work independently of any view.
I.e. if you call proxy.rowCount() before and after changing the filter it should print accurate values depending on both filter criteria.




Currently my proxyModel is a member of the TableView.

That's definitely ok.



It sounds like any optimization of TableView update delays might require moving filtering to the dataModel itself?

The proxy should react to any changes, direct or delayed, in the model. It should apply filter criteria changes immediately though.
If you want to delay the filtering, another option would be to subclass QSortFilterProxyModel, implement filterAcceptsRow() and do the delay between setting of your own filter data (not re-using any of the options of QSortFilterProxyModel) and the call to invalidateFilter().



(would there be any sence in having a signal sent to the setData() function in my AbstractModel everytime the RegExp(list) is updated, causing the Model to push a dataChanged() callout to the TableView as a sort of adhoc refresh?)

No, the proxy should be doing all the necessary signalling already.



The lookup-table sounds like it might be route to check out, and have not come across such a topic in documentation.
Would this involve incorporating a function within the dataModel that receives a limiting-list of 'IDs' to populate tableView, everytime the user selects a new range?

Yes. Your model would need a method to set the filter items, e.g. as a set of strings or a list.

Maybe you could try the setup with a simple QStringListModel as the source model?

Cheers,
_

jkrienert
30th December 2014, 16:50
I might have spoken awkwardly earlier about needs. No delay in filtering is needed, nor wanted actually.
The current setup requires the user to jostle the scroll bar rapidly to update TableView relative to the selected-values; of which I hope to alleviate.

Conclusively - it seems like the proxyFilter will work great for my cause; ergo - not really needing further optimization.
That is of course, only if I am able to overcome the hurdle(s) expressed in the video link below.
The actions depicted lead me to feel I am missing a crucial command, or some important aspect of ordering things:
(green tracers around the cursor when moved over the TableView indicate jostling the scroll wheel)
-Notice that the 'new selection rowCount()' does not equal 'last selection rowCount()'

http://www.instructables.com/files/orig/FRU/I0OW/I48CHLIO/FRUI0OWI48CHLIO.avi
Any clues?

code snip of print region:

features = editLayer_Atrib.selectedFeatures()
for feature in features:
selectListIDs.append(list(feature.attributes())[0])
revisedSelection = "|".join([str(cellID) for cellID in selectListIDs])
print 'before filter (last selection rowCount() ='+str(self.proxyModel.rowCount())+' )'
self.proxyModel.setFilterRegExp('^('+str(revisedSe lection)+')$')
print 'new selection IDs ='+str(revisedSelection)
print 'after filter (new selection rowCount() ='+str(self.proxyModel.rowCount())+' )'

jkrienert
30th December 2014, 19:24
Wow.
The un-technical procedure of 'switch this' ~ 'change that' ~ 'then repeat' revealed my blunder.
Changing activeRows from 35 to a number far exceeding any list length which will ever be used (1*10^12) completely dissolved my problems.

class CSVModel(QtCore.QAbstractTableModel):
activeRows = 35
def __init__(self, fileName, parent=None):
super(CSVModel,self).__init__()
self.header = []
self.rows = []
self.fileName = fileName
with open(self.fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headerIDx = 0
if idx is headerIDx:
self.header.append(row)
elif idx>headerIDx:
items = [field for field in row]
self.rows.append(items)
self.rowsLoaded = CSVModel.activeRows
The question is, why exactly was this limiting things - and is there a more dynamic means of setting this initialization parameter?
(fwiw, that sizable number adds absolutely no lag to opening the TableView - since the population of the table is based on the proxy selection :))

jkrienert
3rd January 2015, 15:32
To revist this a bit,
Maintaining 'activeRows' to a value equal to or greater than len(self.rows) <=[number of rows], works alright for small user selections - but is uncomfortably slow with large selections.
Furthermore, my implementation of AbstractTableModel was to efficiently populate the table based on an incremental loading of rows into the TableView (hence initially setting 'activeRows' to equal 35.
There must be a 'best of both worlds' possible, although my review of the doc's and forums has turned up little in the last week.
Any lingering clues out there?

anda_skoa
4th January 2015, 09:35
As far as I can tell from your code you already do all the loading in the constructor, so you could just set activeRows to len(rows) or discard activeRows as it is not needed, no?

Cheers,
_

jkrienert
4th January 2015, 13:34
Indeed, substituting len(self.rows) does enable functionality without selection error, yet with larger selections there is a major slow-down in load time overall (it is populating the table based on every row of data selected).

The benefit of implementing activeRows (when the number is 35 rather than len(rows)) allowed an initially small parcel of row population for rapid tableview display - then subsequent calling of more rows from the total model as needed (inferred) by moving the scroll bar. Simply put, this is like a 'pagination' method which only loads needed rows - greatly speeding up opening of the view and user workflow.

Ideally, I am hopeful to nest the functionality observed with 'activeRows' within an abstractModel. Is it proper to have a model within a model - and any clues as to if this would be a manner to fulfill said functionality ?
(If my logic is skewed, please bear in mind its product of a novice skill-set)

Maybe this logic sketch will help describe my intent:
10862

Also, this might help; written as current w/out the sketch implemented above:


# QTableView Ui handeler (aka tableView)
class CSVeditor(QtGui.QDialog, QtGui.QWidget):
# Initalize primary connections between [Data]<->[Proxy]<->[View]
# and commit fundamental tableView formating
def __init__(self, iface, fileName, parent=None):
super(CSVeditor, self).__init__(parent)
self.iface = iface
self.fileName = fileName
self.tableView = QtGui.QTableView(self)
self.tableData = CSVModel(fileName)
self.proxyModel = QtGui.QSortFilterProxyModel()
self.proxyModel.setSourceModel(self.tableData)
self.proxyModel.setFilterKeyColumn(0)
self.tableView.setModel(self.proxyModel)
cellSelectionModel = self.tableView.selectionModel()
cellSelectionModel.selectionChanged.connect(self.s elRange)
self.connect(self.proxyModel,QtCore.SIGNAL('dataCh anged(QModelIndex,QModelIndex)'),self.shareChanges )
# create buttons and connections
self.pushButtonSave = QtGui.QPushButton(self)
self.pushButtonSave.setText("Save Changes")
self.pushButtonSave.clicked.connect(self.on_pushBu ttonSave_clicked)
self.pushButtonClose = QtGui.QPushButton(self)
self.pushButtonClose.setText("Exit")
self.pushButtonClose.clicked.connect(self.Cancel)
# implement read only parameters (column specific)
self.ReadOnly = ReadOnly(self.tableView)
self.tableView.setItemDelegateForColumn(0,self.Rea dOnly)
# open Ui and populate based upon QgsLayer feature(s) selection
self.show()
self.selectedObjects()
self.iface.mapCanvas().selectionChanged.connect(se lf.selectedObjects)

# retrieve list of all selected objects on mapCanvas to proxy-filter displayed rows in CSV editor
def selectedObjects(self):
selectListIDs = []
editListIDs = []
editLayer = self.iface.activeLayer().name()
editLayer_Atrib = ftools_utils.getMapLayerByName(unicode("%s") % (editLayer))
totalFeatures = range(editLayer_Atrib.featureCount())
features = editLayer_Atrib.selectedFeatures()
for feature in features:
selectListIDs.append(list(feature.attributes())[0])
revisedSelection = "|".join([str(cellID) for cellID in selectListIDs])
self.proxyModel.setFilterRegExp('^('+str(revisedSe lection)+')$')
print self.proxyModel.rowCount()
print self.tableData.rowCount(self)

# nessecary slot and signal functions
@QtCore.pyqtSlot()
def on_pushButtonSave_clicked(self):
self.tableData.saveIt()
def Cancel(self):
self.close()

# main startup
if __name__ == "__main__":
app = QtGui.QApplication(sys.argv)
app.setApplicationName('CSV Editor')
main = CSVeditor(fileName)
main.show()
sys.exit(app.exec_())

#---------------------------------------------------------------------------------------------------

# readOnly column establishments in QtableView
class ReadOnly(QtGui.QItemDelegate):
def __init__(self, parent):
QtGui.QItemDelegate.__init__(self, parent)
def createEditor(self, parent, option, index):
item = QtGui.QLineEdit(parent)
item.setReadOnly(True)
return item
def setEditorData(self, editor, index):
editor.blockSignals(True)
editor.setText(index.model().data(index))
editor.blockSignals(False)
def setModelData(self, editor, model, index):
model.setData(index, editor.text())

#---------------------------------------------------------------------------------------------------

# QTableModel data handeler (aka tableData)
class CSVModel(QtCore.QAbstractTableModel):

activeRows = 1000000000000 #<=== this value is equivalent to len(rows) or ensuring that all rows are initially filtered via proxy
# load in csv data, and establish inital limiter on loaded row count (for efficency)
def __init__(self, fileName, parent=None):
super(CSVModel,self).__init__()
self.header = []
self.rows = []
self.fileName = fileName
with open(self.fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headerIDx = 0
if idx is headerIDx:
self.header.append(row)
elif idx>headerIDx:
items = [field for field in row]
self.rows.append(items)
self.rowsLoaded = CSVModel.activeRows

# relative axis index counts
def rowCount(self,index):
if not self.rows:
return 0
if len(self.rows) <= self.rowsLoaded:
return len(self.rows)
else:
return self.rowsLoaded
def columnCount(self,index):
return len(self.header[0])

# extended data retrieval based upon Qtableview / proxyFilter demands (scrolling)
def canFetchMore(self,index):
if len(self.rows) > self.rowsLoaded:
return True
else:
return False
def fetchMore(self,index):
reminder = len(self.rows) - self.rowsLoaded
itemsToFetch = min(reminder,CSVModel.activeRows)
self.beginInsertRows(index,self.rowsLoaded,(self.r owsLoaded+itemsToFetch-1))
self.rowsLoaded += itemsToFetch
self.endInsertRows()
def addRow(self,row):
self.beginResetModel()
self.rows.append(row)
self.endResetModel()

# source value data for cell(s) in QtableView
def data(self,index,role):
if index.isValid() and role == Qt.DisplayRole:
return self.rows[index.row()][index.column()]

# establish header data
def headerData(self,section,orientation,role):
if role != Qt.DisplayRole:
return
if orientation == Qt.Vertical:
return int(section)
if orientation == Qt.Horizontal:
return self.header[0][section]

# write changes from QtableView and proxyFilter to (self) tableData
def setData(self, index, value, role):
if index.isValid() and role == Qt.EditRole:
self.rows[index.row()][index.column()]=str(value)
self.dataChanged.emit(index, index)
return True
return False

# distinguish capabilities of total cells in tableData model
def flags(self, index):
return QtCore.Qt.ItemIsEditable | QtCore.Qt.ItemIsEnabled | QtCore.Qt.ItemIsSelectable

# one toast & peanutbutter-jelly sandwitch please
def saveIt(self):
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header[0])
for rowNumber in range(len(self.rows)):
fields = self.rows[rowNumber]
writer.writerow(fields)

jkrienert
4th January 2015, 15:54
Or, can a QSortFilterProxyModel be a trunk; housing, filtering, providing, and applying changes?

10864

In hindsight, this seems to not be a fishable alternative.

anda_skoa
4th January 2015, 17:02
A proxy model is always in between two entities, e.g. between the actual data model and the view, or between the data model and another proxy, or between two other proxies.
I.e. it always works on a source model.

Your own model can do anything you want it to do, including sorting, filtering, data manipulation, etc.

The proxy model approach is mostly a matter of convenience, e.g. not having to deal with sorting/filtering in your own code, or when there is no access to the source code of the actual model or changing it is not an option.

Cheers,
_

jkrienert
4th January 2015, 17:47
What would be the recommended manner of filtering within an AbstractModel?

Also, since the entirety of the source data is loaded and saved in the AbstractModel (used here), does internal filtering run into (write) complexities ? Hopefully not, since I am presuming the filtering method would mearly send a list of rows to table view, rather then edit the housed bulk data.

If this route is taken, there would be less overhead I assume (than with the QsortFilterProxyModel @ current)?

anda_skoa
4th January 2015, 18:15
Well, since you want to eventually write the data again, if I remember correctly, you will have to edit the bulk data.

One approach for filtering could be via a list of indexes into the actual data.

When unfiltered, the index list would be as long as the number of data rows, each entry being the index of the entry itself


[ 0, 1, 2, 3, 4, 5 .... ]


When you apply a filter, this index list gets reduced to the "visible" portion of the data.
Lets say the first two rows would be discarded by the filter:


[ 2, 3, 4, 5 .... ]


rowCount() would always be the length of this list. When data needs to be read or written, the row() of QModelIndex is the index into this second list, the value at that position is the actual row in the data list.

Cheers,
_

jkrienert
4th January 2015, 18:57
While the following (code below) runs without error, the tableView is never populated with rows.
Yet if I call 'print self.rows' after the 'filterData' function is ran - the list depicts every selected value which should be
published to the tableView.

Is this an incorrect means of doing the filtering directly; e.g. - within the AbstractModel?


class CSVModel(QtCore.QAbstractTableModel):

# load in csv data, and establish inital limiter on loaded row count (for efficency)
def __init__(self, iface, fileName, parent=None):
super(CSVModel,self).__init__()
self.iface = iface
self.rows = []
self.fileName = fileName
self.loadCSV(fileName)
self.rowsLoaded = len(self.rows)
self.iface.mapCanvas().selectionChanged.connect(se lf.filterData)

def loadCSV(self,fileName):
self.header = []
self.data = []
with open(self.fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headerIDx = 0
if idx is headerIDx:
self.header.append(row)
elif idx>headerIDx:
items = [field for field in row]
self.data.append(items)

def filterData(self):
editLayer_Atrib = ftools_utils.getMapLayerByName(unicode(self.iface. activeLayer().name()))
totalFeatures = range(editLayer_Atrib.featureCount())
features = editLayer_Atrib.selectedFeatures()
self.beginResetModel()
for feature in features:
ID = feature.attributes()[0]
self.rows.append(self.data[ID])
self.endResetModel()

# relative axis index counts
def rowCount(self,index):
if not self.rows:
return 0
if len(self.rows) <= self.rowsLoaded:
return len(self.rows)
else:
return self.rowsLoaded
def columnCount(self,index):
return len(self.header[0])

# source value data for cell(s) in QtableView
def data(self,index,role):
if index.isValid() and role == Qt.DisplayRole:
return self.rows[index.row()][index.column()]

# establish header data
def headerData(self,section,orientation,role):
if role != Qt.DisplayRole:
return
if orientation == Qt.Vertical:
return int(section)
if orientation == Qt.Horizontal:
return self.header[0][section]

# write changes from QtableView and proxyFilter to (self) tableData
def setData(self, index, value, role):
if index.isValid() and role == Qt.EditRole:
self.rows[index.row()][index.column()]=str(value)
self.dataChanged.emit(index, index)
return True
return False

# distinguish capabilities of total cells in tableData model
def flags(self, index):
return QtCore.Qt.ItemIsEditable | QtCore.Qt.ItemIsEnabled | QtCore.Qt.ItemIsSelectable

# Save Changes Made
def saveIt(self):
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header[0])
for rowNumber in range(len(self.rows)):
fields = self.rows[rowNumber]
writer.writerow(fields)



EDIT:
At this point I'm inclined to wonder if this thread should be laid to rest - and the topic moved anew considering the use of SortFilterProxyModel is possibly going to be bypassed...