PyQt5 SegFault when adding rows to QTreeView using QSortFilterProxyModel

[atlas]

the project is called Vivisect and it has been using PyQt5. Vivisect is a binary analysis framework (eg. a disassembler)
this is a relatively stable project, but while recently attempting to add in filtering capabilities to various QTreeView widgets using QSortFilterProxyModel, i'm getting somewhat inconsistent segfaults, with very little context (ie. none other than printed debug statements).

i've looked through this thread but i haven't been able to stop the segfaulting completely: https://www.qtcentre.org/threads/69208-PyQt5-QSortFilterProxyModel-index-from-wrong-model-passed-to-mapFromSource-segfalt

when i start up analysis, it always crashes (it populates the various widgets):

$ vivbin /bin/chown
Loaded (0.0685 sec) /bin/chown
Beginning analysis...
...analyzing exports.
0x0200ad31: Emulation Found 0x0200afd0 (from func: 0x0200ad20) via call 0x0200afd0
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
...
Dynamic Branch found at 0x2001d48 call rax
Segmentation fault (core dumped)

here's the relative code snippits (links to full code listed below):


class VivFilterModel(QSortFilterProxyModel):
def __init__(self, parent=None):
QSortFilterProxyModel.__init__(self, parent=parent)
self.setDynamicSortFilter(True)
self.setFilterKeyColumn(-1)

def __getattr__(self, name): # the existing "navModel" includes a couple functions
return getattr(self.sourceModel(), name)


class VivFilterView(QWidget):
'''
This is the primary window for the VQViv*Views if they want to include filtering
'''
window_title = '__undefined__'
view_type = None

def __init__(self, vw, vwqgui, *args, **kwargs):
QWidget.__init__(self)

self.view = self.view_type(vw, vwqgui, *args, **kwargs)
self.ffilt = VQFilterWidget(self)

layout = vq_basics.VBox(self.view, self.ffilt)
self.setLayout(layout)

self.ffilt.filterChanged.connect(self.textFilterCh anged)
self.setWindowTitle(self.view.window_title)

def textFilterChanged(self):
regExp = QtCore.QRegExp(self.ffilt.text(),
self.ffilt.caseSensitivity(),
self.ffilt.patternSyntax())

self.view.filterModel.setFilterRegExp(regExp)

class VQTreeModel(QtCore.QAbstractItemModel):
'''
A QT tree model that uses the tree API from visgraph
to hold the data...
'''

columns = ( 'A first column!', 'The Second Column!')
editable = None
dragable = False

def __init__(self, parent=None, columns=None):

if columns != None:
self.columns = columns

QtCore.QAbstractItemModel.__init__(self, parent=parent)
self.rootnode = VQTreeItem((), None)

if self.editable == None:
self.editable = [False,] * len(self.columns)

def vqEdited(self, pnode, col, value):
return value

def append(self, rowdata, parent=None):
if parent == None:
parent = self.rootnode

pidx = self.createIndex(parent.row(), 0, parent)
i = len(parent.children)
self.beginInsertRows(pidx, i, i)
node = parent.append(rowdata)
self.endInsertRows()
self.layoutChanged.emit()
return node
...
def sort(self, colnum, order=0):
cmpf = VQTreeSorter(colnum, order)
self.layoutAboutToBeChanged.emit()
self.rootnode.children.sort(cmp=cmpf)
self.layoutChanged.emit()

def flags(self, index):
if not index.isValid():
return 0
flags = QtCore.QAbstractItemModel.flags(self, index)
col = index.column()
if self.editable[col]:
flags |= QtCore.Qt.ItemIsEditable
if self.dragable:
flags |= QtCore.Qt.ItemIsDragEnabled
return flags

def columnCount(self, parent=None):
return len(self.columns)

def data(self, index, role):
if not index.isValid():
return None

item = index.internalPointer()
if role == QtCore.Qt.DisplayRole:
return item.data(index.column())

if role == QtCore.Qt.UserRole:
return item

return None

def setData(self, index, value, role=QtCore.Qt.EditRole):

node = index.internalPointer()
if not node:
return False

# If this is the edit role, fire the vqEdited thing
if role == QtCore.Qt.EditRole:
value = self.vqEdited(node, index.column(), value)
if value == None:
return False

node.rowdata[index.column()] = value
self.dataChanged.emit(index, index)

return True

def headerData(self, column, orientation, role):
if ( orientation == QtCore.Qt.Horizontal and
role == QtCore.Qt.DisplayRole):

return self.columns[column]

return None

def index(self, row, column, parent):

if not self.hasIndex(row, column, parent):
return QtCore.QModelIndex()

pitem = parent.internalPointer()
if not pitem:
pitem = self.rootnode

item = pitem.child(row)
if not item:
return QtCore.QModelIndex()

return self.createIndex(row, column, item)

def parent(self, index):

if not index.isValid():
return QtCore.QModelIndex()

item = index.internalPointer()
if not item:
return QtCore.QModelIndex()

pitem = item.parent

if pitem == self.rootnode:
return QtCore.QModelIndex()

if pitem == None:
return QtCore.QModelIndex()

return self.createIndex(pitem.row(), 0, pitem)
...


the working pull request is here: https://github.com/vivisect/vivisect/pull/237/files
the full branch is here: https://github.com/atlas0fd00m/vivisect/tree/filtered_views

i expect the bug is some simple thing i'm doing (eg. not providing a parent, or maybe connecting signals in incompatible ways, etc...)
but i'm at a complete loss. i don't understand why adding in a QSortFilterProxyModel would cause the entire python interpreter to segfault.

thank you in advance for any help and guidance you can provide? i'm relatively new to pyqt and qt in general, so don't worry about offending me.
let me know if you need any other context (or clarification)

@

Added after 9 minutes:

i ran out of room for these bits of relevant code.


class VQVivFunctionsViewPart(VQVivTreeView):

_viv_navcol = 0
window_title = 'Functions'
columns = ('Name','Address', 'Size', 'Ref Count')


def __init__(self, vw, vwqgui):
VQVivTreeView.__init__(self, vw, vwqgui, withfilter=True)
self.navModel = VivNavModel(self._viv_navcol, self, columns=self.columns)
self.filterModel = VivFilterModel()
self.filterModel.setSourceModel(self.navModel)
self.setModel(self.filterModel)

self.vqLoad()
self.vqSizeColumns()

class VQVivFunctionsView(VivFilterView):
view_type = VQVivFunctionsViewPart

class VQTreeView(QTreeView):

def __init__(self, parent=None, cols=None, **kwargs):
QTreeView.__init__(self, parent=parent, **kwargs)
self.setSortingEnabled(True)
self.setAlternatingRowColors(True)

if cols != None:
model = VQTreeModel(parent=self, columns=cols)
self.setModel( model )

def vqSizeColumns(self):
c = self.model().columnCount()
for i in xrange(c):
self.resizeColumnToContents(i)

def setModel(self, model):
model.dataChanged.connect( self.dataChanged )
model.rowsInserted.connect( self.rowsInserted )
return QTreeView.setModel(self, model)

class EnviNavModel(vq_tree.VQTreeModel):

dragable = True

def __init__(self, navcol, parent=None, columns=None):
vq_tree.VQTreeModel.__init__(self, parent=parent, columns=columns)
self.navcol = navcol

def mimeData(self, idx):
pnode = idx[0].internalPointer()
expr = pnode.rowdata[self.navcol]
mdata = QtCore.QMimeData()
mdata.setData('envi/expression',str(expr))
return mdata



if you are looking through the codebase, the PyQt5 files of interest should be:

vivisect/qt/views.py (https://github.com/atlas0fd00m/vivisect/blob/filtered_views/vivisect/qt/views.py)
vqt/tree.py (https://github.com/atlas0fd00m/vivisect/blob/filtered_views/vqt/tree.py)
envi/qt/memory.py (https://github.com/atlas0fd00m/vivisect/blob/filtered_views/envi/qt/memory.py)
vqt/common.py (https://github.com/atlas0fd00m/vivisect/blob/filtered_views/vqt/common.py)
vivisect/qt/main.py (https://github.com/atlas0fd00m/vivisect/blob/filtered_views/vivisect/qt/main.py)

i realize it's a little complex, but i appreciate the help, and i'm happy to provide whatever context may make it easier to troubleshoot.

thanks!
@

[atlas]

i don't know if it's at all related, but since migrating to PyQt5 (from PyQt4), i've been getting a ton of these messages:


QXcbConnection: XCB error: 8 (BadMatch), sequence: 3577, resource id: 190841046, major code: 130 (Unknown), minor code: 3

(running Linux/Kubuntu 18.04)

[anda_skoa]

The filter model should not cause any problem by itself, but it might be triggering bugs in the source model.

The code looks pretty good though.
I can see only one unchecked access of an internal pointer: in data() when returning the value for DisplayRole.

You also don't need to emit layoutChanged() in append, begin/endInsertRows() handle that correctly.

I see you are connecting to the model's dataChanged() and rowsInserted() signals in your view.
Maybe it is holding some data that gets invalidate when the filter removes entries?

Cheers,
_

[atlas]

thank you for your insights! i somehow missed your post. i'll have to check my notifications.

i'll look into that. i also had another user hit me off-list, recommending a change to a couple lines which call model() but which should also directly access sourceModel():

in "vivisect/qt/views.py":

- idx = self.model().createIndex(pnode.row(), col, pnode)
+ idx = self.model().sourceModel().createIndex(pnode.row() , col, pnode)
# We are *not* the edit role...
- self.model().setData(idx, val, role=QtCore.Qt.DisplayRole)
+ self.model().sourceModel().setData(idx, val, role=QtCore.Qt.DisplayRole)
(thanks Norm!)

hopefully these pointers can help identify the instability.
again, i appreciate your insights!
i'll let you know what i find.

Added after 17 minutes:



You also don't need to emit layoutChanged() in append, begin/endInsertRows() handle that correctly.

I see you are connecting to the model's dataChanged() and rowsInserted() signals in your view.
Maybe it is holding some data that gets invalidate when the filter removes entries?

_

if i remove the call to emit, nothing added to the view. i end up with an empty view. i'm not doubting what you're saying about begin/endInsertRows(), i'm wondering how my code is doing odd stuff which is keeping it from operating correctly. and if that may also cause my problems.
as for holding data that gets invalidated when the filter removes entries, unfortunately the segfaults are occurring before i get to filter anything.

thanks again!

[atlas]

The filter model should not cause any problem by itself, but it might be triggering bugs in the source model.

The code looks pretty good though.
I can see only one unchecked access of an internal pointer: in data() when returning the value for DisplayRole.

You also don't need to emit layoutChanged() in append, begin/endInsertRows() handle that correctly.

I see you are connecting to the model's dataChanged() and rowsInserted() signals in your view.
Maybe it is holding some data that gets invalidate when the filter removes entries?

Cheers,
_

i'm having pretty good luck with the changes requested and no longer segfaulting....
i'm now struggling with dramatic delays since adding the filter, and i think it may have to do with my event handling.
i'm trying to remove the self.layoutChanged.emit() and everything is much faster, but the QTreeView displays nothing. i can tell that there's data in the model, because filtering takes a while (this test bin has 20000 entries in the tree) to filter. i've tried connecting things differently, but i'm not seeing changes. i've connected the model to some debugging functions, and i can *see* that rowsAboutToBeInserted and rowsInserted signals are emitted, but nothing seems to cause the QTreeView to visually show any of the data.

how can i get the QTreeView to adjust what data it thinks is there without emitting layoutChanged?

i tried going back to the code pre-filtering and removing the emits, and have similar troubles. however, oddly, i get a few entries listed. i can see that there are way more items than listed, because when i change from sort-ascending to sort-descending, the items are completely different ones (ie. i'm looking at but a window of the available entries). is there some way to get QTreeView to refresh how many items it's supposed to be displaying?

help? thanks in advance :)

[anda_skoa]

This is very strange, layoutChanged() should definitely not be needed when adding or removing rows.

When you say "going back to pre-filtering", do you mean you set the main model directly on the view?
And it still doesn't behave correctly?

Cheers,
_

[atlas]

This is very strange, layoutChanged() should definitely not be needed when adding or removing rows.

When you say "going back to pre-filtering", do you mean you set the main model directly on the view?
And it still doesn't behave correctly?

Cheers,
_


correct, that's what i meant.

however, long story short, i removed it from a different branch (without filtering) but left in two seemingly minor changes, and things *seem* to work properly.

https://github.com/atlas0fd00m/vivisect/tree/atlas_merged
(vqt/tree.py and vivisect/qt/views.py)

the filter_views branch is here:
https://github.com/atlas0fd00m/vivisect/tree/filtered_views


i just had to back out the filter_views branch because, while i no longer segaulted, the gui never come up when filtering. i could get the gui to come up if i removed the layoutChanged.emit() line, but then none of the QTrees showed their data.

any ideas? it seems that we've worked the code around so that the layoutChanged.emit() causes problems with and without filters :) it some warped way, that feels like progress.

@

btw, i apologize again for the delay in response. i just discovered that i wasn't subscribed to this thread :) corrected.