Possible conversion from QStandardItemModel to QAbstractTableModel?

[jkrienert]

At the moment a QStandardItemModel is being used to support QTableView, but a future need to process large CSV files with > 50,000 rows + facilitates the need for an alternative method.

I have read several great PyQt based articles on the subject (see below) of using a QAbstractTableModel to hold the data which will then provide QtableView with items 'on demand', yet novice programing skills are making the transition anything but progressive.

http://www.saltycrane.com/blog/2007/06/pyqt-42-qabstracttablemodelqtableview/
http://sateeshkumarb.wordpress.com/2012/04/01/paginated-display-of-table-data-in-pyqt/
https://gist.githubusercontent.com/MarshallChris/6019350/raw/3b48b8106935e04d4111bfcc24a06bc09b52bd09/combobox.py

Below are the primary bits of code (currently without QAbstractTableModel implemented) called by another piece of software, in all their existing error glory.


import csv
import sys
from PyQt4 import QtGui, QtCore
from PyQt4.QtCore import Qt, QVariant
global xLoc
global yLoc
global itemIdxs

class CSVeditor(QtGui.QDialog, QtGui.QWidget):
def __init__(self, fileName, horizHeaders, vertHeaders, parent=None):
super(CSVeditor, self).__init__(parent)

self.horizHeaders = horizHeaders
self.fileName = fileName
self.model = QtGui.QStandardItemModel(self)
self.model.setHorizontalHeaderLabels(horizHeaders)

self.tableView = QtGui.QTableView(self)
self.tableView.setModel(self.model)
self.tableView.horizontalHeader().setStretchLastSe ction(False)
self.tableView.setAlternatingRowColors(True)
self.tableView.hideColumn(0)
self.tableView.hideColumn(1)
self.tableView.hideColumn(2)
self.tableView.setSelectionMode(QtGui.QAbstractIte mView.ExtendedSelection)
selectionModel = self.tableView.selectionModel()
selectionModel.selectionChanged.connect(self.selRa nge)
shortcut = QtGui.QShortcut(self)
shortcut.setKey("Enter")
self.connect(shortcut, QtCore.SIGNAL("activated()"), self.shareChanges)
self.tableView.clicked.connect(self.Release)

# autoLoad CSV from workspace [to be edited for dropdown CSV upload selection(s)]
with open(fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
header = 0
if idx is header:
pass
elif idx>0:
items = [QtGui.QStandardItem(field) for field in row]
self.model.appendRow(items)
self.model.setVerticalHeaderLabels(vertHeaders)
self.show()

# index of selected range, and parent item (last index) for batch application of user-value changes
def selRange(self,selected,deselected):
selectedIdx = self.tableView.selectedIndexes()
idxCol = []
idxRow = []
for row in selectedIdx:
idxRow.append(row.row())
for column in selectedIdx:
idxCol.append(column.column())
global itemIdxs
itemIdxs = zip(idxRow,idxCol)

def shareChanges(self):
self.tableView.clicked.connect(self.Release)
try:
updatedVal = self.model.data(self.model.index(xLoc,yLoc+3))
print xLoc,yLoc,updatedVal
except AttributeError:
pass
readOnlyCols = [0]
try:
for i in itemIdxs:
row = i[0]
col = i[1]
if col in readOnlyCols:
pass
else:
self.model.setData(self.model.index(row,col), updatedVal, Qt.DisplayRole)
except AttributeError:
print ":("
pass

# button induced save-change to CSV (NO UNDO YET!)
def writeCsv(self, fileName):
with open(fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.horizHeaders)
for rowNumber in range(self.model.rowCount()):
fields = [self.model.data(self.model.index(rowNumber,columnN umber),QtCore.Qt.DisplayRole) for columnNumber in range(self.model.columnCount())]
writer.writerow(fields)
print ':)'

@QtCore.pyqtSlot()
def on_pushButtonWrite_clicked(self):
self.writeCsv(self.fileName)

def Cancel(self):
self.close()

def closeEvent(self,event):
self.deleteLater()

def Release(self,index):
global xLoc
global yLoc
xLoc,yLoc = index.row(),(index.column()-3)

Is it feasible to chop shop existing, or start from scratch rather?

[anda_skoa]

As a first step you could make a table model that holds all the data, e.g. in a 2-dimensional array.
That should allow you to basically use the same loading code you are currently using.

Cheers,
_

[jkrienert]

Thanks for the quickness anda_skoa
In this manner (?) :


class CSVModel(QtCore.QAbstractTableModel):
def __init__(self, CSVdata, fileName, parent = None):
QAbstractTableModel.__init__(self, parent)
self.CSVreader.rows = CSVdata
def rowCount(self, parent):
return len(self.CSVreader.rows)
def columnCount(self, parent):
return len(self.CSVreader.header)
def data(self, index, role):
self.CSVreader.rows[index.row()][index.column()]
def CSVreader(self,fileName):
header = []
rows = []
with open(fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headIDx = 0
if idx is headIDx:
header.append(row)
elif idx>headIDx:
items = [field for field in row]
rows.append(items)

[d_stranz]

but a future need to process large CSV files with > 50,000 rows + facilitates the need for an alternative method.

Constructing QStandardItem instances for every cell in a table is a lot of overhead, so mapping your data from an internal data structure into a table model instead of using QStandardItemModel would be good.

And as anda_skoa said, for the first step, simply implement the table model to hold the entire set of data. If this indeed kills performance, then go to plan B. Don't try to optimize before you know you have to.

[jkrienert]

Thanks d_stranz! I am currently working on doing just what you both have recommended - hopefully to express some results shortly.
Looking forward,

EDIT: Added information

Something seems to be in the right order.
With downward scrolling, the row count (and scroll space) increases. These changes also express quicker load time then previous.
However this haste could be due to the following woes.

Although the initial connection between the QAbstractTableModel and QtableView seems to be working, the table is not getting populated with row data.

class CSVModel(QtCore.QAbstractTableModel):
activeRows = 35
def __init__(self, fileName, parent=None):
super(CSVModel,self).__init__()
header = []
CSVarray = []
fileName = r'E:\Documents\SIUC\2014\Fall\440 - Hydro\QGIS\test_bay\CSVtesting\mfLayer1_Grid.csv'
with open(fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headerIDx = 0
if idx is headerIDx:
header.append(row)
elif idx>headerIDx:
items = [field for field in row]
CSVarray.append(items)
self.header = header
self.rows = CSVarray
self.rowsLoaded = CSVModel.activeRows
def rowCount(self,index):
if not self.rows:
return 0
if len(self.rows) <= self.rowsLoaded:
return len(self.rows)
else:
return self.rowsLoaded
def canFetchMore(self,index):
if len(self.rows) > self.rowsLoaded:
return True
else:
return False
def fetchMore(self,index):
reminder = len(self.rows) - self.rowsLoaded
itemsToFetch = min(reminder,CSVModel.activeRows)
self.beginInsertRows(index,self.rowsLoaded,(self.r owsLoaded+itemsToFetch-1))
self.rowsLoaded += itemsToFetch
self.endInsertRows()
def addRow(self,row):
self.beginResetModel()
for i in self.CSVarray:
self.rows.append(i)
self.endResetModel()
def columnCount(self,index):
return len(self.header[0])
def data(self,index,role=Qt.DisplayRole):
col = index.column()
row = self.rows[index.row()]
if role == Qt.DisplayRole:
if col == 0:
return str('meow 1')
elif col == 1:
return str('meow 2')
return
def headerData(self,section,orientation,role=Qt.Displa yRole):
if role != Qt.DisplayRole:
return
if orientation == Qt.Horizontal:
return self.header[0][section]
return int(section+1)
10836

[d_stranz]

Ah, your problem is most likely lines 49 and 51 of your code. Everyone knows that you can't get cats to do anything you want them to. You should have tried it with dogs instead... :)

I'll think on this code and see if I can see a real problem. Not a Python expert, but code is code.

Edit: If I understand the code, it looks like you are opening your CSV file in binary mode ("rb"). Is that what your csvReader expects?
Further edit: Never mind. You are getting the headers, so that part is OK.

[anda_skoa]

I am not an expert in Python, but aren't you missing one return in your data method?

Also make sure it returns a QVariant instance.

Cheers,
_

[jkrienert]

To d_stranz:
Meow?
[working on 44 -52 presently]

As far as my (novice) knowledge goes - the 'r' designates a read operation, and the 'b' is a platform denotation that the csv is an 'object'. But as you mentioned, the 'b' is referring to binary formatting?
...You might have a point:

http://stackoverflow.com/questions/1020053/writing-with-pythons-built-in-csv-module

Removing the 'b' didn't present any difference in current operation, and this sections ('rb') presence in the original QStandardItemModel caused no errors - so I suppose it might be ok... (reference image of table in 'standard' below)
10838
Thanks for taking the time to birds-eye this code. I would be happy to post the whole enchilada - if it would help.
There are several portions of the code which need resorting. For example, mouseRelease event to update a variable for application to all selected 'cells' in the table. Such is better left for another day until overcoming the current.



To anda_skoa:
Your right about that return!
For some reason (be it the version of Python3 or Qt4) I am unable to get functionality from QVariant instances.
This could be due to (my) syntax errors though...

To all:
This bit doesn't seem to work either ... It must not be barking like a dog should (@d_stranz)

def data(self,index,role):
col = index.column()
row = self.rows[index.row()]
if role == Qt.DisplayRole:
for idx, i in enumerate(self.header):
if col == idx:
return row[idx]

update
Hazah!

def data(self,index,role):
col = index.column()
row = self.rows[index.row()]
if role == Qt.DisplayRole:
return row[col]
Table now populates but editing is not possible.
Probably much is needed to enable such...
10840

[anda_skoa]

Table now populates but editing is not possible.
Probably much is needed to enable such...


Implement flags() to make the cell's editable and setData() to take the modified data and store it.
Emit dataChanged() when the data actually changed.

Cheers,
_

[jkrienert]

Implement flags() ... and setData() to take the modified data and store it.

These aspects were successfully incorporated (thank you!) - yet moving forward to enabling a proper update of the original CSV has run into some snags.

The vertical header (row numbers) is properly associated with the total number of rows in the AbstractTableModel, but the entire tables data is 'static' to the last columns value starting at the furthest extent of the initial QtableView load-up [35 rows] (picture below might be more descriptive).
Possibly due to conflict of the CSVwriter function (within the QtableView class), and syntax errors?

10841

Also, is there anyway to start the row index represented in the vertical header at 0 rather then 1? It has an association with IDs in the CSV dataset, and would be useful to keep in sync.


Writer function within QtableView Class:

def writeCsv(self, fileName):
with open(fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.horizHeaders)
for rowNumber in range(len(self.tableData.rows)):
fields = [self.tableData.data(self.tableData.index(rowNumber ,columnNumber),QtCore.Qt.DisplayRole) for columnNumber in range(self.tableData.columnCount(self))]
writer.writerow(fields)
print ':)'

[anda_skoa]

The vertical header (row numbers) is properly associated with the total number of rows in the AbstractTableModel, but the entire tables data is 'static' to the last columns value starting at the furthest extent of the initial QtableView load-up [35 rows] (picture below might be more descriptive).
Possibly due to conflict of the CSVwriter function (within the QtableView class), and syntax errors?

Have you checked the array?
Does it contain other values for these indexes?



Also, is there anyway to start the row index represented in the vertical header at 0 rather then 1? It has an association with IDs in the CSV dataset, and would be useful to keep in sync.

You just need to implement the "orientation is vertical" branch of headerData()

Cheers,
_

[jkrienert]

Indeed the CSVarray, which is referenced via 'self.rows' is a full dataset of the CSV source.

The image I posted regarding the save error was a CSV reopened in the editor.
Ergo, the extent to which you scroll in the initial open CSV is the maximum edits saved to the revised CSV.
for example, when I scrolled halfway through the total rows, the saved CSV has proper data up to that point (halfway), the rest being '98' duplicates of the value in the last column.
If I don't scroll at all, the only saved updates are to the last row visible (loaded) in the table window.

I'm inclined to believe that maybe this has to do with the relationship (position) of the CSVwriter from the AbstractTableModel?



implement the "orientation is vertical" branch of headerData()

Success!

def headerData(self,section,orientation,role=Qt.Displa yRole):
if role != Qt.DisplayRole:
return
if orientation == Qt.Vertical:
return int(section)
if orientation == Qt.Horizontal:
return self.header[0][section]

[anda_skoa]

I'm inclined to believe that maybe this has to do with the relationship (position) of the CSVwriter from the AbstractTableModel?

I think your problem is the asymmetry in how you handle I/O.

You do the loading inside the model and the writing outside.

Maybe put the saving into the model as well, and operate on the actual data instead of going through the abstraction that only the view needs?

For inspiration have a look at QSqlTableModel.
It also provide read/write access to tabular data and also implements load/save from/to backend, with different strategies for different behavioral trade-offs. Your target might be something like its OnManualSubmit strategy

Cheers,
_

[d_stranz]

You do the loading inside the model and the writing outside.

Yes, exactly. QTableView will load rows only as needed (eg. when they become visible through scrolling or programmatically). So, since your save method is based on what is contained in the view, that's all that will be written out.

As anda_skoa said, you need to move the save code so it works off the model itself and not the view of it.

[jkrienert]

Your advice makes sence; emplace the 'save' function within the model class to ensure write of the full data contents.
However, my first take on such reveals no call of the save function from a button click in the View class:

View Class button references:

def __init__(...)
...
self.pushButtonSave = QtGui.QPushButton(self)
self.pushButtonSave.setText("Save Changes")
self.pushButtonSave.clicked.connect(self.on_pushBu ttonSave_clicked)

...

@QtCore.pyqtSlot()
def on_pushButtonSave_clicked(self):
print 'meow 1'
CSVModel.saveIt

Model Class save function:

def saveIt(self):
print 'meow 2'
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header)
for rowNumber in range(len(self.rows)):
fields = [self.data(self.index(rowNumber,columnNumber),QtCor e.Qt.DisplayRole) for columnNumber in range(self.columnCount(self))]
writer.writerow(fields)
print 'happy cat'

While I do get a return for 'meow 1', 'meow 2' never shows. This occurs without error prompt(s), exhibiting a situation as if the save function content simply read 'pass'.






UPDATE:
Added after 1hr:

Turns out the View Class 'on_pushButtonSave_clicked' was a bit mixed up. This got things working:

@QtCore.pyqtSlot()
def on_pushButtonSave_clicked(self):
CSVModel.saveIt(CSVModel(self))

Although, now it seems my 'saveIt' function in the Model Class is a bit screwy. Error lays in the row write statements [line 6]:

def saveIt(self):
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header[0])
for rowNumber in range(len(self.rows)):
fields = [self.data(self.index(rowNumber,columnNumber),QtCor e.Qt.DisplayRole) for columnNumber in range(self.columnCount(self))]
writer.writerow(fields)
The same problem as previous is exhibited (partial write-out of data ~ see below) when reopening the CSV file. Working on this now...
10842

[anda_skoa]

The same problem as previous is exhibited (partial write-out of data ~ see below) when reopening the CSV file. Working on this now...


You still access the data through the model's data() method, might be more straight forward to use self.rows.

Cheers,
_

[jkrienert]

For simplicitys sake, I agree - but it seems I might be making variable associations improperly or some other reference error.
When using the following, I get a fully saved CSV - yet only displaying the original values, without any of the 'applied' edits:

def saveIt(self):
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header[0])
for rowNumber in range(len(self.rows)):
fields = self.rows[rowNumber]
writer.writerow(fields)
print ':)'


I have a feeling the mishap is spawned in this reference earlier in the AbstractModel:

header = []
CSVarray = []
self.fileName = r'E:\Documents\SIUC\2014\Fall\440 - Hydro\QGIS\test_bay\CSVtesting\mfLayer1_Grid.csv'
with open(self.fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headerIDx = 0
if idx is headerIDx:
header.append(row)
elif idx>headerIDx:
items = [field for field in row]
CSVarray.append(items)
self.header = header
self.rows = CSVarray
self.rowsLoaded = CSVModel.activeRows


I wonder my error occurs in sending user updates to the array data?

[anda_skoa]

How does your setData() look like?

Are you writing into self.rows there?

Cheers,
_

[jkrienert]

For fullness sake:

class CSVModel(QtCore.QAbstractTableModel):
activeRows = 35
def __init__(self, fileName, parent=None):
super(CSVModel,self).__init__()
header = []
CSVarray = []
self.fileName = r'E:\Documents\SIUC\2014\Fall\440 - Hydro\QGIS\test_bay\CSVtesting\mfLayer1_Grid.csv'
with open(self.fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headerIDx = 0
if idx is headerIDx:
header.append(row)
elif idx>headerIDx:
items = [field for field in row]
CSVarray.append(items)
self.header = header
self.rows = CSVarray
self.rowsLoaded = CSVModel.activeRows

def rowCount(self,index):
if not self.rows:
return 0
if len(self.rows) <= self.rowsLoaded:
return len(self.rows)
else:
return self.rowsLoaded
def canFetchMore(self,index):
if len(self.rows) > self.rowsLoaded:
return True
else:
return False
def fetchMore(self,index):
reminder = len(self.rows) - self.rowsLoaded
itemsToFetch = min(reminder,CSVModel.activeRows)
self.beginInsertRows(index,self.rowsLoaded,(self.r owsLoaded+itemsToFetch-1))
self.rowsLoaded += itemsToFetch
self.endInsertRows()
def addRow(self,row):
self.beginResetModel()
self.rows.append(row)
self.endResetModel()
def columnCount(self,index):
return len(self.header[0])
def data(self,index,role):
col = index.column()
row = self.rows[index.row()]
if role == Qt.DisplayRole:
return row[col]
def headerData(self,section,orientation,role):
if role != Qt.DisplayRole:
return
if orientation == Qt.Vertical:
return int(section)
if orientation == Qt.Horizontal:
return self.header[0][section]
def setData(self, index, value, role):
print value
if role == Qt.EditRole:
col = index.column()
row = self.rows[index.row()]
row[col]=value
return True
return False
def flags(self, index):
return QtCore.Qt.ItemIsEditable | QtCore.Qt.ItemIsEnabled | QtCore.Qt.ItemIsSelectable
def saveIt(self):
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header[0])
for rowNumber in range(len(self.rows)):
fields = self.rows[rowNumber]
writer.writerow(fields)
print ':)'

[anda_skoa]

yes, dataChanged() should be emitted, but that should not impact the saving.

Does line 60 get a reference to the row or does it create a copy?

If you call data() at the end of setData(), does it return the new value?

Cheers,
_

[jkrienert]

It is odd, that printing 'self.rows' at the end of the setData fuction returns a list with proper updates, yet if printing 'self.rows' at the beginning of the saveIt fuction, the list does not show any of these updates (please see attached image at footer)
It seems maybe anda_skoa is right; in that possibly rather than reference - the current is creating a duplicate array...

Below are changes reflected to ensure that (previously line 60) is getting a reference and not a copy [hopefully its correct]:

def setData(self, index, value, role):
if index.isValid() and role == Qt.EditRole:
## col = index.column()
## row = self.rows[index.row()]
## row[col]=value
self.rows[index.row()][index.column()]=value
print self.rows[0]
self.dataChanged.emit(index, index)
return True
return False
def flags(self, index):
return QtCore.Qt.ItemIsEditable | QtCore.Qt.ItemIsEnabled | QtCore.Qt.ItemIsSelectable
def saveIt(self):
print self.rows[0]
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header[0])
for rowNumber in range(len(self.rows)):
fields = self.rows[rowNumber]
writer.writerow(fields)
print ':)'
10846

[jkrienert]

Suspicious of calling saveIt() in the AbstractModel via the QtableView nested button. I wonder if its causing the initial load-up of the CSV data to overwrite any editing just before save?
If this is the case, how can such be avoided?

Button call function in QtableView:

@QtCore.pyqtSlot()
def on_pushButtonSave_clicked(self):
CSVModel(self).saveIt()

__init__ for AbstractTableModel, and subsequent saveIt() function therein:

def __init__(self, fileName, parent=None):
super(CSVModel,self).__init__()
self.header = []
self.rows = []
self.fileName = r'E:\Documents\SIUC\2014\Fall\440 - Hydro\QGIS\test_bay\CSVtesting\mfLayer1_Grid.csv'
with open(self.fileName, "rb") as fileInput:
for idx, row in enumerate(csv.reader(fileInput)):
headerIDx = 0
if idx is headerIDx:
self.header.append(row)
elif idx>headerIDx:
items = [field for field in row]
self.rows.append(items)
self.rowsLoaded = CSVModel.activeRows

def saveIt(self):
with open(self.fileName, "wb") as fileOutput:
writer = csv.writer(fileOutput)
writer.writerow(self.header[0])
for rowNumber in range(len(self.rows)):
fields = self.rows[rowNumber]
writer.writerow(fields)

[anda_skoa]

Could it be that you are creating a new model and call saveIt() on that instance instead of calling saveIt() in the model used by the view?

Cheers,
_

[jkrienert]

[SOLVED]

It could be, and would be indeed!
A glaring mistake I overlooked. Thank you anda_skoa.
This postings conclusion successfully achieved the migration from standard to abstracttablemodel.
Now on to refining flags and connections in this module!

For reference, the following resolved the problem:


self.tableData = CSVModel(self)

...

@QtCore.pyqtSlot()
def on_pushButtonSave_clicked(self):
self.tableData.saveIt()