[PyQt] QTableView displying 100.000.000 rows

[Gad82]

Hi everyone,

as the title says I'm trying to display 100.000.000 rows in my QTableView but I'm having some problems. When I launch my script, the widget seems to hang. After some investigations it turns out that the problem seems to be related to the headerData function which is called once for each row in the table. It does not seems to be normal, since I expect it to be called only for the currently displayed rows.
Here a test code displaying the problem:

import sys 
from PyQt4.QtCore import * 
from PyQt4.QtGui import * 
 
 
class MyWindow(QWidget): 
    def __init__(self, *args): 
        QWidget.__init__(self, *args) 
 
        # create table
        table = QTableView()
        tm = MyTableModel(self) 
        table.setModel(tm)
 
        # layout
        layout = QVBoxLayout()
        layout.addWidget(table) 
        self.setLayout(layout) 
 
class MyTableModel(QAbstractTableModel):  
    def rowCount(self, parent): 
        return 100000000
 
    def columnCount(self, parent): 
        return 10
 
    def data(self, index, role): 
        if not index.isValid(): 
            return QVariant() 
        elif role != Qt.DisplayRole: 
            return None
        return str((index.row(), index.column()))
 
    def headerData(self, section, orientation, role):
        print "headerData", section
        if role == Qt.DisplayRole:
            return section
        return None
 
if __name__ == "__main__":
    app = QApplication(sys.argv) 
    w = MyWindow() 
    w.resize(900, 600)
    w.show() 
    sys.exit(app.exec_())

import sys 
from PyQt4.QtCore import * 
from PyQt4.QtGui import * 
 
 
class MyWindow(QWidget): 
    def __init__(self, *args): 
        QWidget.__init__(self, *args) 

        # create table
        table = QTableView()
        tm = MyTableModel(self) 
        table.setModel(tm)
        
        # layout
        layout = QVBoxLayout()
        layout.addWidget(table) 
        self.setLayout(layout) 
 
class MyTableModel(QAbstractTableModel):  
    def rowCount(self, parent): 
        return 100000000
 
    def columnCount(self, parent): 
        return 10
 
    def data(self, index, role): 
        if not index.isValid(): 
            return QVariant() 
        elif role != Qt.DisplayRole: 
            return None
        return str((index.row(), index.column()))

    def headerData(self, section, orientation, role):
        print "headerData", section
        if role == Qt.DisplayRole:
            return section
        return None

if __name__ == "__main__":
    app = QApplication(sys.argv) 
    w = MyWindow() 
    w.resize(900, 600)
    w.show() 
    sys.exit(app.exec_())

To copy to clipboard, switch view to plain text mode 

Anyone can point me out what I'm doing wrong? is it a bug? The test has been run on windows 7 (64 bit), PyQt 4.11.
Thanks in advance!

[jefftee]

Does a user interface with 100 million rows really sound like a good idea?

[Gad82]

In the years of big data the answer is probably yes

[jefftee]

In the years of big data the answer is probably yes

Good luck with that then...

Edit: Regarding your original question, are you by chance using QHeaderView::ResizeToContents?

[Gad82]

No, as shown in the test script I only set up a simple model and then put it into the QTableView....

[d_stranz]

In the years of big data the answer is probably yes

In the years of big data, no one actually looks at the big data. If they look at it at all, they look at visualizations that condense the data into something a human viewer can comprehend. The individual items are for the most part, nearly irrelevant. No one, absolutely no one, will look at anything more than the first page or two of a table with 100 million entries, so why even bother with such a thing?

If you feel you must do this, then you should implement a model which retrieves only what the user is currently viewing on the screen and not attempt to build either a model or a view that contains all of it.

[jefftee]

Looks like you are not differentiating between horizontal and vertical headers. If you don't want header data for rows, then you should be returning QVariant() when the orientation is not horizontal.

I don't code in Python, if None is not the same as QVariant(), then you should change your code to return QVariant() instead of None. Just guessing but perhaps your headerData is being repeatedly called because you're not returning QVariant() in the case where you have no data to return.

[lotfimira]

Even when returning QVariant() in header data the QTableView header still tries to allocate a vector the size of the row count.
Is it possible to have hundred millions of rows in the model without an out of memory crash in the table?

[bpepers]

I hit the same problem and when I looked into it, the problem is QHeaderView. It internally calculates the offset of each row in pixels but it uses an int so limited to 2 billion and if you overflow it you end up negative. With 100M rows if the rows are over 20 pixels or so, which on a 4K monitor in particular is very likely, then these pixel offsets overflow and the code goes haywire causing very long loops that make it appear to have hung.

I wish there was a QAbstractHeaderView that things like QTableView used with QHeaderView just an implementation we could replace with something better. But when I looked into the code QHeaderView has a very intimate knowledge and tie into the private data of other views and models so there seems to be very poor separation of concerns in the code and it would be very difficult to replace it.

Off to look for some other way to show 100M rows!

[d_stranz]

Instead of making a model that reports it has 100M rows (which is what the header is relying on in its calculations), devise a model that can page through a subset of your data and report fewer rows so the header view stays well-behaved.

Thinking that your table must accommodate all 100M rows is an example of "confusing the map for the territory". Your underlying data structure (the territory) might contain 100M entries, but your view into it (the map) does not have to.

One possible solution is to implement a sliding window onto your underlying data that contains say, twice as many rows as can be viewed onscreen. When you scroll up or down, you slide the window along, moving the first and last index rows accordingly. This should also significantly improve your UI response time, since the view s only have to base their calculations on a few hundred rows instead of thousands or millions. This could probably be done with a proxy model. If having an accurate row number on the vertical header is important, you can re-implement your model's headerData() method to return a string containing the actual row number from the underlying data.

[jhosek]

I thought the default QTableView already did something like this?

[d_stranz]

I thought the default QTableView already did something like this?

It does, as far as displaying rows is concerned. But apparently the header view layout takes the entire model into account when calculating row location, at least according to @bpepers' post. I am guessing that this is how it determines whether a row is visible on screen or not.