Huge tableview low performance

[semoser]

Hi all!

I need to display very large table with more then 100000x100000 cells
I'am reimplement QAbstractItemModel to represent my internal data in each cell of standard QTableView

But created table view is too slow for me ...
I was testing simple item model what store 100000x100000 data structure and returns empty QVariant() for each data cell ... but even so tableview repaints too slow ....

why?

Or i have to reimplemen QAbstractItemView for better perfomance ... :eek:

Qt version used: 4.1.1

[wysota]

Correct me if I'm wrong but 100000 x 100000 items equals E+10 (10G) items. If each item occupies let's say 10 bytes of memory (it probably occupies more), this gives... wow.... Do you have 100GB of memory in your system? Maybe you should try to limit the number of items a little? Anyway, I guess you should try to see what is the bottleneck of your data model. You can probably introduce some caching and thus increase performance.

[semoser]

I am using my own ItemModel where each item presents as row and it's data in shows in cells. Cells count depends from item data size. Each byte of item data in one cell.

Item data size ~ 10000
Items count = rows count ~ 1000000

Im reading items from file on the fly and store in memory only vivible part of the table.
It slows down perfomance, but not so critical as painting and scrolling

As i wrote earlier. I tested QTableView with empty ItemModel, where data() method always returns empty QVariant, and table scrolling was not so fast, as, for example, program from QT3 special example with reimplemented QTable millionXmillion

I suppose that problem in new complex Model/View architecture ...

Small example: 662
Compare prefofmance with .... MS Excel for example :confused:

[wysota]

I suppose that problem in new complex Model/View architecture ...

I suppose you are trying to turn the architecture upside down, so don't expect it to work properly, it was designed for something completely different. If you only keep those items in the model which are visible in the view, then you don't have a model-view architecture any more. The whole point for the pattern is that the model doesn't care what does the view display and the view doesn't care how the model gets its data. If you only want to store the amount of data which is visible, then simply subclass QWidget, keep a QVector< QVector< QVariant> > with visible items and paint them on the widget directly.

If you insist on keeping the model-view framework, then make a small model and implement some "scrollLeft", "scrollRight", "scrollDown" and "scrollUp" slots (which have to be connected to proper signals of the view which you probably have to subclass then) and in those slots refill your model with new data. But expect it to be kind of slow. Anyway this won't have much in common with the MVC pattern anymore.

[semoser]

I don't keep any items in the model. Data stored in the file (it may be very large file) in some internal format. Model only obtain data offset in that file from QModelIndex row and column and returns data for this row and column.

I think it is correspond to MVC pattern, isn't it?

[wysota]

I don't keep any items in the model.
That doesn't matter, QDirModel doesn't store any data as well. The model is not for storing data (physically) but for providing a generic interface for outside world.


Data stored in the file (it may be very large file) in some internal format. Model only obtain data offset in that file from QModelIndex row and column and returns data for this row and column.
That's fine - in theory. In practice I believe it is slower than WindowsXP running on 128MB of RAM ;) Imagine you get two subsequent calls for the model:
1. index(0, 0)
2. index(rowCount()-1, 0)

This way you have to go through the whole file (logically, in reality only the right block needs to be accessed if records have constant size) before reading the data.


I think it is correspond to MVC pattern, isn't it?

Yes, it does. But you said you store only that part of data which is visible. And that doesn't correspond to MVC. Especially that now you say you don't store any data :)

How did you implement rowCount()?

Anyway, the problem with itemviews is that sometimes it like to go through the whole model reading data like crazy. Don't ask me why... someone has recently asked about it, but maybe he was wrong and it is not true that the whole model is scanned as this would seem odd, because the view knows exactly which items it currently displays. The problem might (should?) only occur when using a model proxy as the proxy would have to scan the complete model now and then.

I suggest you at least introduce some caching or indexing and try to run your application under a profiler to see what slows it down the most.

About Excel - it stores all its data in memory or some disk buffers (who knows?), it doesn't scan through the .xls file all the time. If that was the case, you couldn't reference a cell which is currently not visible or it would be slow like hell or cells would have to have constant sizes (as it wouldn't be possible to easily (quickly) determine the offset where particular data is stored).

[semoser]

Imagine you get two subsequent calls for the model:
1. index(0, 0)
2. index(rowCount()-1, 0)


When model created i am seek through whole file and create QHash of all items offsets. key = row, value = qint64(offset)
And when i need to get item data i am seek directly to interesting pos in file and read it.



How did you implement rowCount()?


rowCount equals to hash size



you said you store only that part of data which is visible. And that doesn't correspond to MVC. Especially that now you say you don't store any data :)


When i told model storing data (in "Million example") i meen it provide this amount of data. See the attached file in my second post.
In that example no any data readed by model. Only QTableView works for painting empty cells. And it works quite slow then i am try to scroll it up/down very quickly



I suggest you at least introduce some caching or indexing and try to run your application under a profiler to see what slows it down the most.


Then i debug my application most time spended to repaint QTableView ...

P.S. Jal' ja ne mogu tolkom ob`jazsnit' to chto mne nujno. My english is too bad ... :(

[wysota]

When model created i am seek through whole file and create QHash of all items offsets. key = row, value = qint64(offset)
And when i need to get item data i am seek directly to interesting pos in file and read it.

rowCount equals to hash size
So your model is read-only, right?


Then i debug my application most time spended to repaint QTableView ...
The question is - what slows down the application - the view, the model or the disk operations. Simple debugging won't give you that information.



P.S. Jal' ja ne mogu tolkom ob`jazsnit' to chto mne nujno. My english is too bad ... :(

If you want to write some russian text, use russian letters. It's hard to read russian text written in latin alphabet ;) Of course I don't encourage you to do that - it is an english based site.

BTW. Your english is ok and you described it properly so I understand your problem - it's not the matter of language.

Cheers!

[semoser]

Yes model is read only. I am need only to display data.
I will try to explain main task i have to do:

I have a file with network traffic. For example in PCAP format
I have to display packets in grid where:
row - one packet
col - one byte of packet data in Hex, Dec, Ascii, etc. format ()

I understand that converting each byte to printable form is slow operation, and reading data from disk can be slow too.

I want to understand does it make sense to reimplement QAbstractItemView and paintEvent() in it subclass to improve simple drawing off my data. Or it can be done only by changing my ItemModel behaviour and manipulating with standard QTableView?

[wysota]

Don't touch the paint event. The widget is too complex to change its paint event.
Maybe you could get rid of the model view framework if it's just about display and use some simpler mechanism, like QTextDocument with a table.