PDA

View Full Version : QTableView and Large data sets



KenJustKen
24th January 2016, 20:27
I am in the process of rewritting an app that does the following. Pulls in tens of millions of 64 Byte records from a data acquisition card and then display the data in a table with about 4-6 columns. I will not need to sort the data, I just want to display the data. The previous owner was trying to just load the data into a QAbstractItemModel and then load this directly into a QTableView. However, as several posts here and other places point out that this is really really slow with this size of dataset. Some suggest using a "proxyModel or Custom Model" to only pull the part of the data set from the file into the QTableView using the location of the current scroll bar as the "index". Where can I find a good example of this or what would you suggest?

Ken

anda_skoa
24th January 2016, 23:09
What do you mean with "load the data into a QAbstractItemModel"?
Usually the model is only the interface to the data.

How does your application store and access the data?

Cheers,
_

KenJustKen
25th January 2016, 03:12
The data is stored as 32 byte binary records on the disk after being read from the acquisition card.

anda_skoa
25th January 2016, 10:16
If the data format allows you to determine file position based on the index of a record, then you could create a model that access the data in the file directly.

Potentially using a memory mapped file and/or caching some records in memory.

E.g. some header offset aside, if the first record starts at 0*32 and the second starts at 1*32 and so on, then you can simply determine the model's row count by fileSize/32.
And the model's data can simply seek to index.row() * 32 to retrieve the record.

Cheers,
_