Results 1 to 4 of 4

Thread: QTableView and Large data sets

  1. #1
    Join Date
    Nov 2011
    Posts
    30
    Qt products
    Qt3
    Platforms
    MacOS X

    Default QTableView and Large data sets

    I am in the process of rewritting an app that does the following. Pulls in tens of millions of 64 Byte records from a data acquisition card and then display the data in a table with about 4-6 columns. I will not need to sort the data, I just want to display the data. The previous owner was trying to just load the data into a QAbstractItemModel and then load this directly into a QTableView. However, as several posts here and other places point out that this is really really slow with this size of dataset. Some suggest using a "proxyModel or Custom Model" to only pull the part of the data set from the file into the QTableView using the location of the current scroll bar as the "index". Where can I find a good example of this or what would you suggest?

    Ken
    Last edited by KenJustKen; 24th January 2016 at 20:35.

  2. #2
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,419
    Thanks
    37
    Thanked 1,545 Times in 1,495 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: QTableView and Large data sets

    What do you mean with "load the data into a QAbstractItemModel"?
    Usually the model is only the interface to the data.

    How does your application store and access the data?

    Cheers,
    _

  3. #3
    Join Date
    Nov 2011
    Posts
    30
    Qt products
    Qt3
    Platforms
    MacOS X

    Default Re: QTableView and Large data sets

    The data is stored as 32 byte binary records on the disk after being read from the acquisition card.

  4. #4
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,419
    Thanks
    37
    Thanked 1,545 Times in 1,495 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: QTableView and Large data sets

    If the data format allows you to determine file position based on the index of a record, then you could create a model that access the data in the file directly.

    Potentially using a memory mapped file and/or caching some records in memory.

    E.g. some header offset aside, if the first record starts at 0*32 and the second starts at 1*32 and so on, then you can simply determine the model's row count by fileSize/32.
    And the model's data can simply seek to index.row() * 32 to retrieve the record.

    Cheers,
    _

Similar Threads

  1. Replies: 5
    Last Post: 2nd July 2012, 20:49
  2. Need help with QTableView/QTableWidget and large data
    By aguayro in forum Qt Programming
    Replies: 2
    Last Post: 29th June 2012, 12:09
  3. Replies: 1
    Last Post: 23rd September 2010, 05:45
  4. Replies: 1
    Last Post: 23rd December 2008, 15:42
  5. Creating images from sets of data.
    By maverick_pol in forum Qt Programming
    Replies: 5
    Last Post: 26th February 2008, 09:25

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.