Results 1 to 6 of 6

Thread: What is the best way to use QDataStream or QFile

  1. #1
    Join Date
    Dec 2006
    Posts
    426
    Thanks
    8
    Thanked 18 Times in 17 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default What is the best way to use QDataStream or QFile

    Hi,

    I have a large binary file consisting 3000x3000x2000 floating data points (around 67G in size). For each interval of 2000, I need to read one float value. So in total I need to read 3000x3000 points. What is the fastest way to read it?

    I did the following way and it takes a decade to read it out.
    Qt Code:
    1. QFile file( ... );
    2. QDataStream stream( &file );
    3.  
    4. qint64 pos( 0);
    5. char data[ 4 ];
    6. for ( int ix = 0; ix < 3000; ix++ ) {
    7. for ( int iy = 0; iy < 3000; iy++ ) {
    8. stream.device()->seek( pos );
    9. stream.readRawData( data, 4 );
    10. ...
    11. pos += 2000;
    12. }
    13. }
    To copy to clipboard, switch view to plain text mode 

  2. #2
    Join Date
    Jun 2011
    Posts
    56
    Thanks
    7
    Qt products
    Qt4

    Default Re: What is the best way to use QDataStream or QFile

    Well, if the Qt built-in approaches don't work, try an external library (like boost), that supports memory-mapped files and try that. I've read that sometimes, reading goes faster that way.

  3. #3
    Join Date
    Feb 2011
    Location
    Romania
    Posts
    53
    Thanks
    1
    Thanked 11 Times in 9 Posts
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: What is the best way to use QDataStream or QFile

    Quote Originally Posted by ugluk View Post
    Well, if the Qt built-in approaches don't work, try an external library (like boost), that supports memory-mapped files and try that. I've read that sometimes, reading goes faster that way.
    Qt framework already have memory mapping file support.

  4. #4
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: What is the best way to use QDataStream or QFile

    I somehow doubt he'll be able to map 67 gigabytes to memory. I'd say that reading 9 million floats (which gives at least 36 megabytes of data) just has to take time, especially if one reads it value at a time. A trivial optimization is to read all values at once and then iterate over them.

    Ordering of data in the file is poor as well, skipping 2kB of data (why 2 and not 8? 2k of 4 byte values is 8kB) before each read significantly limits the disk cache hit ratio, even for a 32MB cache it will give you lots of cache misses. Nothing will change that, even mapping the file to memory (unless you have enough physical RAM to actually be able to map the whole file at once). If you can't change the file structure then I suggest you invest in a faster disk (SSD?) or more RAM (96GB should do).
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  5. #5
    Join Date
    Dec 2006
    Posts
    426
    Thanks
    8
    Thanked 18 Times in 17 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: What is the best way to use QDataStream or QFile

    Quote Originally Posted by wysota View Post
    I somehow doubt he'll be able to map 67 gigabytes to memory. I'd say that reading 9 million floats (which gives at least 36 megabytes of data) just has to take time, especially if one reads it value at a time. A trivial optimization is to read all values at once and then iterate over them.

    Ordering of data in the file is poor as well, skipping 2kB of data (why 2 and not 8? 2k of 4 byte values is 8kB) before each read significantly limits the disk cache hit ratio, even for a 32MB cache it will give you lots of cache misses. Nothing will change that, even mapping the file to memory (unless you have enough physical RAM to actually be able to map the whole file at once). If you can't change the file structure then I suggest you invest in a faster disk (SSD?) or more RAM (96GB should do).
    You are absolutely right. It should be "pos += 8000".

    I am restructuring the data so it will read data block by block to maximize the disk cache. What is the best block size I should use? Is there a way to query this parameter in the program?

    Many thanks!

  6. #6
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: What is the best way to use QDataStream or QFile

    If you are restructuring the file so that each 3000x3000 block is contiguous in the file then you want to read 9000000 * sizeof(float) bytes at a time. Provided you can allocate the required 36000000 byte (or bigger) block of memory to receive the result, just ask the operating system to read it and it will do it as fast as is possible. If you break the read into 3000 reads of 3000 floats, or something else, then you add overhead for little gain.

Similar Threads

  1. Replies: 4
    Last Post: 9th May 2011, 10:52
  2. QFile &QFile::operator= is private
    By Fallen_ in forum Newbie
    Replies: 1
    Last Post: 15th March 2011, 16:08
  3. QFile - QDataStream read and write each character
    By nhs_0702 in forum Qt Programming
    Replies: 2
    Last Post: 4th May 2010, 20:03
  4. QFile, QDataStream reading binary data
    By yren in forum Qt Programming
    Replies: 1
    Last Post: 15th April 2009, 07:34
  5. QDataStream
    By TheKedge in forum Qt Programming
    Replies: 1
    Last Post: 23rd August 2006, 15:40

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.