Results 1 to 4 of 4

Thread: Read huge files effectively

  1. #1
    Join Date
    Apr 2014
    Posts
    7
    Thanks
    2
    Qt products
    Qt5

    Default Read huge files effectively

    Hello guys,

    im trying to read huge *.dat files (~ 2gb) containing a tag i would like to extract (SoftwareVersionNumber). This piece of code works fine, but the processing time is terrible.

    Qt Code:
    1. //Retrieves SW Version of a *.dat file
    2. QString ScanFilesWorker::GetSWVersion(QString strFullPath)
    3. {
    4. QByteArray strTag1 = "<m_SoftwareVersionNumber_aui8>";
    5. QByteArray strTag2 = "</m_SoftwareVersionNumber_aui8>";
    6.  
    7. QFile DatFile(strFullPath);
    8. DatFile.open(QIODevice::ReadOnly);
    9.  
    10. while (!DatFile.atEnd())
    11. {
    12. QByteArray DatContent = DatFile.readLine();
    13. if (DatContent.contains(strTag1))
    14. {
    15. qint64 iFirst = DatContent.lastIndexOf(strTag1);
    16. qint64 iLast = DatContent.lastIndexOf(strTag2);
    17.  
    18. QString SW_Version = DatContent.mid(iFirst+strTag1.length(),iLast-iFirst-strTag2.length()+1);
    19. qDebug() << SW_Version;
    20.  
    21. return SW_Version;
    22. }
    23. }
    24.  
    25. return "No SW Version found!";
    26.  
    27. }
    To copy to clipboard, switch view to plain text mode 

    DatFile.readAll could be an option, but this function fails for files larger then 600 mb. Maybe it is possible to read through the file from the end to the beginning (SoftwareVersionNumber is located at the end of the file). I also tried QXmlReader library, but it failed as well (*.dat files contain some raw byte data).

    Thank you for your help.

  2. #2
    Join Date
    Mar 2008
    Location
    Kraków, Poland
    Posts
    1,477
    Thanked 274 Times in 269 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Read huge files effectively

    Use QFile::seek and QFile::size methods to set position in a file.

  3. The following user says thank you to Lesiok for this useful post:

    MrAnderson1983 (30th April 2014)

  4. #3
    Join Date
    Oct 2012
    Posts
    132
    Thanks
    10
    Thanked 21 Times in 21 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows Android

    Default Re: Read huge files effectively

    I think XML is not the right choice when dealing with such large files. XML is also not the right choice when dealing with raw data. Additionally it is not the best idea to locate meta information at the end of the file.

    I you want the meta information to be placed at the end of the file, store a size prefix before the binary data or the start offset of your meta data at the beginning of the file. Then use the QFile::seek method as already recommended.

  5. The following user says thank you to Infinity for this useful post:

    MrAnderson1983 (30th April 2014)

  6. #4
    Join Date
    Apr 2014
    Posts
    7
    Thanks
    2
    Qt products
    Qt5

    Default Re: Read huge files effectively

    Thank a lot for your help. The seek method worked pretty good.

Similar Threads

  1. read a huge file over 100GB
    By jesse_mark in forum Qt Programming
    Replies: 3
    Last Post: 23rd April 2013, 23:11
  2. QWebView print() generates huge PDF files on Windows
    By brush.tyler in forum Qt Programming
    Replies: 2
    Last Post: 26th September 2012, 14:01
  3. Huge Mac .dmg files, can anything be done to make them smaller?
    By colin207 in forum Installation and Deployment
    Replies: 3
    Last Post: 10th February 2011, 08:06
  4. Replies: 12
    Last Post: 17th June 2009, 05:34
  5. Huge error list from qt files
    By Raccoon29 in forum Qt Programming
    Replies: 4
    Last Post: 8th September 2008, 15:22

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.