Read huge files effectively

[MrAnderson1983]

Hello guys,

im trying to read huge *.dat files (~ 2gb) containing a tag i would like to extract (SoftwareVersionNumber). This piece of code works fine, but the processing time is terrible.

//Retrieves SW Version of a *.dat file
QString ScanFilesWorker::GetSWVersion(QString strFullPath)
{
    QByteArray strTag1 = "<m_SoftwareVersionNumber_aui8>";
    QByteArray strTag2 = "</m_SoftwareVersionNumber_aui8>";
 
    QFile DatFile(strFullPath);
    DatFile.open(QIODevice::ReadOnly);
 
    while (!DatFile.atEnd())
    {
        QByteArray DatContent = DatFile.readLine();
        if (DatContent.contains(strTag1))
        {
            qint64 iFirst = DatContent.lastIndexOf(strTag1);
            qint64 iLast = DatContent.lastIndexOf(strTag2);
 
            QString SW_Version = DatContent.mid(iFirst+strTag1.length(),iLast-iFirst-strTag2.length()+1);
            qDebug() << SW_Version;
 
            return SW_Version;
        }
    }
 
    return "No SW Version found!";
 
}

//Retrieves SW Version of a *.dat file
QString ScanFilesWorker::GetSWVersion(QString strFullPath)
{
    QByteArray strTag1 = "<m_SoftwareVersionNumber_aui8>";
    QByteArray strTag2 = "</m_SoftwareVersionNumber_aui8>";

    QFile DatFile(strFullPath);
    DatFile.open(QIODevice::ReadOnly);

    while (!DatFile.atEnd())
    {
        QByteArray DatContent = DatFile.readLine();
        if (DatContent.contains(strTag1))
        {
            qint64 iFirst = DatContent.lastIndexOf(strTag1);
            qint64 iLast = DatContent.lastIndexOf(strTag2);

            QString SW_Version = DatContent.mid(iFirst+strTag1.length(),iLast-iFirst-strTag2.length()+1);
            qDebug() << SW_Version;

            return SW_Version;
        }
    }

    return "No SW Version found!";

}

To copy to clipboard, switch view to plain text mode 

DatFile.readAll could be an option, but this function fails for files larger then 600 mb. Maybe it is possible to read through the file from the end to the beginning (SoftwareVersionNumber is located at the end of the file). I also tried QXmlReader library, but it failed as well (*.dat files contain some raw byte data).

Thank you for your help.

[Lesiok]

Use QFile::seek and QFile::size methods to set position in a file.

[Infinity]

I think XML is not the right choice when dealing with such large files. XML is also not the right choice when dealing with raw data. Additionally it is not the best idea to locate meta information at the end of the file.

I you want the meta information to be placed at the end of the file, store a size prefix before the binary data or the start offset of your meta data at the beginning of the file. Then use the QFile::seek method as already recommended.

[MrAnderson1983]

Thank a lot for your help. The seek method worked pretty good.