Processing Large Text File in Qt

[dan1973]

Hi,

I am reading a Large txt file and processing some value. The program suddenly exits at about 4000000 record approximately. The file size is 7228916 lines or records. Here is my code snippet:

QTextStream in(&fEngg);
            slEnggFileRecs.clear();
 
            etmrOldPlot.start();
            iFileIndx = 0;
            slXVals1.clear();
            while(!in.atEnd()) {
                slEnggFileRecs.append(in.readLine());
                if(iFileIndx > 0) {
                    slXVals1.append(slEnggFileRecs.at(iFileIndx).split(',').at(1));
                }
                ++iFileIndx;
                qInfo() << "iFileIndx:" << iFileIndx;
            }
            qInfo() << "RunTab: Read Engg file:" << etmrOldPlot.elapsed();
            qInfo() << "Total Recs in Engg File:" << slEnggFileRecs.length() - 1 << " slXVals1.len:" << slXVals1.length();

QTextStream in(&fEngg);
            slEnggFileRecs.clear();

            etmrOldPlot.start();
            iFileIndx = 0;
            slXVals1.clear();
            while(!in.atEnd()) {
                slEnggFileRecs.append(in.readLine());
                if(iFileIndx > 0) {
                    slXVals1.append(slEnggFileRecs.at(iFileIndx).split(',').at(1));
                }
                ++iFileIndx;
                qInfo() << "iFileIndx:" << iFileIndx;
            }
            qInfo() << "RunTab: Read Engg file:" << etmrOldPlot.elapsed();
            qInfo() << "Total Recs in Engg File:" << slEnggFileRecs.length() - 1 << " slXVals1.len:" << slXVals1.length();

To copy to clipboard, switch view to plain text mode 

Is there any solution for this? Thanks in adv.

Dan

[d_stranz]

I don't see any obvious bug in this, except for possibly one:

slXVals1.append(slEnggFileRecs.at(iFileIndx).split(',').at(1));

slXVals1.append(slEnggFileRecs.at(iFileIndx).split(',').at(1));

To copy to clipboard, switch view to plain text mode 

If the line is blank or has only one field in it, "at(1)" will likely crash.

But some questions (which you might have good reasons for doing):

1 - Why do you store the entire contents of the file in "slEnggFileRecs"? Do you need to go back and process the strings after reading them in? If not, then you're just creating excess overhead.

qInfo() << "RunTab: Read Engg file:" << etmrOldPlot.elapsed();

qInfo() << "RunTab: Read Engg file:" << etmrOldPlot.elapsed();

To copy to clipboard, switch view to plain text mode 

2 - Storing the line in "slEnggFileRecs" and then retrieving it to split it is inefficient. Read it into a QString and if you have to, save the QString in "slEnggFileRecs" but also pass it to the split() method. Using a temporary QString would also allow you to easily check to see if it is empty. Storing the result of the split in a temporary QStringList would allow you to determine that it had enough pieces. Try it both ways and see if your elapsed time improves.

3 - "slXVals1" looks like it could be an X coordinate or at least a numerical value. Why store it as a string? Why not convert it immediately to a number and store that?

qInfo() << "Total Recs in Engg File:" << slEnggFileRecs.length() - 1 << " slXVals1.len:" << slXVals1.length();

qInfo() << "Total Recs in Engg File:" << slEnggFileRecs.length() - 1 << " slXVals1.len:" << slXVals1.length();

To copy to clipboard, switch view to plain text mode 

4 - You have already counted the number of lines and x values in "iFileIndx". If you are storing the records only so you can get the count, then there is no need to do that since you have this variable.