fastest way to read large files (with mixed content)

[timmu]

I have large tabular text files that contain both numbers and letters. I want to use the numbers. What is the fastest way to read these data.
Currently I'm doing

QFile genofile(Genoname); if(!genofile.open(QIODevice::ReadOnly | QIODevice::Text)) {exit(1);}
QTextStream geno_stream(&genofile);
 
     while(!geno_stream.atEnd())
     {
     QString genoRow = geno_stream.readLine();
     QStringList row = genoRow.split("\t");
 
         for(int i=0; i<row.size(); i++)
         {
           double value;
           if(row.at(i)!="NA") value=row.at(i).toDouble();
           else value =-9.0
         }
     }

QFile genofile(Genoname); if(!genofile.open(QIODevice::ReadOnly | QIODevice::Text)) {exit(1);}
QTextStream geno_stream(&genofile);

     while(!geno_stream.atEnd())
     {
     QString genoRow = geno_stream.readLine();
     QStringList row = genoRow.split("\t");
        
         for(int i=0; i<row.size(); i++)
         {
           double value;
           if(row.at(i)!="NA") value=row.at(i).toDouble();
           else value =-9.0
         }
     }

To copy to clipboard, switch view to plain text mode 

I need the process to be maximally fast. Is this a fast way?

What would be the best way when the data don't contain letters (all are numbers)?

[high_flyer]

Why not just use QTextStream?

[timmu]

Is QTextStream faster?

Basically, I'd like to know how fast (in relative terms) is QString to double conversation by .toDouble()?
Or is this conversion something that needs to be avoided for the sake of speed?

[Uwe]

String to double conversions are expensive - don't do it, when you need performance. The fastest way might be to store the values as binary data and to use QFile::map(). Then you don't need to read the file at all. ( See mmap(). )

HTH,
Uwe

[high_flyer]

Is QTextStream faster?

Since this is just two or so lines of code, why not just test and see?

Basically, I'd like to know how fast (in relative terms) is QString to double conversation by .toDouble()?

The performance is not only in the string conversion, but also in the iteration through the text and extracting the correct substrings to be converted to double.
So you will have to make a more complex test case to measure it with any practical results.

@Uwe: but if you have binary data - how do you get to the correct fields?
You will have to implement what QTextStrem is doing (more or less).
Probably I just don't follow on what you mean...

[Uwe]

@Uwe: but if you have binary data - how do you get to the correct fields?
You will have to implement what QTextStrem is doing (more or less).

No, when the data in the file is aligned like in memory( usually dumped from a C/C++ array ) you can simply map it to an address ( a pointer for the same data structure where the data has been dumped before ) and access it like regular memory using C/C++ classes/structs. No reading or parsing of the file is necessary. F.e. when the file is simply an array of doubles you can cast the address, where the file was mapped to, to a double * - or QPointF * pointer and access it like an usual array.

See http://en.wikipedia.org/wiki/Memory-mapped_file

Of course such an implementation is not cross platform ( f.e. big endian, little endian problems, compiler incompatibilities ). But storing doubles as strings has rounding errors and the conversion is expensive.

Uwe

[high_flyer]

Yes, all of that is clear - but it means you need to know the format of the data you are looking for, and its not what I understood is the case.

I have large tabular text files that contain both numbers and letters.

If it is (that is, you know where to look for numbers and where not), then ok, this is a good solution.
But if all you know you that you have fields which *might* be what you are looking for, then you need to implement some logic to parse it.

[timmu]

Thank you, High-flyer and Uwe, for this useful discussion. Basically I have 2 situations:
1. In one file I don't know if an entry is a word or a number. I only need to use an entry (and convert it to double) if it is a number. In this situation QTextStream is perhaps not a bad choice.
2. Another situation is where I know all values are doubles. In this case I care only that the process is maximally fast. What is the best method to use then as far as what Qt offers?

My third questions is this: Is string->float conversion faster than string->double.

Thanks!

[high_flyer]

1. I would use QTextStream, yes.
2. Mapping as Uwe suggested.

I would be surprised if there was any major difference between string to float/double - but I didn't look in to that.