reading a binary file full of floats and/or doubles

[OzQTNoob]

Thanks for the replies guys/gals.

I will try and avoid the C/C++ route if I can as I want to keep everything Qt if I can. I will investigate the QFile option more thoroughly. As long as I can set my read position aka seek then I should be alright (QIODevice). I did go over my code last night and cleaned it up somewhat. Being the noob that I am I still need to get used to not having a bunch of really convenient data reading/writing tools as I had in IDL (Look for RSI ENVI if you are curious).

Cheers
Oz

[ChrisW67]

If your reading machine is little-endian (most are) then the problem should be trivial:

const int vectSize = 531;
const int blockSize = vectSize * sizeof(float);
QVector<float> vect(vectSize, 0.0f);
 
int desiredBlock = 0;   // zero based, e.g 0..49999
QFile file("data");
if (file.open(QIODevice::ReadOnly)) {
    if (file.seek(desiredBlock * blockSize)) {
        qint64 bytes = file.read(reinterpret_cast<char*>(vect.data()), blockSize);
        if (bytes != blockSize)
            qFatal("Oops!");
 
        qDebug() << vect;
    }
}

const int vectSize = 531;
const int blockSize = vectSize * sizeof(float);
QVector<float> vect(vectSize, 0.0f);

int desiredBlock = 0;   // zero based, e.g 0..49999
QFile file("data");
if (file.open(QIODevice::ReadOnly)) {
    if (file.seek(desiredBlock * blockSize)) {
        qint64 bytes = file.read(reinterpret_cast<char*>(vect.data()), blockSize);
        if (bytes != blockSize)
            qFatal("Oops!");

        qDebug() << vect;
    }
}

To copy to clipboard, switch view to plain text mode 

Effectively just dropping the raw bytes into the internal buffer of the QVector.

You can use:

#if Q_BYTE_ORDER == Q_BIG_ENDIAN
  // byte swapping code
#endif

#if Q_BYTE_ORDER == Q_BIG_ENDIAN
  // byte swapping code
#endif

To copy to clipboard, switch view to plain text mode 

at line 12 if you need portability.

[wysota]

Qt is C++, so everywhere you use Qt, C++ mechanisms will work as well.

As for convenient tools -- I don't know what conveniences you need to read four bytes from a file. Your current code is not very optimal, for instance performing seek in every iteration is a waste of time since the file pointer is already positioned in the right place.

The fastest way to read, say... 1000 floats is the following:

QFile file(...);
file.open(...);
file.seek(...);
float array[1000];
QByteArray ba = file.read(1000*sizeof(float)); // reading 4000 bytes is faster than reading 4 bytes 1000 times (even though the disk caches the data)
memcpy(&array, ba.constData(), ba.size());

QFile file(...);
file.open(...);
file.seek(...);
float array[1000];
QByteArray ba = file.read(1000*sizeof(float)); // reading 4000 bytes is faster than reading 4 bytes 1000 times (even though the disk caches the data)
memcpy(&array, ba.constData(), ba.size());

To copy to clipboard, switch view to plain text mode 

Of course all that provided the interpretation of float on your machine is the same as the one in the file. Otherwise you'll need to iterate over the byte array and unmangle bytes properly.

[OzQTNoob]

Hi Chris and Wysota,

both methods worked a treat and taught me a little bit more about C++ along the way so thanks for that. All of my work and code is on a PC and will more than likely stay that way. Occasionally though I may get an image file (talking remote sensing image here e.g. spectral imagery) that has come from a BigEndian machine. In this case I can always identify it from its header file (An ascii file). If I am going to swap the endian of the data (assuming that happens) then I will have a look at methods for that (I am sure there are plenty of existing examples around).

In relation to the 2 code snippets you guys posted is their any reason to adopt one over the other? And as an aside, if I don't know the actual array sizes prior to runtime (they can actually vary a lot depending on the instrument used to collect the spectra) I assume that I can replace the expressions that have used the const terms etc with Qt Dynamic equivalents? e.g. use resize with the QVector and/or QByteArray. I will try this with the code snippets I got from you guys and see what happens

Cheers
Oz

[ChrisW67]

What is producing your spectra? ... or would you have to kill me I you told me

[OzQTNoob]

Hi Chris,

no killing needed. The spectra can come from a number of sources such as an ASD field spectrometer (400-2500nm spectral region) or from airborne hyperspectral imagers such as HyMap (an Australian sensor) to name but 2. The datasets that we work with tend to come with an ENVI header file and almost always in the form of a spectral library or a multiband image file (has been adopted by a very large portion of the remote sensing community). Most of my work, although not all of it, is about getting at what minerals are present in a spectral dataset via the spectral absorption features in a given spectrum, and/or correcting the airborne imagery for atmospheric effects etc.

I have written lots of code in IDL to process the various datasets (I work for CSIRO) but the problem that I have with IDL is that it requires licensing and therefore we have to keep coming up money to fund our licences, and ultimately IDL is an interpreted language that defers back to C anyway, the for loops in IDL are painfully slow if you have to use them. Plus, with Qt it still lets me use UI's and is a slightly easier way for me to ease into C++ programming as well

[ChrisW67]

Cool. Only ask because there is much spectra processing in astronomy (one of my quals).

[OzQTNoob]

Indeed, in fact there are whole IDL libraries dedicated to astronomy software and routines, idlastro.gsfc.nasa.gov to name but one. Some of my best routines have been lifted from astro code
"To steal the work of one is plagiarism, to steal the work of many is research"