Hi,
I'm currently trying to read a gzipped file. It contains an ASCII table, so I would like to read it line-by-line in order to get the values from the table. For this I wrote a small class CompressedFile, which is derived from QIODevice.
The overloaded readLineData function looks like this:
qint64 CompressedFile::readLineData(char *data, qint64 maxlen)
{
qint64 len = 0;
while(len < maxlen) {
if (_bufPos >= _bufSize) {
fillBuffer();
if (_bufSize == 0) return 0;
}
data[len++] = _buffer[_bufPos++];
if (data[len-1] == '\n') {
data[len-1] = '\0';
break;
}
}
return len;
}
qint64 CompressedFile::readLineData(char *data, qint64 maxlen)
{
qint64 len = 0;
while(len < maxlen) {
if (_bufPos >= _bufSize) {
fillBuffer();
if (_bufSize == 0) return 0;
}
data[len++] = _buffer[_bufPos++];
if (data[len-1] == '\n') {
data[len-1] = '\0';
break;
}
}
return len;
}
To copy to clipboard, switch view to plain text mode
Since I got problems with qUncompress, I'm using zlib directly, so fillBuffer() looks like this:
void CompressedFile::fillBuffer()
{
_bufOffset += _bufSize;
_bufSize = gzread(_file, _buffer, BUFSIZE);
_bufPos = 0;
}
void CompressedFile::fillBuffer()
{
_bufOffset += _bufSize;
_bufSize = gzread(_file, _buffer, BUFSIZE);
_bufPos = 0;
}
To copy to clipboard, switch view to plain text mode
So in principle I uncompress the file into a buffer (I tried sizes between 1kB and 1MB) and read from that. When I come to the end of the buffer, I read again.
I use it like this:
// open file
CompressedFile file(filename);
// start reading
while (!(line = file.readLine()).isEmpty()) {
// do something
}
// open file
CompressedFile file(filename);
file.open(QIODevice::ReadOnly);
// start reading
QString line;
while (!(line = file.readLine()).isEmpty()) {
// do something
}
To copy to clipboard, switch view to plain text mode
My problem is, that with one of the files I want to read with this, it takes about 35 seconds. I've got some IDL and python code here, that does the same in ~10 seconds. Any ideas, how can speed things up?
Cheers,
fallen
Bookmarks