Increasing QProcess read performance

[mcostalba]

$ time data_producer >> tmp.txt
3.34user 0.64system 0:03.99elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+15238minor)pagefaults 0swaps

BTW HD write activity is done by Linux _after_ execution ends. So data_producer really writes to memory, indeed to OS disk write cache.

[jacek]

BTW HD write activity is done by Linux _after_ execution ends. So data_producer really writes to memory, indeed to OS disk write cache.

But IMO it's still better than measuring how fast OS can discard data.

When your application is processing the data, it probably blocks event loop for a while. Maybe you could process the data in smaller chunks or invoke QApplication::processEvents() from time to time to read new data and free the buffer?

In Qt4 there's QProcess::setupChildProcess(), but I couldn't find anything similar for Qt3. Maybe you could tweak the output buffer in data producer (if it's possible, since I've never tried to do this)?

[mcostalba]

Ok. With receiver slot like:

void DataLoader:n_readyReadStdout() {

proc.readStdout();
}

I have about 6.5s, not a biggie.

The real receiver is:

void DataLoader:n_readyReadStdout() {

// we use a circular buffer to store data chunks from loading process
QByteArray* b = new QByteArray(proc.readStdout()); // copy c'tor uses shallow copy
buffersRing.insert(buffersRingHead, b);

if (++buffersRingHead == BUF_RING_SIZE)
buffersRingHead = 0;
}

But I really don't know how to make it faster. Could you spot some hidden copies on above code?

Thanks
Marco

[mcostalba]

I have tweaked the code to process only a limited amount of data each call, instead of all the available data, so to not stall the pipe, but the results are worst!!! About half second more.

I think this is due to higher context switches numbers between two processes, reciever and producer. It seems it is better, one switched the context, to stay there a little bit more then less.

[wysota]

What kind of data to you receive? If it's text data you can process the incoming data only if a newline is received.

What exactly is buffersRing? How long does it take to insert an item there?

[jacek]

I have tweaked the code to process only a limited amount of data each call, instead of all the available data, so to not stall the pipe, but the results are worst!!!

How long does it take if you first read all of the data and then process it?

[mcostalba]

Here I am again. These are the answers you asked for:

1) QPtrVector<QByteArray> buffersRing;

2) It's text data formed by records, each record is delimited by \0, so I have first to find the delimiting \0 with the data still in QByteArray format, then I can convert the chunk to QString:

// find next chunk, about 5% of total time
endIdx = (startIdx < (int)ba.size()) ? ba.find(0, startIdx) : -1;

// deep copy here by fromAscii(), about 4% of total time
const QString& chunk(QString::fromAscii(&(ba[startIdx]), endIdx - startIdx));


Where ba it's one of the buffers pointed by buffersRing. I do this outside of fast path, on a timer set to 500ms. There some logic, here omitted, to handle half lines / half records and so on.


3) Times get only slightly better if I first read all of the data and then process it, but the GUI is blank until the end of loading, so the _feeling_ from the user is of more slowness.


Now what I have found:

QProcess::readyReadStdout() is emitted at furious paces, always less then 10ms each, very often less then 5ms. And the data read in proc.readStdout() is very small 5-10KB, so you can imagine how many calls are needed to read 30MB of stuff!

So I have disconnected the signal from the slot and used a timer set to 50ms to manually call the on_readyReadStdout() slot. Now data chunks are quite bigger, about 100-200KB and frequency is less. Total time got better by about 5%, though it's still much higher then when data_producer writes directly to a file.

Setting the timer to a bigger interval 100-200ms does not give better results.

I can perform other test if you want.

Thanks
Marco