Reading lots of Data from QProcess without freezing

[nightghost]

Hello,

I want to read a lot of data from a program started with QProcess and show the output in at QTextEdit. The problem is to get a good tradeoff between speed and response time (The programs are producing really a lot of output):

In our first implementation wie used QProcess::readAllStandardOutput(), but then the GUI freezed and it was nearly impossible to do something.

then we used in our SLOT:



int lines_read = 0;
while (process->canReadLine() && lines_read < 10) {

QByteArray data = process->readLine();

if (!data.isEmpty()) {

Info(_log, data.constData());
}

++lines_read;
}


After the program is finsihed in the finished handler all remaining output is written once into the log, then the GUI only freezes shortly (1-2 sec), but that is ok. the "Info()" macro inserts the data into a buffer and is then written to the QTextEdit.

The GUI is not freezing anymore while writing the output, but the output is written far to slow (3 min program and 17 min output (after the program has finished) If I increase the lines read once from 10 to lets say 100 is faster (but not fast enough), but the GUI is freezing again. I also played around with QApplication::processEvents(), but that did not solve the problem. (No freezing, but not fast enough either)

I assume, that the (one of the problems) problem is in reading the output from the QProcess, because after the QProcess is finsihed all data is written once to the buffer, and the data is only inserted into the QTextEdit.

Does anyone has a good guess what to do? Maybe use a separate Thread to read the output and/or write the data to the QTextEdit (if this is possible at all)

[high_flyer]

The correct way is to put such task in other threads.
The simple quick and dirty way would be to add processEvents() to your while() loop.

[nightghost]

The quick'n dirty way I already tried. Nothing freezes, but its far to slow (the outputrate in the QTextEdit)

You think of observe the output of the qprocess with a thread and write the data into our internal buffer, right?

[wysota]

How exactly do you input text to QTextEdit? How did you setup QTextEdit? Can't you use QPlainTextEdit instead?

[high_flyer]

In addition to what wysota wrote, it would probably be better if you first read in to a QString or QStringList, and only after the file reading is done, to do one setText() in to the QTextEdit.

[nightghost]

wysota: I am using append to insert line by line. Before I check every line with a QRegExp for containing links
high_flyer: Using setText seems not to be a option, since the QTextEdit works like a console (line after line and not 500 lines once)

Using PlainTextEdit gives a far more better performance. Thats what I need. Sadly PlainTextEdit does not support links (which I need), but maybe its better to do a tradeoff here:

Under normal circumstances the performace of QTextEdit is ok. Only if a lot of output is produced I ran into problems. Maybe I should switch from QTextEdit to a PlainTextEdit if my internal buffer reaches a specific size (that should be an indicator, that the inserting is to slow) I should switch to a PlainTextEdit. Than I lost the links, but the program remains stable and fast.


Nevertheless I have another idea: I already limited the maximum lines to 1000. I also observed, that sometimes my internal buffer reaches 50.000 remaining lines. In the PlainTextEdit the are inserted really fast (to fast to follow) and maybe its a good idea to skip just 49.000 lines and show only the last 1000 lines, since the other lines are useless for the user.

[boudie]

... and maybe its a good idea to skip just 49.000 lines and show only the last 1000 lines, since the other lines are useless for the user.[/INDENT]

That's just what I thought when reading this topic.
Why would you try to show something on screen when it's already to fast to be handled in software?
There are other ways to keep your users informed that something is happening behind the curtains... ;)

[wysota]

wysota: I am using append to insert line by line.
Bad idea, this method is terribly slow. Use QTextCursor interface instead.


Before I check every line with a QRegExp for containing links
Ouch, that slows it down even more. What do you need the links for?


Using PlainTextEdit gives a far more better performance. Thats what I need. Sadly PlainTextEdit does not support links (which I need), but maybe its better to do a tradeoff here:
It's not a big of a problem. You can visually mark the links using a syntax highlighter (which will make your QRegExp go away or at least move to a different place) and then you can handle the links by implementing mouse event handlers for your editor.


Under normal circumstances the performace of QTextEdit is ok.
I would certainly not agree with that statement.

[nightghost]

1.) A ok. I did'nt know, that this would make a difference
2.) The output is mostly from the GCC. The QRegExp is used to detect absolute paths and make links of them to click and jump (warnings, errors messages)
3.) Nice idea, but is this really faster? Matching the links in the SyntaxHighLighter or before adding sounds not like a big difference. But it sounds like a good idea to enable the link-feature and leave the log output untouched
4.) Hmm, ok :-) Its fast enough to be not annoying ;-)

[wysota]

3.) Nice idea, but is this really faster? Matching the links in the SyntaxHighLighter or before adding sounds not like a big difference. But it sounds like a good idea to enable the link-feature and leave the log output untouched
In "my" version you are using plain text. In "your" version you are using rich text with injected html tags. There is a huge difference.

[nightghost]

Ah ok. The SyntaxHighligher is a lot faster in formating the text then the rich-text engine. Thanks for the tip

[wysota]

The rich text engine has to parse the text each time it is being laid out (or at least each time the text changes). The syntax highlighter uses QTextDocument infrastructure.

[nightghost]

Thanks for the explanation. I'll have to investigate that further