Storing and displaying "large" amount of data

[FreddyKay]

Hey guys!

I have data stored in vectors that I would like to show in a QTextEdit. What I do now is I create a QTextDocument and a QTextCursor and I append all data as QStrings into that Document like this (altered some things so you can try to compile if needed):



QTextDocument* parseDoc(std::vector<double>& time, std::vector<std::array<double,3>>& pos, std::vector<std::array<double,3>>& pos){
parsedDocument = new QTextDocument();
myDocumentCursor = new QTextCursor(parsedDocument);

QTextCharFormat textFormat = myDocumentCursor->charFormat();
textFormat.setFontFamily("Courier");
myDocumentCursor->setCharFormat(textFormat);


QString formattedString;
formattedString = QString("This is th Data of Some object. It was retrieved on %2\r\n").arg(QDateTime::currentDateTime().toString());
myDocumentCursor->insertText(formattedString);

myDocumentCursor->insertBlock();

formattedString = QString("Epoch").leftJustified(21, ' ');
formattedString.append(QString("Position x-Axis").leftJustified(21, ' '));
formattedString.append(QString("Position y-Axis").leftJustified(21, ' '));
formattedString.append(QString("Position z-Axis").leftJustified(21, ' '));
formattedString.append(QString("Velocity x-Axis").leftJustified(21, ' '));
formattedString.append(QString("Velocity y-Axis").leftJustified(21, ' '));
formattedString.append(QString("Velocity z-Axis").leftJustified(16, ' '));
myDocumentCursor->insertText(formattedString);

myDocumentCursor->insertBlock();

formattedString.clear();

for(int i = 0; i < time.size(); i++){

formattedString = QString("%1 %2 %3 %4 %5 %6 %7").arg(time[i], -16, 'f', -1)
.arg(pos[i][0], -16, 'f', -1).arg(pos[i][1], -16, 'f', -1).arg(pos[i][2], -16, 'f', -1)
.arg(vel[i][0], -16, 'f', -1).arg(vel[i][1], -16, 'f', -1).arg(vel[i][2], -16, 'f', -1);

myDocumentCursor->insertText(formattedString);
myDocumentCursor->insertBlock();
}

return parsedDocument;

}



Basically this works out quite smoothly, just takes a few minutes as I have roughly 500K lines of data. But there are 2 major problems that I ran into: 1. The memory usage after putting all data into the Document is around 1.2GB, 2. It takes a HUGE amount of time(~15min) to display the document in a QTextEdit (via QTextEdit::setDocument(QTextDocument*)).

So obviously I am doing something wrong. So do you guys have an idea what I should be doing differently?

[anda_skoa]

Since you basically have a tabular structure, e.g. several values belonging to one record, each record having the same data fields, maybe consider using a table model and a table view?

Cheers,
_

[FreddyKay]

Since you basically have a tabular structure, e.g. several values belonging to one record, each record having the same data fields, maybe consider using a table model and a table view?

Cheers,
_

That might be possible, though I would rather like to have it in textform, as its way easier to use in my opinion. Is it normal, that QTextDocument is causing that much memory usage? If I put all those strings into a QStringList the amount of data in RAM is aprox. 75MB (just like a TextFile if I save all entries into a file). The TextDocument needs almost 15 times as much apparently. This makes me believe I made a mistake somewhere and there are maybe memory leaks or data dulpicates or somehting.

Can you think of a way that would reduce memory usage and display time to more reasonable levels, where I can still use the QTextEdit / QPlainTextEdit?

[d_stranz]

I don't see any obvious memory leaks, unless you are calling this method more than once. Each time through you create a new QTextDocument instance; if you aren't getting rid of these and are calling this method over and over, then you'll keep accumulating dangling pointers to the old documents and using up more and more memory.

This really does scream to be represented as a table. You're formatting it that way, so just use a QTableWidget or QTableView and turn off the grid lines.

The amount of time could be because the document is recomputing its internal layout with each line you add to it and possibly pushing everything onto undo / redo stacks with each addition. The default value for the undoRedoEnabled property is true, and I don't see where you disable it. So you might have an undo stack with 500K blocks on it.

[FreddyKay]

I don't see any obvious memory leaks, unless you are calling this method more than once. Each time through you create a new QTextDocument instance; if you aren't getting rid of these and are calling this method over and over, then you'll keep accumulating dangling pointers to the old documents and using up more and more memory.

This really does scream to be represented as a table. You're formatting it that way, so just use a QTableWidget or QTableView and turn off the grid lines.

The amount of time could be because the document is recomputing its internal layout with each line you add to it and possibly pushing everything onto undo / redo stacks with each addition. The default value for the undoRedoEnabled property is true, and I don't see where you disable it. So you might have an undo stack with 500K blocks on it.

Alright. Thanks a lot you two. I'll give QTableView a try, though I will also try disabling the undoRedo thingy.