QStandardItemModel performance with high number of elements

[Wurgl]

I have here some Data model which hold about 100.000 rows and each row has 3 columns. When I iterate through all rows to modify one of the three items, this takes a little bit more than a minute. On the other hand if I have only 6.500 rows in that model, than the same change for all items in one row is below 1 second. So the time for this change is not linear, it is either quadratic or something like n*log(n).

Is there any recommended way of iterating through such a pretty large set of data?

I used (this one took 86 seconds):

for(int r = 0; r < _model->rowCount(); ++ r) {
    QStandardItem *item = _model->item(r, 0);
    item->setData(foo(item->data().toUInt()), Qt::DisplayRole);
}

for(int r = 0; r < _model->rowCount(); ++ r) {
    QStandardItem *item = _model->item(r, 0);
    item->setData(foo(item->data().toUInt()), Qt::DisplayRole);
}

To copy to clipboard, switch view to plain text mode 

And I tried (which took 76 seconds):

QModelIndex idx = _model->indexFromItem(_model->item(0, 0));
for(int r = 0; r < _model->rowCount(); ++ r) {
    _model->setData(idx, foo(_model->data(idx, Qt::UserRole + 1).toUInt()), Qt::DisplayRole);
    idx = idx.sibling(r + 1, 0);
}

QModelIndex idx = _model->indexFromItem(_model->item(0, 0));
for(int r = 0; r < _model->rowCount(); ++ r) {
    _model->setData(idx, foo(_model->data(idx, Qt::UserRole + 1).toUInt()), Qt::DisplayRole);
    idx = idx.sibling(r + 1, 0);
}

To copy to clipboard, switch view to plain text mode 

The initial filling of the model with all these 100.000 rows was done within 1 or 2 seconds. However, this was done in the constructor before show() was called. So maybe turning off some graphical issues might also improve the programm.

Any ideas?

[Lesiok]

What is a time when You change call of function foo() to some static value.

[Wurgl]

Think of foo() as a placeholder. It was a noop in the tests. But look at the numbers, when I have 20 times more elements, then the time to process is beyond 70 times larger.

150.000 rows takes 236 seconds, do you see the (maybe) quadratic time behaviour.

[wysota]

The recommended way is to avoid using QStandardItemModel for a large set of data.

[Wurgl]

Okay. No QStandardItemModel. Instead I am using TreeWidget.

for(QTreeWidgetItemIterator it(_tree); *it; ++ it) {
    (*it)->setData(0, Qt::DisplayRole, foo((*it)->data(0, Qt::UserRole + 1).toUInt()));
}

for(QTreeWidgetItemIterator it(_tree); *it; ++ it) {
    (*it)->setData(0, Qt::DisplayRole, foo((*it)->data(0, Qt::UserRole + 1).toUInt()));
}

To copy to clipboard, switch view to plain text mode 

About same speed as before; 150.000 rows in 279 seconds.

The recommended way is to avoid QStandardItemModel. Fine. The recommended way seems to avoid QTreeWidget too. What else shall be avoided, or more helpful what shall be used?

[Santosh Reddy]

The recommended way is to avoid QStandardItemModel. Fine. The recommended way seems to avoid QTreeWidget too. What else shall be avoided, or more helpful what shall be used?

Use a custom model derived from QAbstractItemModel, then you will have complete control over how to iterate though the data/rows and will also have control when to inform the view/display to update itself. But be aware writing custom QAbstractItemModel is not as stright forward as using QStandardItemModel/QTreeWidget. When using custom QAbstractItemModel you will need to implement certain virtual functions, which you can and figure out form Qt documentation.
Refer Simple Tree Model Example

[wysota]

Okay. No QStandardItemModel. Instead I am using TreeWidget.

That's even worse

What else shall be avoided, or more helpful what shall be used?

A custom model class that will be tailored to the behaviour of your data. What I say might be obvious but a lot of people still do that mistake -- if you want your model to be faster than QStandardItemModel then don't implement it in a way that mimics QStandardItemModel. Instead think about characteristics of your data so that you can avoid problems that make QStandardItemModel slow. The "Simple Tree Model" example is one that mimics the standard item model so treat it as a tutorial on model programming rather than a guide to implement your own model. You should spend 80% of time thinking and only then 20% of time coding.

[neuronet]

Instead think about characteristics of your data so that you can avoid problems that make QStandardItemModel slow. The "Simple Tree Model" example is one that mimics the standard item model so treat it as a tutorial on model programming rather than a guide to implement your own model.

I didn't realize simpletreemodel is like that, but it makes sense. Is there a good practical example (i.e., not abstract hint) on how to optimize a model, e.g., one that digs in and uses hints like those in the model/view overview on lazy population of model data?

Also, do you know of any discussion of the size bounds on different model types? E.g., if you are below X, then use TreeWidget, if below Y, then QSTandardItemModel, otherwise use QAbstractItemModel? It seems at some size these classes hit some kind of wall, then it is time to move up to the next one. How much explicit has been written on this?


Sorry I know this is an old thread...

[wysota]

There are no bounds like that. In general QTreeWidget is ok for prototyping and very simple use-cases but as its complexity grows, it gets more and more difficult to get new things done without breaking others. QStandardItemModel is an ok-ish approach is your data is very standard -- items don't depend on other items, all operations involve modifying single items and you don't have an existing data backend or you are ok with copying data and dealing with synchronization. Other than that implementing a custom model is almost always a desired approach when dealing with real-world use cases.