rowCount() is calling for all the top level items

[prasad_N]

I have customized QAbstractItemModel & QTreeView.

rowCount() is calling for all the top level items in a view, is it a default behavior of model ? //I set setAllColumnsShowFocus( true ); setUniformRowHeights( true );
If so, can we change this so that it will get called only for currently viewing items in view.

I am dealing with huge data (millions of rows) because of this behavior there is some delay in viewing items.

Any suggestions ?

[wysota]

rowCount() is calling for all the top level items in a view, is it a default behavior of model ?
No, it's a default behaviour of QTreeView since the tree wants to know whether each of the items has child items.


If so, can we change this so that it will get called only for currently viewing items in view.
Hmm... I don't think the view cares about items that are not visible unless you asked items to be expanded, in which case it needs to calculate the total size of the tree for the scrollbar.

[prasad_N]

Hmm... I don't think the view cares about items that are not visible unless you asked items to be expanded, in which case it needs to calculate the total size of the tree for the scrollbar.

you mean we can change this default behavior right ?
If yes, How can we do this.

[wysota]

This example shows that the view does not query the whole model when it initializes:


#include <QtWidgets>


class Model : public QAbstractListModel {
public:
Model(QObject *parent = 0) : QAbstractListModel(parent) {}

int rowCount(const QModelIndex &parent = QModelIndex()) const {
qDebug() << Q_FUNC_INFO << parent;
if(parent.isValid()) return 0;
return 1000;
}
QModelIndex index(int row, int column, const QModelIndex &parent = QModelIndex()) const {
if(parent.isValid()) return QModelIndex();
return createIndex(row, column, (void*)0);
}

QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const {
if(role == Qt::DisplayRole) return index.row()+1;
return QVariant();
}
};

int main(int argc, char **argv) {
QApplication app(argc, argv);
QTreeView v;
Model model;
v.setModel(&model);
v.show();
return app.exec();
}

[prasad_N]

This example shows that the view does not query the whole model when it initializes:

No, it's a default behaviour of QTreeView since the tree wants to know whether each of the items has child items:
these 2 statements are conflicting right ?

1. In my model rowCount is calling for all the top level items.

my model is :


QModelIndex TreeModel::index( int row, int column, const QModelIndex &parent ) const
{
if ( ! hasIndex(row, column, parent) )
return QModelIndex();

TreeItem *parentItem;

if ( ! parent.isValid() )
parentItem = m_rootItem;
else
parentItem = static_cast<TreeItem*>( parent.internalPointer() );

TreeItem *childItem = parentItem->child( row );
if ( childItem )
return createIndex( row, column, childItem );
else
return QModelIndex();
}

QModelIndex TreeModel::parent( const QModelIndex &index ) const
{
if ( ! index.isValid() )
return QModelIndex();

TreeItem *childItem = static_cast<TreeItem*>( index.internalPointer() );
TreeItem *parentItem = childItem->parent();

if ( (parentItem == m_rootItem) || (parentItem == NULL))
return QModelIndex();

return createIndex( parentItem->row(), 0, parentItem );
}

int TreeModel::rowCount( const QModelIndex &parent ) const
{
if ( parent.column() > 0 )
return 0;

TreeItem *parentItem;
if ( ! parent.isValid() )
parentItem = m_rootItem;
else
parentItem = static_cast<TreeItem*>( parent.internalPointer() );

return parentItem->childCount();
}


2. When I change your model little bit on order to check tree function calling. in the index if(parent.isValid()) is not at all executing.


int rowCount(const QModelIndex &parent = QModelIndex()) const {
qDebug() << Q_FUNC_INFO << parent;
if(parent.isValid()) return 2; //each parent with 2 child
return 10; // 10 top most parrents
}

QModelIndex index(int row, int column, const QModelIndex &parent = QModelIndex()) const {
if(parent.isValid()){ // this condition is not at all executing
qDebug() << "Yes isValid = " << parent.child(row, column);
return createIndex(row, column, (void*)parent.child(row, column).internalPointer()); //create &return child index
}

return createIndex(row, column, (void*)0); // return top most elements index's
}


What wrong am I doing here ?

I don't want to use treeItems in my model for tree(if possible, I think it is possible with above implementation).

[wysota]

these 2 statements are conflicting right ?
Yes, I didn't express myself clearly. I meant that it is the view which is causing what you see, not the model. I didn't mean to say that "the view has to query all the indexes".


1. In my model rowCount is calling for all the top level items.

my model is :


QModelIndex TreeModel::index( int row, int column, const QModelIndex &parent ) const
{
if ( ! hasIndex(row, column, parent) )
return QModelIndex();

TreeItem *parentItem;

if ( ! parent.isValid() )
parentItem = m_rootItem;
else
parentItem = static_cast<TreeItem*>( parent.internalPointer() );

TreeItem *childItem = parentItem->child( row );
if ( childItem )
return createIndex( row, column, childItem );
else
return QModelIndex();
}

QModelIndex TreeModel::parent( const QModelIndex &index ) const
{
if ( ! index.isValid() )
return QModelIndex();

TreeItem *childItem = static_cast<TreeItem*>( index.internalPointer() );
TreeItem *parentItem = childItem->parent();

if ( (parentItem == m_rootItem) || (parentItem == NULL))
return QModelIndex();

return createIndex( parentItem->row(), 0, parentItem );
}

int TreeModel::rowCount( const QModelIndex &parent ) const
{
if ( parent.column() > 0 )
return 0;

TreeItem *parentItem;
if ( ! parent.isValid() )
parentItem = m_rootItem;
else
parentItem = static_cast<TreeItem*>( parent.internalPointer() );

return parentItem->childCount();
}

This is the implementation of QAbstractItemModel::hasIndex():

bool QAbstractItemModel::hasIndex(int row, int column, const QModelIndex &parent) const
{
if (row < 0 || column < 0)
return false;
return row < rowCount(parent) && column < columnCount(parent);
}

Therefore each time you call you index() implementation, it calls hasIndex() which in turn calls rowCount() for the parent. Possibly this applies to other methods and their default implementations.


2. When I change your model little bit on order to check tree function calling. in the index if(parent.isValid()) is not at all executing.


int rowCount(const QModelIndex &parent = QModelIndex()) const {
qDebug() << Q_FUNC_INFO << parent;
if(parent.isValid()) return 2; //each parent with 2 child
return 10; // 10 top most parrents
}

QModelIndex index(int row, int column, const QModelIndex &parent = QModelIndex()) const {
if(parent.isValid()){ // this condition is not at all executing
qDebug() << "Yes isValid = " << parent.child(row, column);
return createIndex(row, column, (void*)parent.child(row, column).internalPointer()); //create &return child index
}

return createIndex(row, column, (void*)0); // return top most elements index's
}


What wrong am I doing here ?
Most likely you didn't change the base class to QAbstractItemModel. Apparently QTreeView has an optimization to not check child nodes for flat models.


#include <QtWidgets>

#define BASEMODEL QAbstractItemModel

class Model : public BASEMODEL {
public:
Model(QObject *parent = 0) : BASEMODEL(parent) {}

int columnCount(const QModelIndex &parent = QModelIndex()) const { return 1; }

int rowCount(const QModelIndex &parent = QModelIndex()) const {
qDebug() << Q_FUNC_INFO << parent;
if(!parent.isValid()) return 10;
if(parent.isValid() && !parent.parent().isValid()) return 2; // limit to two levels

return 0;
}
QModelIndex index(int row, int column, const QModelIndex &parent = QModelIndex()) const {
qDebug() << Q_FUNC_INFO << row << column << parent;
if(parent.isValid()) {
qDebug() << Q_FUNC_INFO << "parent valid";
// we only need id of the parent since it identifies the item
return createIndex(row, column, parent.row());
}
return createIndex(row, column, -1); // top level item has invalid parent
}

QModelIndex parent(const QModelIndex &ind) const {
qDebug() << Q_FUNC_INFO << ind;
if(!ind.isValid()) return QModelIndex();
int id = ind.internalId();
if( id == -1) return QModelIndex();
return index(id, 0); // return top-level item
}

QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const {
if(role == Qt::DisplayRole) return index.row()+1;
return QVariant();
}
};

int main(int argc, char **argv) {
QApplication app(argc, argv);
QTreeView v;
Model model;
v.setModel(&model);
v.show();
return app.exec();
}


I don't want to use treeItems in my model for tree(if possible, I think it is possible with above implementation).
Then don't use them :)

By the way, changing the base class to QAbstractItemModel does cause the tree view to query rowCount for all the top-level items at start, I will try to see why this happens. You can probably switch some property and it will stop doing that.

Edit: It looks like during layout of the tree, the widget calls rowCount() on items that have children. What is worrying is that it also calls canFetchMore() && fetchMore() for them which in my opinion defeats the purpose of implementing those two methods at all at least for two level models.

[prasad_N]

but when I test your new model I have seen all the top levels items rowsCount() is getting called, I am struggling exactly with this behavior only.


int rowCount(const QModelIndex &parent = QModelIndex()) const
{
qDebug() << parent; //little change in debug msg

if(!parent.isValid()) return 10;
if(parent.isValid() && !parent.parent().isValid()) return 2; // limit to two levels

return 0;
}

int main(int argc, char **argv) {
QApplication app(argc, argv);
QTreeView v;
v.setFixedSize(200, 50); //set height to 50 so that view can accommodate 2 top level elements only
Model model;
v.setModel(&model);
v.show();
return app.exec();
}

The out put I observed is :


QModelIndex(-1,-1,0x0,QObject(0x0) )
QModelIndex(-1,-1,0x0,QObject(0x0) )
QModelIndex(-1,-1,0x0,QObject(0x0) )
QModelIndex(0,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(1,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(2,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(3,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(4,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(5,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(6,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(7,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(8,0,0xffffffff,QAbstractItemModel(0x28 fe18) )
QModelIndex(9,0,0xffffffff,QAbstractItemModel(0x28 fe18) )

//rowCount() is getting called for all top level items here.



One more thing I have observed is, When i return 10000000(top level items) in rowCount() (in old model you suggested & in new model also), My application got crashed with runtime error (on Windows 8, 64bit application).
And when I return 1000000 row count & when I try to scroll down its hanging like hell, not able to scroll at all.

[wysota]

Which Qt version are you using? How much RAM does your machine have?

[prasad_N]

I am using 4.8.6 & 5.4.0 with windows 8 and 4GB RAM, I tested on both the versions it failed.

[prasad_N]

Hi Any idea, why is this happening ?

[wysota]

4E9 bytes for 1E7 items yields 4E2 (=400) bytes per item. My guess is you are running out of memory :)

[prasad_N]

No I am not, It is taking around 128MB (for 10000000 top level items) && 28MB (for 1000000 top level items).
still view is freezing like anything when I scroll, did you reproduce same when you pass more top level count.

[wysota]

If you call setUniformRowHeights(true) then there is no such effect. I can scroll the view very quickly.

By the way, I don't think you calculated the memory use correctly. For 10M top-level items with 2 child items each the application uses over 600MB for the data segment.

[prasad_N]

If you call setUniformRowHeights(true) then there is no such effect. I can scroll the view very quickly.
- forgot to set setUniformRowHeights(true), Its working fine now

- The only & main issue I have now is, rowCount() is still calling for all the top level items , Any idea how to prevent it.?


By the way, I don't think you calculated the memory use correctly. For 10M top-level items with 2 child items each the application uses over 600MB for the data segment.
Sorry I ran 32-bit application, It is taking around 280MB in a same case.

[prasad_N]

Hi Wysota,
Did you find any solution for this (rowCount() is calling for all the top level items),
I have observed one more thing that "Even for each mouse click on cell is calling rowCount() for all the top level items"
I think this is the reason for my recent post - http://www.qtcentre.org/threads/63074-Updating-header-data-is-too-slow-as-data-grows-high-in-Model-view-architecture?highlight=

what kind of optimizations we can do for both the problems.

[wysota]

Why is that a problem that the framework calls rowCount() of every top-level item? Do you get noticable slowdowns because of that? The only "solution" I can see is to file a bug.

[prasad_N]

Why is that a problem that the framework calls rowCount() of every top-level item? Do you get noticable slowdowns because of that? The only "solution" I can see is to file a bug.

yes, There is a delay in populating data (I have populated 30 million rows with column count > 10).

One more problem:
And rowCount() is calling for all top level items when I emit headerDataChanged() signal which leads to further delay in selecting item & changing header.
Item is not getting selected until header data changes. there is a considerable delay in changing header data after that only item is getting selected.
my post related to this issue : http://www.qtcentre.org/threads/63074-Updating-header-data-is-too-slow-as-data-grows-high-in-Model-view-architecture?highlight=

[wysota]

yes, There is a delay in populating data (I have populated 30 million rows with column count > 10).
Maybe you can populate the model in batches? I think you could use canFetchMore() && fetchMore() for the invalid model index.

One more problem:

And rowCount() is calling for all top level items when I emit headerDataChanged() signal which leads to further delay in selecting item & changing header.
Think how to avoid changing the header too often. You can always show a custom header instead of the default one.

[prasad_N]

Sorry I could not get you properly.


Maybe you can populate the model in batches? I think you could use canFetchMore() && fetchMore() for the invalid model index.

You mean populating model with half data.
Actually I have more than 100 million of rows (would be more also) because of this BUG (https://bugreports.qt.io/browse/QTBUG-28631), I went for pages (I will have combo box top of the view with 0 - totalRows/30 million ranges), so each time I chose a number I am building a tree of respective range and showing that tree so that I am showing 30 million of rows at a time in view, but this is also causing performance problems.

with fetchMore() & canFetchMore() i will have scroll bar problem I think, when I come to end of the scroll then only I would come to know there is more elements and then I will load the data and then the scroll bar will adjust according to that, I don't want this behavior scroll bar size should be decide at first instance only(This is the reason I am not using fetchMore functionality in my app).



One more problem:

Think how to avoid changing the header too often. You can always show a custom header instead of the default one.

You mean, you want me to customize the header also ?


Can you be more verbose for the above 2 statements :-)

[wysota]

You mean populating model with half data.
I mean populate as much as is currently useful for you. If the user scrolls down, populate more, etc.


with fetchMore() & canFetchMore() i will have scroll bar problem I think, when I come to end of the scroll then only I would come to know there is more elements and then I will load the data and then the scroll bar will adjust according to that, I don't want this behavior scroll bar size should be decide at first instance only(This is the reason I am not using fetchMore functionality in my app).
You are out of luck then :) You can file a bug or provide a patch for the (IMO incorrect) behaviour.


You mean, you want me to customize the header also ?


Can you be more verbose for the above 2 statements :-)

You can hide the original header and replace it with your own instance of QHeaderView that is not tied to the model as I understand you want your header to reflect the current view state rather than the model's data.

[prasad_N]

I report a bug in Qt bug tracker for this.


You can hide the original header and replace it with your own instance of QHeaderView that is not tied to the model as I understand you want your header to reflect the current view state rather than the model's data.

I want my header to be changed depends on the row I clicked (for ex: if I click on a even row (index.row()%2 == 0 ) I put one header set & for odd row I put another header set).
in this case how a customized header can help me ?
And I could not see any function for setting data (if I customize QHeaderView) rather than QHeaderView::setRootIndex(QModelIndex);

Thanks.

[wysota]

The header takes its data from headerData() method of the model. Simply provide a custom model for it. It can even be empty just make it return whatever you need from headerData().

[prasad_N]

The header takes its data from headerData() method of the model. Simply provide a custom model for it. It can even be empty just make it return whatever you need from headerData().

Thanks, It did worked with the sample I provided here, need to check with my project hope will work :-)