beginRemove() endRemove = noRemove

[RolandHughes]

I have a QTableView and a model based on QAbstractTableModel. The model is periscoping over what will be a massive dataset. Naturally we had to have that *(&^)(*&^(*&ing flick scrolling on the view so we had to have an external scrollbar to cover the entire database range. (I thought the user should be punished and made to flick 80 rows at a time through a multi-million row set, but that is another story.)

Quite dutifully my model calls beginRemove(), deletes data, then calls endRemove(). Scrolling down stutters and jerks like I expect it would. Scrolling up is smooth as silk. I guess I should have made them fill out that dataset to maximum size for initial testing because trying to fit north of 1TB binary data into 4G of RAM in text form would have pointed out the issue early on. The data is removed from the model, but not the view.

I will dig further into the QTableView documentation after supper, but here is the question: Does the default implementation of QTableView ignore beginRemove() and endRemove()? It appears to ignore this but respect beginAppend() and endAppend().

Thanks,

Oh. Qt 4.8.x on Linux Target is embedded, but problem visible on both.

[wysota]

Does the default implementation of QTableView ignore beginRemove() and endRemove()?

No, it doesn't. beginRemove*() and endRemove*() cause persistent indexes to be updated and then emit respective (rows|columns)Removed() signals.

wysota@Teletraan3:~/kompilacje/qt-everywhere-opensource-src-4.8.6/src/gui/itemviews$ grep rowsRemoved qtableview.cpp
disconnect(d->model, SIGNAL(rowsRemoved(QModelIndex,int,int)),
connect(model, SIGNAL(rowsRemoved(QModelIndex,int,int)),

[RolandHughes]

Thanks for taking the time to respond. Not exactly an answer, but a response, so thanks for taking the time to do it. I looked at the source file you indicated and it confirms what I suspected/see happening. The "view", meaning the scrolling region, is not getting content trimmed/updated. The model is physically removing data from itself and some black bag stuff is being updated behind the scene, but the "text" the scrolling area knows about is not being cleaned. This appears to be organized for user drive deletions not model driven.

Before we go off on a tangent allow me to provide a picture. Other than being illustrative and conceptually what is happening, it has no relationship to what is actually being done.

Consider the model to be a bookshelf of some fixed size able to support limitless weight but only so many feet wide. When the db scrollbar (scrolling by ID, not by pixel like the view scroll area) is moved it sees if the model has any rows for the new ID and then tells the view to scroll N or -N pixels to get to the first of what could be many rows for that ID (i.e. Some books are many inches thicker than others). When the ID is not there the it tells the model to get more books. The model shoves those books onto one end or the other of the shelf. It then makes note how many books need to fall off the other end for things to fit, does a beginRemove(), deletes the data, then does an endRemove(). It's not a math problem with pixels/row because it does scroll to correct information.

Debug statements for the hidden viewport scroll bar show the pixel position growing and growing and growing. Despite the fact the model has physically removed the data and done the begin/end remove stuff, the scroll area is not tidied up.

Oh well, long day. Some wine, some sleep, and tomorrow add a throw away user delete method to verify if this was setup to be user driven instead of data driven and needs selected row(s).

Thanks again.

[d_stranz]

Oh well, long day. Some wine, some sleep.

My preferred solution to vexing programming problems. I usually insert "some good jazz" between the wine and sleep, but often the wine and jazz are concurrent.

[wysota]

Thanks for taking the time to respond. Not exactly an answer, but a response, so thanks for taking the time to do it.
Not very kind of you to say that.


I looked at the source file you indicated and it confirms what I suspected/see happening. The "view", meaning the scrolling region, is not getting content trimmed/updated. The model is physically removing data from itself and some black bag stuff is being updated behind the scene, but the "text" the scrolling area knows about is not being cleaned.
What exactly do you mean by "the text the scrolling area knows about"? The view does not hold the model data anywhere, that's the model's responsibility.


Before we go off on a tangent allow me to provide a picture. Other than being illustrative and conceptually what is happening, it has no relationship to what is actually being done.

Consider the model to be a bookshelf of some fixed size able to support limitless weight but only so many feet wide. When the db scrollbar (scrolling by ID, not by pixel like the view scroll area) is moved it sees if the model has any rows for the new ID and then tells the view to scroll N or -N pixels to get to the first of what could be many rows for that ID (i.e. Some books are many inches thicker than others). When the ID is not there the it tells the model to get more books. The model shoves those books onto one end or the other of the shelf. It then makes note how many books need to fall off the other end for things to fit, does a beginRemove(), deletes the data, then does an endRemove(). It's not a math problem with pixels/row because it does scroll to correct information.

Debug statements for the hidden viewport scroll bar show the pixel position growing and growing and growing. Despite the fact the model has physically removed the data and done the begin/end remove stuff, the scroll area is not tidied up.


I don't know how you are implementing your model and your view but according to me you have two choices compliant with the item-views architecture. Basically you can either report the proper size of the model and cache its data partially or modify the model to fit what is displayed in the view using insert/remove rows. I'm assuming you have taken the latter approach so I will focus on the former one first.

Here is an example which implements a model with 40M items where only 1k items are kept in memory at once (unfortunately the index is int-based so we can't have much more rows).


#include <QApplication>
#include <QAbstractTableModel>
#include <QTableView>
#include <QContiguousCache>
#include <QtDebug>

class Model : public QAbstractTableModel {
public:
Model(int from, int to, int cacheSize, QObject *parent = 0) : QAbstractTableModel(parent),
m_from(from),
m_to(to)
{
m_cache.setCapacity(cacheSize);
}
int rowCount(const QModelIndex &parent = QModelIndex()) const { return parent.isValid() ? 0 : m_to-m_from+1; }
int columnCount(const QModelIndex &parent = QModelIndex()) const { return 1; }
QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const {
if(role != Qt::DisplayRole) return QVariant();
int row = index.row();
if(row > m_to) return QVariant();
if(row < m_from) return QVariant();
while(row > m_cache.lastIndex())
m_cache.append(populate(m_cache.lastIndex()+1));
while(row < m_cache.firstIndex())
m_cache.prepend(populate(m_cache.firstIndex()-1));
return m_cache.at(row);
}
protected:
int populate(int row) const {
return row*2; // simple populate function
}

private:
mutable QContiguousCache<int> m_cache;
int m_from;
int m_to;
};

int main(int argc, char *argv[])
{
QApplication a(argc, argv);
QTableView view;
Model model(0, 40000000L, 1000);
view.setModel(&model);
view.show();
return a.exec();
}

You can fine tune the model by providing some look-ahead buffer, add data in batches and/or use a thread to populate the look-ahead buffer. The docs for QContignousCache might give you some ideas.

As for the other approach, maybe we could have a look at your code or some example which behaves in a similar fashion? Are you making use of canFetchMore()/fetchMore()?

[RolandHughes]

My preferred solution to vexing programming problems. I usually insert "some good jazz" between the wine and sleep, but often the wine and jazz are concurrent.

I've been more into "space music" lately, especially while coding.

Added after 51 minutes:



What exactly do you mean by "the text the scrolling area knows about"? The view does not hold the model data anywhere, that's the model's responsibility.


What I mean is the model physically has the rows removed and has called the beingRemove/endRemove stuff accordingly but the pixel count for the scrolling area increases with each append without ever decreasing for the remove.




I don't know how you are implementing your model and your view but according to me you have two choices compliant with the item-views architecture. Basically you can either report the proper size of the model and cache its data partially or modify the model to fit what is displayed in the view using insert/remove rows. I'm assuming you have taken the latter approach so I will focus on the former one first.


There-in lies the rub. One database record could be one row in the table or it could be 100+ depending on selection criterea. The proprietary data encoding cannot be counted or expanded with SQL. In short the rowCount of the universe cannot realistically be known. I was not using a contiguous cache but rather a QList of QLists since I have x-y columns depending on configuration. My model communicates with another part of the system which pulls chunks of data out of the database. My model then expands it based on the settings creating a QList of column data which gets added to the QList of QLists for rows. The rowCount returned is always the current size of my row QList. The size of that QList is checked after each manipulation and entries from the other end are removed until we are below the target.



As for the other approach, maybe we could have a look at your code or some example which behaves in a similar fashion? Are you making use of canFetchMore()/fetchMore()?


Not at this time. I experimented with that several months ago when I first did this table but it didn't seem viable. It is all well and good for adding chunks, but not so good for the periscope model.

No, I cannot change the database. I fought that battle and have all of the changes I will ever get.

I will spend the rest of the day working on it. If I don't get it then I will have to spend half a day coding something completley unrelated from scratch which demonstrates the problem without violating NDA, SECRETS, or any other agreement.

Thanks again.

[wysota]

What I mean is the model physically has the rows removed and has called the beingRemove/endRemove stuff accordingly but the pixel count for the scrolling area increases with each append without ever decreasing for the remove.
The scroll is purely virtual, QTableView inherits QAbstractScrollArea where no actual scrolling is really done. What you draw is simply shifted based on the values of the scrollbars (ranging from verticalScrollBar()->minimum() to verticalScrollBar()->maximum()).


#include <QApplication>
#include <QTableView>
#include <QStandardItemModel>
#include <QtDebug>
#include <QScrollBar>

int main(int argc, char **argv) {
QApplication app(argc, argv);
QTableView view;
QStandardItemModel model(400,1);
view.setModel(&model);
view.setVerticalScrollMode(QAbstractItemView::Scro llPerPixel);
view.show();
app.processEvents();
qDebug() << view.verticalScrollBar()->minimum() << view.verticalScrollBar()->maximum();
model.removeRows(0, 50);
app.processEvents();
qDebug() << view.verticalScrollBar()->minimum() << view.verticalScrollBar()->maximum();
return 0;
}

yields:

0 11831
0 10331

I do not observe any pixel count not decreasing upon removal.

There-in lies the rub. One database record could be one row in the table or it could be 100+ depending on selection criterea. The proprietary data encoding cannot be counted or expanded with SQL. In short the rowCount of the universe cannot realistically be known.
That is not a problem. You can always add new rows to the model. Removing seems to be your problem which the use of QContignousCache (or any other similar approach) avoids in an elegant way. And in my opinion you should really look into using canFetchMore()/fetchMore() (if your backend API allows to implement it).

[RolandHughes]

I'm working on a stand alone example which removes all database, etc and doesn't expose anything requiring a wink and a nod to a blind man to know.

What is getting lost in translation here is the "book" concept. An entire book has to fall off, not just a few pages.

The model has to keep track of which database IDs (hmm Dewey Decimal Numbers actually work well for this). What is set is the number of database IDs we allow the model to utilize at any given time. The number of rows is, however, many it needs to be. Each database ID could be dozens or hundreds of rows in the table. When an ID gets shoved out of the list all of the rows for that ID get nuked from the model. There is no realistic method of retrieving and reconstructing part of a book, has to be the entire thing, you don't get to rip the first chapter out and stuff it back in when you are done.

It's not of my choosing, it simply is what it is.

Btw, after others punted on this and it was handed to me I changed the model to be based on QAbstractTableModel which may not have been the wisest choice.

Now that I'm typing this and thinking about it, what I really need is a model of a model. One model containing database IDs which feeds/communicates with the next model which does the expansion and removal.

[wysota]

I'm working on a stand alone example which removes all database, etc and doesn't expose anything requiring a wink and a nod to a blind man to know.

What is getting lost in translation here is the "book" concept. An entire book has to fall off, not just a few pages.

The model has to keep track of which database IDs (hmm Dewey Decimal Numbers actually work well for this). What is set is the number of database IDs we allow the model to utilize at any given time. The number of rows is, however, many it needs to be. Each database ID could be dozens or hundreds of rows in the table. When an ID gets shoved out of the list all of the rows for that ID get nuked from the model. There is no realistic method of retrieving and reconstructing part of a book, has to be the entire thing, you don't get to rip the first chapter out and stuff it back in when you are done.
I don't understand how that is relevant to anything. In the end it all boils down to adding and removing rows. The internal representation doesn't really matter here as long as you know which row is which. If your model is hierarchical (you have IDs and each ID expands to a number of rows) then maybe you should simply have a hierarchical model (derived from QAbstractItemModel with parents and stuff) and not a flat one? Of course you will not be able to use QTableView to view it properly unless you employ a proxy model that will flatten your hierarchical model.

[d_stranz]

Of course you will not be able to use QTableView to view it properly unless you employ a proxy model that will flatten your hierarchical model.

And this is doable and has been in KDE's FlatProxyModel (http://api.kde.org/bundled-apps-api/calligra-apidocs/plan/html/classKPlato_1_1FlatProxyModel.html), which I found after a lot of trial, error, more error, and searching.

By simply adding / removing top-level IDs from the tree model, the flat model will automatically add / remove all the rows corresponding to the IDs. Your "list of lists" model should be easily mappable to a tree model.

Flick away!

[RolandHughes]

Thanks d_stranz. You wouldn't happen to know if that works on embedded Linux under -qws woudl you? I suspect it relies quite a bit on KDE.

Sadly I have not been able to put together an example which can be approved for external consumption so I'm on my own.

[wysota]

I suspect it relies quite a bit on KDE.
It's a purely Qt class, you can see the includes here: http://api.kde.org/bundled-apps-api/calligra-apidocs/plan/html/kptflatproxymodel_8h_source.html.

I don't know if the license suits you though. But even then you can implement such proxy model yourself, I've done that once or twice and it worked.

[RolandHughes]

It's a purely Qt class, you can see the includes here: http://api.kde.org/bundled-apps-api/calligra-apidocs/plan/html/kptflatproxymodel_8h_source.html.

I don't know if the license suits you though. But even then you can implement such proxy model yourself, I've done that once or twice and it worked.

Thanks,

Wouldn't happen to have a link to a useage example would you? I'm still not certain it will work. I'm actually getting close to making what I have work now, but, if it isn't working at quitting time I need to look at a completely different approach. Throw everything everybody did out and start over.

[wysota]

Wouldn't happen to have a link to a useage example would you?
No, sorry, that code was not opensource.


I'm still not certain it will work.
If your last description of the problem is precise, it will work.

I have to say I don't really like how they implemented the model. It keeps persistent indexes to all items in the source model. I don't think this is required in your case, of course it depends how often data in your items is updated and whether rows can move in the model.

[RolandHughes]

No, sorry, that code was not opensource.


If your last description of the problem is precise, it will work.

I have to say I don't really like how they implemented the model. It keeps persistent indexes to all items in the source model. I don't think this is required in your case, of course it depends how often data in your items is updated and whether rows can move in the model.

Potentially rows are continually added to the beginning which is why we must get min and max id each time we retrieve any data. All that really means though is the scroll bar which runs over IDs must be updated. Rows in the middle cannot move. Rows cannot be added at end either.

I was a bit shocked to see this in the source:

bool FlatProxyModel::removeRows(int row, int count, const QModelIndex &parent)
{
Q_UNUSED(row);
Q_UNUSED(count);
Q_UNUSED(parent);
//TODO
return false;
}

nice.

So, this requires a tree model, not a table model? Need to wait until ready to punt before digging further into this thing. Gotta love the documentation for it. Not even one snippet.

[wysota]

Potentially rows are continually added to the beginning
Within the same parent? It doesn't look like that based on what you wrote earlier.


I was a bit shocked to see this in the source:

bool FlatProxyModel::removeRows(int row, int count, const QModelIndex &parent)
{
Q_UNUSED(row);
Q_UNUSED(count);
Q_UNUSED(parent);
//TODO
return false;
}

Why? It makes perfect sense. You should be removing rows from the base model, not the proxy.


So, this requires a tree model, not a table model?
Yes. ID is the top level, their children is the lower level.


Gotta love the documentation for it. Not even one snippet.
Not even one snippet of what?

Isn't that snippety enough?
Simple Tree Model Example

[RolandHughes]

Within the same parent? It doesn't look like that based on what you wrote earlier.

The list is in reverse ID order. New records added to the database will/should appear at top of table fully exploded out. Nothing is added to them later.


Why? It makes perfect sense. You should be removing rows from the base model, not the proxy.

The //TODO comment was quite glaring. It shouldn't have been there because there was nothing TODO


Not even one snippet of what?

Isn't that snippety enough?

Of the FlatProxyModel not the Qt stuff.

[d_stranz]

I suspect it relies quite a bit on KDE.

As wysota said, no, it is purely a Qt class. I have an external 3-level hierarchical data structure that is mapped to a tree-shaped QAbstractItemModel. I display the model unchanged as a tree in a QTreeView (with a variable number of columns, depending on level), then use an adaptation of the KDE flat proxy model to expand that tree into a flat table (in a QTableView) with many more columns, one row per leaf in the tree. I've also passed the flat proxy into a sort / filter proxy that selects only certain columns for display in scatterplots (derived from QAbstractItemView). Selection works up and down the chain of proxies - select something in the tree, it gets selected in the table and the plot, select something in the table or plot, it gets selected everywhere else. It's quite a powerful programming model once you get your head around it.

I did retain the QPersistentModelIndex array from the KDE model. I don't know if it is necessary, but it seemed the most straightforward way to implement a difficult source-proxy mapping problem.


Of the FlatProxyModel not the Qt stuff.

Overlapping posts. There is no real need for an example. Take any tree model. give it to the out-of-the-box KDE flat proxy model, and hand that to an out-of-the-box QTableView. Every leaf in the tree model will appear as a row in the table view.

[RolandHughes]

Thanks to all who responded. After much tossing out and rewriting I got what we originally had (at least concpetually had) to work. The powers that be didn't want to take a plunge on something new/unknown until all possibilities were exhausted. Now it looks like the control break report they wanted with flick scrolling and database level scrolling. Yet to be tested with 4 million+ database rows (which could expand to 100 or more visible table rows) but that is another stage of the project.

[wysota]

Thanks to all who responded. After much tossing out and rewriting I got what we originally had (at least concpetually had) to work. The powers that be didn't want to take a plunge on something new/unknown until all possibilities were exhausted. Now it looks like the control break report they wanted with flick scrolling and database level scrolling. Yet to be tested with 4 million+ database rows (which could expand to 100 or more visible table rows) but that is another stage of the project.

So which approach did you finally take?

[RolandHughes]

So which approach did you finally take?

I fixed what was there. The cache class did not fit the data flow and given the follow on description of each leaf becoming a row for the flat proxy model I'm kind of glad we didn't try going down that road.

Basically, the flick scrolling tells the model to get more data when it hits the end of the scrolling region as before, but now when we get that data we don't add and remove, instead we reset model then reset the hidden scroll bar to the middle of its range, or wherever, so we can continue scrolling. The main database level scroll bar only smoth scrolls over the range of currently loaded data which is a tiny fraction of its range. Once the user exceeds that amount nothing happens until they let go of the slider. We get the current slider value and have the model reload itself using that as the starting point.

It may not seem elegant, but you don't want to do the IO for Gigs or TBs worth of binary data the user doesn't give two hoots about since they are trying to position a long way back.

I realize some people thing Apple does amazing design, but I have to tell you, flick scrolling sucks. Just trying to get 100 records back on a little embedded screen when each record expands N-hundreds of rows absolutely blows. Carpel Tunnel hear I come!