Llarge set of data in QML List view/C++ List model

[ashwasimha]

Hello All,

In my project, I need to display list of some data using QML List view.And, the QT version I am using is QT 5.0.2. Since, scroll bar view is not available in this version of Qt, I managed to implement it by following this example.

Now, the data source is an other component which has provided C++ APIs to request for range of data. And the number of elements in the data set can be huge (in terms of tens of thousands).

I have followed below steps to achieve my requirements :

1. I have a created a C++ class that inherits from QAbstractListModel and I have overridden rowCount, data, canfetchmore and fetch more functions.
2. And in the QML, I have referred the instance C++ as model.
3. Whenever fetchmore() function is called, I am requesting next set of data using C++ APIs that are provided by another component and appending the same to my list model.

So far, it is good. But I need few clarifications -

1. If I keep appending the data to my list model,then I will end up storing thousands of elements in the list (I don't want to do that). I want to limit the number of elements in the list and use it as a ring buffer. Is this possible?
2. QModelIndex parameter in fetchmore is received as invalid Index. Why?
3. Assume a case : alphabets are mentioned on scrollbar, and when user selects Y on scrollbar, I should fetch elements that starts with Y. Here, I can fetch the data from my external component and reset the list model with new data. But, if I do that,then my list will start with Y-elements and I will not be able to scroll up to get previous elements. -- Is there any other way to achieve this?
4. Also, my other component takes considerable time if I ask huge set of data (for example in cases as mentioned in point 3) - Any suggestions to make a better design to handle this case?
5. Is there any way, other than fetchmore(), to update the model dynamically.

I have spent good amount of time on google to get answers to my questions and I have successfully FAILED.

Any help appreciated. Thanks in advance.

[anda_skoa]

If I keep appending the data to my list model,then I will end up storing thousands of elements in the list (I don't want to do that). I want to limit the number of elements in the list and use it as a ring buffer. Is this possible?

The model only needs to be able to provide access to data, it doesn't have to store the data.
If you actual data allows you to query for specific data then the model can use that as well.

QModelIndex parameter in fetchmore is received as invalid Index. Why?

A list is a flat structure, there are not parent nodes.

Assume a case : alphabets are mentioned on scrollbar, and when user selects Y on scrollbar, I should fetch elements that starts with Y. Here, I can fetch the data from my external component and reset the list model with new data. But, if I do that,then my list will start with Y-elements and I will not be able to scroll up to get previous elements. -- Is there any other way to achieve this?

It depends on what you use the reset for.
When you reset the model and only provide Y elements, then that is all the data you have as far as the view is concerned.

If you want to reset the model to only see a specific subset, e.g. to make a scrollbar not work on the whole set but only on the subset, then you need to detect when you need to switch out of that "detail" mode.
E.g. some UI to "go back" to the full list, or header/footer items that trigger this return to the full mode.

If you reset for some other reason the maybe the reset is not the right way to achieve the target result.

Also, my other component takes considerable time if I ask huge set of data (for example in cases as mentioned in point 3) - Any suggestions to make a better design to handle this case?

That's difficult to answer generically, depends on what these components to and how they interact with the data.

Is there any way, other than fetchmore(), to update the model dynamically.

A model can at any given times annouce that it is adding or removing rows, or announce that data of existing rows has changed.
Adding/removing can be done through protected helper methods, data change is notified via the dataChanged() signal.

Cheers,
_

[ashwasimha]

Hi anda_skoa,

Thank you for your response.

The model only needs to be able to provide access to data, it doesn't have to store the data.
If you actual data allows you to query for specific data then the model can use that as well.
_

Agreed. But, my external component takes considerable time to retrieve the data. Hence, a buffer of data is maintained in model to achieve smooth scrolling.

A list is a flat structure, there are not parent nodes.
_

Ok. Understood! Thanks!

It depends on what you use the reset for.
When you reset the model and only provide Y elements, then that is all the data you have as far as the view is concerned.

If you want to reset the model to only see a specific subset, e.g. to make a scrollbar not work on the whole set but only on the subset, then you need to detect when you need to switch out of that "detail" mode.
E.g. some UI to "go back" to the full list, or header/footer items that trigger this return to the full mode.

If you reset for some other reason the maybe the reset is not the right way to achieve the target result.

That's difficult to answer generically, depends on what these components to and how they interact with the data.


A model can at any given times annouce that it is adding or removing rows, or announce that data of existing rows has changed.
Adding/removing can be done through protected helper methods, data change is notified via the dataChanged() signal.

Cheers,
_

My case is like this,

1. Data source is an external component (another process) that provides data in chunks (It has provided APIs to request data in chunks). Number of elements in the source can be huge (in terms of tens of thousands).
2. I have a class (say CDataFetcher) that interfaces with the external data source and fetches the data on need basis.
3. Now, CListModel is my model that has inherited from QAbstractListModel.

class CListModel : public QAbstractListModel
{
    /*other methods and data members....*/
    int rowCount(const QModelIndex &parent = QModelIndex())
                                                    const Q_DECL_OVERRIDE;
    QVariant data(const QModelIndex &index, int role = Qt::DisplayRole)
                                                    const Q_DECL_OVERRIDE;
    QHash<int ,QByteArray>roleNames() const Q_DECL_OVERRIDE;
 
    QStringList   myListData;
 
};

class CListModel : public QAbstractListModel
{
    /*other methods and data members....*/
    int rowCount(const QModelIndex &parent = QModelIndex())
                                                    const Q_DECL_OVERRIDE;
    QVariant data(const QModelIndex &index, int role = Qt::DisplayRole)
                                                    const Q_DECL_OVERRIDE;
    QHash<int ,QByteArray>roleNames() const Q_DECL_OVERRIDE;

    QStringList   myListData;

};

To copy to clipboard, switch view to plain text mode 

Here myListData is the buffer of data that I am talking about. As the data can be huge, I am implementing a logic to depict myListData as ring buffer.

Now, if, user scrolls the list either by dragging up or down on the list (please note this action is not done using the scroll bar), then I can provide the data to list view that is available in myListData. If there is no data available in myListData, then,I will fetch the data using CDataFetcher.

QVariant data(const QModelIndex &index, int role = Qt::DisplayRole)
{
      /*check if the index is in with in the number of elements in myListData*/
      /*If yes, return the data*/
      /*else, fetch the data from local copy in CDataFetcher*/
}
 
QStringList CDataFetcher::FetchData()
{
     /*This class works in a different thread*/
     /*Local copy is always maintained in this class*/
     /*If the copy is already used by model, then this class fetches the data and makes a local copy*/
}

QVariant data(const QModelIndex &index, int role = Qt::DisplayRole)
{
      /*check if the index is in with in the number of elements in myListData*/
      /*If yes, return the data*/
      /*else, fetch the data from local copy in CDataFetcher*/
}

QStringList CDataFetcher::FetchData()
{
     /*This class works in a different thread*/
     /*Local copy is always maintained in this class*/
     /*If the copy is already used by model, then this class fetches the data and makes a local copy*/
}

To copy to clipboard, switch view to plain text mode 

This way, though I am not storing the entire data in memory, I am achieving smooth scrolling by maintaining small chunks of data. This works fine for linear scrolling on the list.


Now, in an other case (case 3 as I mentioned in my previous post), suppose user is viewing elements that starts with alphabet "C" and user selects alphabet "Y" on the scrollbar, I should show elements that starts from "Y". Here, this behaviour is similar to contact list behaviour in iphone.
I can get list of elements that starts with "Y", but if I have to show it, (as per my design) I have to buffer it in my model. To do this, I will have to erase previous contents and store my current contents.

When I do this, list view assumes that my list starts with elements "Y" and when user scrolls up on the list to view elements that starts with "X" or "W" or "V"..., I will not get any request to get previous data How can I fix this? Similar problem is explained in this forum.

Ideally, if I had an API like scrollToItem or JumpToIndex, I would not have ran in to this problem.

Let me know your comments on this. Please note that QT version that I am using 5.0.2

[anda_skoa]

You must have really long strings if a couple dozend thousand is "huge".

I see a couple options:

1) you maintain the cache differently, e.g. by caching the data in a file and maintaining just indexes into the file inside the model
2) you maintain a "sliding window" which has more rows than can possibly be shown. when the view starts requesting outside of the cached range, add new rows on that end and remove from the other
3) similar to 2 but instead of adding/removing rows change content of existing rows to achieve the same end result

Cheers,
_

[ashwasimha]

Hi anda_skoa,

Apologies for delay in response. I was in vacation.

You must have really long strings if a couple dozend thousand is "huge".

I see a couple options:

1) you maintain the cache differently, e.g. by caching the data in a file and maintaining just indexes into the file inside the model
2) you maintain a "sliding window" which has more rows than can possibly be shown. when the view starts requesting outside of the cached range, add new rows on that end and remove from the other
3) similar to 2 but instead of adding/removing rows change content of existing rows to achieve the same end result

Cheers,
_

Thank you for your options. Yes, I have followed option that is similar to "option-3" that you have mentioned.

Brief explanation of strategy that was followed (hopefully it will help someone) :
There will be a separate thread that requests the data from other component and fills in to a buffer(buffer size if configurable,for ex : an array of 100 elements). When qt framework requests for data using "data(const QModelIndex &index, int role = Qt:: DisplayRole)" function, get the data that is available in buffer. I have implemented a sliding window logic to make sure that data is always available in the buffer when user is scrolling up/down. This way I managed to have only 100 elements in memory at any instance of time.

Thank you for your support.