Need to show SQL data in tree view with lazy loading.

[DURGAPRASAD NEELAM]

Hi,

I have a huge data in mysql data base. The data in data base as follows for example.

Level	   Date	        name	       rank
0	   1/1/15	 Rajesh	        1
1	   1/1/15	 Krish	        2
2	   1/1/15        Ram	        3
1	   1/1/15        Krish 	        4
0	   1/1/15        Krish	        5
1	   1/1/15	 Krish	        6
1	   1/1/15	 Krish	        7	   
0
1
1
2
2
1
.
.
.

Level	   Date	        name	       rank
0	   1/1/15	 Rajesh	        1
1	   1/1/15	 Krish	        2
2	   1/1/15        Ram	        3
1	   1/1/15        Krish 	        4
0	   1/1/15        Krish	        5
1	   1/1/15	 Krish	        6
1	   1/1/15	 Krish	        7	   
0
1
1
2
2
1
.
.
.

To copy to clipboard, switch view to plain text mode 

So, From the above table I need to build the tree view based on level column.
All zeros are parents followed by 1’s are child's of 0's above that, and 2’s are child's of 1’s above that (tree should be formed as below).



Tree:

0
---------1
---------------------------2
----------1
0
-----------1
-----------1
0
-----------1
-----------1
---------------------------2
---------------------------2
------------1

I have huge set of records, more than 20 lak , here I have to do lazy loading i.e. I should not read all the data and put in some container and then building parent and child relationship.

Lazy Loading: instead of reading and putting all the data memory, read the data which is currently viewing.

Here I have to do lazy loading, I want read the data of how many tree items I am showing on view currently.
I am not able to get how to build parent and child relationship with this requirement,
Any suggestions for the above requirement ???

Requirement = Mysql data base (huge set of data) + Tree view + Lazy loading. (I am using Qt 4.8.5)


Thanks in advance.

[wysota]

Implement QAbstractItemModel::canFetchMore() and QAbstractItemModel::fetchMore() for your model.

[DURGAPRASAD NEELAM]

Hi wysota,

Thanks for the replay. I have gone through the functions you have suggested above, and I have gone through the Fetch-more example presents in the Qt example's.
After this I came to the conclusion (May be I'm wrong, correct me if I'm) that, at the end (as we go down with scroll bar till end) we will have all the data into memory (All the records will be present in memory), Right ?

If so, Could you please suggest a way to implement this with out loading all the data(even at the end).. at any point of time I want to load only the data which I'm showing in view.

Thanks :-)

[wysota]

Hi wysota,

Thanks for the replay. I have gone through the functions you have suggested above, and I have gone through the Fetch-more example presents in the Qt example's.
After this I came to the conclusion (May be I'm wrong, correct me if I'm) that, at the end (as we go down with scroll bar till end) we will have all the data into memory (All the records will be present in memory), Right ?

It depends how you implement your model. It is entirely up to you to decide which data to keep in memory and which to discard.

If so, Could you please suggest a way to implement this with out loading all the data(even at the end).. at any point of time I want to load only the data which I'm showing in view.

Use QContignousCache as storage in the model and fetch data from the remote server if user requests range of data which is not in the cache. Note however this will cause stalls.

[DURGAPRASAD NEELAM]

Hi, Some how I could able to read and show the data in view with lazy population, But has below problems.

I have implemented canFetchData & featchData functions
I am using proxy model too, setting below TreeModel as a sourse model;

bool TreeModel::canFetchMore(const QModelIndex &parent ) const
{
	qint64 totalRowCount = sql_reader->getTotalRowsNumber(); //returns Total no.of rows  
 
	if(totalRowCount > lastreadRowNumber)   //I am storing Last read Row
	{
		return true;   //still not reached end of the table
	}
 
	return false;
}

bool TreeModel::canFetchMore(const QModelIndex &parent ) const
{
	qint64 totalRowCount = sql_reader->getTotalRowsNumber(); //returns Total no.of rows  

	if(totalRowCount > lastreadRowNumber)   //I am storing Last read Row
	{
		return true;   //still not reached end of the table
	}

	return false;
}

To copy to clipboard, switch view to plain text mode 

void TreeModel::fetchMore(const QModelIndex& parent)
{
	qint64 totalRows = sql_reader->getRowNumber();
    	int    readBulk  = 1000;    //reading 1000 rows on demand
 
    if( totalRows  <  ( readBulk + lastreadRowNumber) )
    {
    	readRows = totalRows - lastreadRowNumber; //adjustment for last few rows
    }
 
 
	beginInsertRows(QModelIndex(), m_lastreadRowIndex, m_lastreadRowIndex+readRows-1);
 
	emit fetchMoreData(lastreadRowNumber, m_lastreadRowIndex+readRows);
 
	//by emiting signal, I am getting data from sql database of range above
	//with that data I am forming My Parent and child relationship 
	//and saving lastreadRowNumber	
 
        endInsertRows();
}

void TreeModel::fetchMore(const QModelIndex& parent)
{
	qint64 totalRows = sql_reader->getRowNumber();
    	int    readBulk  = 1000;    //reading 1000 rows on demand

    if( totalRows  <  ( readBulk + lastreadRowNumber) )
    {
    	readRows = totalRows - lastreadRowNumber; //adjustment for last few rows
    }


	beginInsertRows(QModelIndex(), m_lastreadRowIndex, m_lastreadRowIndex+readRows-1);

	emit fetchMoreData(lastreadRowNumber, m_lastreadRowIndex+readRows);
	
	//by emiting signal, I am getting data from sql database of range above
	//with that data I am forming My Parent and child relationship 
	//and saving lastreadRowNumber	

        endInsertRows();
}

To copy to clipboard, switch view to plain text mode 

//With this I am able to load all the data, But when loading a data, I am getting below messages on console

QSortFilterProxyModel: invalid inserted rows reported by source model

//When I try to move and exapand the tree i am able to do it for 2/3 times then tool suddnley geting crashed by showing
Error message

QSortFilterProxyModel: invalid inserted rows reported by source model


When I tried: beginInsertRows(parent, m_lastreadRowIndex, m_lastreadRowIndex+readRows-1);
I got below error and tool got crashed

QSortFilterProxyModel: invalid inserted rows reported by source model
QSortFilterProxyModel: index from wrong model passed to mapFromSource
QSortFilterProxyModel: index from wrong model passed to mapFromSource
QSortFilterProxyModel: index from wrong model passed to mapFromSource



Could you please let me know, what is the wrong thing I am doing here ?

[wysota]

First your implementation of canFetchMore and fetchMore are incorrect if what you are building is a tree model because you are neglecting the parent index. Second of all if you want to lazy populate the data and only keep the data the user sees in the view, how do you expect this to happen if you apply a sorting model on top of that? If sorting requires the first and last row to be nwighbours then obviously your base model will need to contain both the first and last item and thus it will need all other items as well.

[DURGAPRASAD NEELAM]

1. as you said, isn't it possible to apply sorting/filtering with proxy Model when we are doing lazy load ? , If so, Is there any way to achieve this ??

2. and As you observed I am loading data step by step, this I will get back as i am still exploring and implementing.

Thanks :-)

[wysota]

1. as you said, isn't it possible to apply sorting/filtering with proxy Model when we are doing lazy load ? , If so, Is there any way to achieve this ??

Lazy initialization is not the problem. Your other requirement is.

[DURGAPRASAD NEELAM]

1st requirement is lazy loading.
2nd is, need to apply filtering only on top of this(sorting is not required).

[DURGAPRASAD NEELAM]

Hi, I have implemented fetchMore and canFetchMore() functions as below.
But still I am getting the same Error: QSortFilterProxyModel: invalid inserted rows reported by source model;

Please let me know where to change the code, I have lost somewhere and tree view is bit confusing for me.

bool TreeModel::hasChildren ( const QModelIndex & parent ) const
{
    DebugTreeItem *parentItem;
    if ( parent.column() > 0 )
        return false;
 
    if ( ! parent.isValid() )
        parentItem = m_rootItem;
    else
        parentItem = static_cast<DebugTreeItem*>( parent.internalPointer() );
 
    return parentItem->hasChildren();
}
 
bool TreeModel::canFetchMore(const QModelIndex &parent ) const
{
       if(!parent.isValid())
		return true;
 
        if(m_totalRowCount > m_lastreadRowIndex)    //I am storing lst read row value
        {
                return true;
        }
 
        return false;
}
 
void TreeModel::fetchMore(const QModelIndex& parent)
{
 
    //to fect readRows od records (rows) from sql
    int	   readRows  = 1000;
 
    if( m_totalRowCount  <  ( readRows + m_lastreadRowIndex ) )
    {
        readRows = m_totalRowCount - m_lastreadRowIndex;
    }
 
    //will get me last row number
    qint64 lastRow = m_lastreadRowIndex ;
 
    emit fetchMoreData(readRows);   //will fecth records and form tree
 
    qint64 parentsInFetchedData = getZerosInFetchedData();
 
 
    //i have writtem logic so that, next inserted row should be 0 (so parent should be m_rootItem always (it is my top most parent))
    //so I am passing m_rootItem as root and last inserted row number, num of row to insert. 
    beginInsertRows(QModelIndex(), lastRow , lastRow+parentsInFetchedData -1 );  //same result for beginInsertRows(parent, lastRow , lastRow+parentsInFetchedData -1 ); 
    endInsertRows();
}

bool TreeModel::hasChildren ( const QModelIndex & parent ) const
{
    DebugTreeItem *parentItem;
    if ( parent.column() > 0 )
        return false;

    if ( ! parent.isValid() )
        parentItem = m_rootItem;
    else
        parentItem = static_cast<DebugTreeItem*>( parent.internalPointer() );

    return parentItem->hasChildren();
}

bool TreeModel::canFetchMore(const QModelIndex &parent ) const
{
       if(!parent.isValid())
		return true;

        if(m_totalRowCount > m_lastreadRowIndex)    //I am storing lst read row value
        {
                return true;
        }

        return false;
}

void TreeModel::fetchMore(const QModelIndex& parent)
{
   
    //to fect readRows od records (rows) from sql
    int	   readRows  = 1000;

    if( m_totalRowCount  <  ( readRows + m_lastreadRowIndex ) )
    {
        readRows = m_totalRowCount - m_lastreadRowIndex;
    }

    //will get me last row number
    qint64 lastRow = m_lastreadRowIndex ;

    emit fetchMoreData(readRows);   //will fecth records and form tree
    
    qint64 parentsInFetchedData = getZerosInFetchedData();


    //i have writtem logic so that, next inserted row should be 0 (so parent should be m_rootItem always (it is my top most parent))
    //so I am passing m_rootItem as root and last inserted row number, num of row to insert. 
    beginInsertRows(QModelIndex(), lastRow , lastRow+parentsInFetchedData -1 );  //same result for beginInsertRows(parent, lastRow , lastRow+parentsInFetchedData -1 ); 
    endInsertRows();
}

To copy to clipboard, switch view to plain text mode 

[wysota]

Your canFetchRow() still makes no sense. Lines #17-18 say "If you are asking about top-level items, I always have more of them". The error you are getting is probably related rather to an invalid implementation of index() rather than fetch-more.

[DURGAPRASAD NEELAM]

Finally I could do lazy loading, but not as I expected.

I am reading X amount of rows and building parent and child relationship. as we keep on scroll down I am getting remaining data by using canFetchMore() and fetchMore() functions.
once I get the data I am appending this data to existing data by forming parent and child relationship.

1. The 1st problem is, I am not able to delete the old data, If I tried to delete old data and keep only new data I am not able to see any scroll bars.
2. the 2nd problem is when I have huge data, As the user keep on scroll down, scroll bar is becoming small. if we have a huge amount of data it becoming difficult for the user to really go to the end of the view and wait for the result to fetch and once results got fetched then come to know that there are some more records are available and still scroll down.

for the second problem I tried to create vertical scroll bar and set a value for it and then set it to the view. but It did not workout.
could you please suggest some solution for this 2 problems.




Below is the code snippet, please suggest how to delete old data and maintain only current records in a memory.

bool TreeModel::canFetchMore(const QModelIndex &parent ) const
{
        if(m_totalRowCount > m_lastFetchedRowNumber)
        {
        	return true;
        }
 
        return false;
}
 
void TreeModel::fetchMore(const QModelIndex& parent)
{
    int	   readRows  = 1000;  //reading 1000 records/rows every time
 
    if( m_totalRowCount  <  ( readRows + m_lastFetchedRowNumber ) )  //boundary condition
    {
        readRows = m_totalRowCount - m_lastFetchedRowNumber;
    }
 
    qint64 l_lastInsertedRow = m_lastreadRowIndex;
 
 
 
//m_parent is my parent, I tried by doing this;  delete m_parrent; m_parrent = createParentItem(); // but it did not workout.
 
 
    emit fetchMoreData(readRows);                         // will fetch readRows no.of rows, may vary depends on logic(next fetch would start from a row which Level is 0) I have written.
 
   // I have written a logic so that every time I will get 0 as 1st row, so every time I insert data, parent must me QModelIndex()
 
    beginInsertRows(QModelIndex(), l_lastInsertedRow, m_lastreadRowIndex-1); 
    endInsertRows();
 
}

bool TreeModel::canFetchMore(const QModelIndex &parent ) const
{
        if(m_totalRowCount > m_lastFetchedRowNumber)
        {
        	return true;
        }

        return false;
}

void TreeModel::fetchMore(const QModelIndex& parent)
{
    int	   readRows  = 1000;  //reading 1000 records/rows every time

    if( m_totalRowCount  <  ( readRows + m_lastFetchedRowNumber ) )  //boundary condition
    {
        readRows = m_totalRowCount - m_lastFetchedRowNumber;
    }

    qint64 l_lastInsertedRow = m_lastreadRowIndex;



//m_parent is my parent, I tried by doing this;  delete m_parrent; m_parrent = createParentItem(); // but it did not workout.


    emit fetchMoreData(readRows);                         // will fetch readRows no.of rows, may vary depends on logic(next fetch would start from a row which Level is 0) I have written.

   // I have written a logic so that every time I will get 0 as 1st row, so every time I insert data, parent must me QModelIndex()
   
    beginInsertRows(QModelIndex(), l_lastInsertedRow, m_lastreadRowIndex-1); 
    endInsertRows();

}

To copy to clipboard, switch view to plain text mode 

Thanks in advance.

[wysota]

1. The 1st problem is, I am not able to delete the old data, If I tried to delete old data and keep only new data I am not able to see any scroll bars.
2. the 2nd problem is when I have huge data, As the user keep on scroll down, scroll bar is becoming small. if we have a huge amount of data it becoming difficult for the user to really go to the end of the view and wait for the result to fetch and once results got fetched then come to know that there are some more records are available and still scroll down.

for the second problem I tried to create vertical scroll bar and set a value for it and then set it to the view. but It did not workout.
could you please suggest some solution for this 2 problems.

Your model needs to report its full size to the caller but it doesn't mean you have to actually have all the data at hand. As I said before you can use e.g. QContignousCache.

[DURGAPRASAD NEELAM]

Hi, Could you please help me out for this :

http://www.qtcentre.org/threads/6232...281#post276281