Need lazy load approach for treeview & wanted to maintain only viewing items in mem

[prasad_N]

Hi,

I wanted to build a tree with lazy loading (not lazy population), The problem I have is, I have lots of data (around 10 billion to 100 billion rows) building a tree with this much data is obviously crashing tree as (memory issue). So I wanted make lazy loading (At a time I want to maintain those items which I show in view).

I have few doubts with this approach

1. I can use canFetchMore() & fetchMore() functions. when I scroll down I want to get the below items (which is possible with this functions) and I want to delete items which gone out of view.
Loading data when scroll down can get from above functions but how to get the data when I scroll up ??

2. how can I delete items which gone out of view when I scroll ??

3. how can I maintain scroll bar size with this approach ??

4. last but one is I wanted to implement filter also on top of this.

I have seen fetchMore() example but its for list & where items are not getting deleted when gone out of view. I want to get same behavior but with a tree & want to delete items which gone out of view & want to maintain scroll bar properly.

any suggestions please.

[ars]

Hello,

I ran into some problems when displaying more than 72 million rows in a table (the view only showed garbage). The reason here was that internally the view components do all computations with integer (signed 32 bit) and the default row height is 30, so the view produces integer overflows when working with that amount of data. There is already a bug note on this in the Qt bug tracker.

For working around this I have chosen the following approach (maybe this could help you too):
1. I divided my model into "pages" consisting of up to 60 million rows. The view will display only the currently loaded "page" and the scrollbar is used for scrolling inside this "page".
2. I added a - and + button at the top and bottom of the view vertical scrollbar. Clicking the + button will move the model page by half page size down and pressing - button will move it by half page size up. The view will get updated when clicking those buttons.

In your situation you can unload old / load new data into the model when the +/- buttons are clicked.

Regards
ars

[prasad_N]

Hello,

I ran into some problems when displaying more than 72 million rows in a table (the view only showed garbage). The reason here was that internally the view components do all computations with integer (signed 32 bit) and the default row height is 30, so the view produces integer overflows when working with that amount of data. There is already a bug note on this in the Qt bug tracker.

Hi thanks,
you mean, can't we show/handle more than 70+ million items in view ?

Regards,
Durga.

[ars]

Hello,

have a look at https://bugreports.qt.io/browse/QTBUG-28631 describing the bug in Qt 4.7 and 4.8. They have not fixed it.

I also tried working around this limitation by using a tree view instead of of a table view and group my data into 60 million sub rows that get expanded or collapsed by the user clicking on the parent item. Here the parent item represented a range of rows. This was so incredibly slow that it was unusable.

Another point is how to use the scrollbar when navigating through 10 billion rows. Let's assume the scrollbar covers 1000 pixels on the screen, then each scrollbar position maps to 10 million rows of data. For navigating through such big data sets, I think a scrollbar alone is not adequate. This also applies to my application. I have a graphical representation of the data (trace) where the user can put a marker at some point that looks interesting and then may scroll the table to the position of the marker to see the raw data in detail.

Hope that helps you a little bit.

Regards
ars

[prasad_N]

Thanks, I will look into the link.

Edited : gone through the link, Looks like they have not fixed the bug instead they suggested to build 64-bit application, my application is 64 bit any how .

BTW: how did you do memory management in this case, As you have huge row & we need to build tree before population it onto view.
So did you manage the memory. did you create entire tree (items) and populate it or in some other way ??

[ars]

I have 2 different applications for this. In one application the data are stored in a file, all with equal record size. So if I know a row number, I can compute the location in the file and load the data from there. Of course I do some caching of data to improve performance.

In the second application a single column of data is stored in memory. From the context of the application I know that there is a limitation of 200 million data points (each a double) and I have to keep those data in memory (simple std::vector) anyway because a bunch of numerical algorithms are applied to the data basically reducing the millions of double values into pass/fail results. So I need up to 1.6GB of ram. Allocating this is no big problem in today PCs with 64 bit OS and 8 or 16GB ram. That's for the first column in my table. There are further columns, but the elements of these columns are computed on the fly from the data in the first column when the data in the other column gets read by a view. Therefore I do not need to allocate memory for these columns.

As I mentioned before, I store my data in a std::vector object. The model used by my table view does not store data by itself. Instead, it gets a reference to my data vector and accesses the vector for reading the data. That prevents duplicating data just for displaying them in a table. So in my case, the Qt table model serves as a "glue" between my data (std::vector) and the Qt table view objects.

Best regards
ars

[prasad_N]

Oops.. Thanks for the detailed answer.
In my case this may not work because before populating a tree, need to build the tree this is not a case with table or list view. This is the reason I wanted to go with lazy population model and left with the above doubts.

[prasad_N]

Any suggestions from others ???