Visualize BIIIG data

[anjohn]

Heyho,

I'm working on my master thesis and it is about Delaunay-Triangulation. Now I have a huge set of triangles in 16384 files. A small number of files can be rendered with QGraphicsView and QGraphicsScene:
9349
Unfortunately 1000 of these files take more than 1GB of memory.

Do you have any smart ideas how to visualize all files at once?

I realized that I need to convert each file into a QPixmap. But I think there might be a problem: Each file doesn't have a straight scene rect, there are allways small noses at the edges because the points that were converted to this triangulation don't lie on a grid! In fact if I combine a number of pixmaps, I get some unpleasent white stripes between both pixmaps.

I don't know how to explain it better ... sorry!

Anyone an idea?

Thank you!

best regards,
anjohn

[rockdemon]

I realized that I need to convert each file into a QPixmap

Or is it better to create 1 image or a few images that allow the user to see an overview and then drill down?

[anjohn]

Would be nice but I cannot create one image because loading all 16384 files would take more than 16GB of RAM. I allways can convert just a small number of files into an image.

I think I have to build one image by combining several images ...

[rockdemon]

Is this an instance where you need to do 2 passes of the information? Maybe first pass somehow order the information so you can then go through in one pass to draw it or at least know which information is important? Surely you would load each file in turn, not all 16384 at once?

Added after 7 minutes:

are you storing them as individual bitmaps or a list of points? The latter must be more use?

[anjohn]

It is ordered somehow. The Delaunay-Triangulation was processed as a stream with spatially ordered points at the stream input. And in this order the output is written.

Of course I can select only some files and show their content like in the picture I uploaded above.

But there must be a way to compute a single picture as overview for my whole triangulation. Something like loading 50,000 triangles, then converting them to pixels and storing them in my view, and then deleting them for having space for the next 50,000 pixels...

[rockdemon]

right - so do you mean the resulting image would be 16gb rather than the size of the individual files? If so i guess you need to composite the image file straight to the disk? do a first pass through all files and find the maximum extents of the bitmap, allocate a file and populate it by writing the bytes directly?

If you mean the files add up to more than 16gb, do you have to load them all together? Is it possible to open each individual file, draw to a bitmap, then close disposing of the data, then load the next file or are the files independent?

Rich

[anjohn]

right - so do you mean the resulting image would be 16gb rather than the size of the individual files?
No the files take a few hundred MB. The B-Tree of the scene class or of the view class of qt takes so much memory. I think it is possible to paint onto a widget with the QPainter class uses less memory.
But this streaming triangulation should work with point clouds that are several GB big, these 16000 files were created by streaming a point cloud of 300 MB size.


do a first pass through all files and find the maximum extents of the bitmap, allocate a file and populate it by writing the bytes directly?

Should try it like this, although I don't know how to get this to work.

I would like to render pixels directly instead of storing triangles in a data structure.

[rockdemon]

I think it is possible to paint onto a widget with the QPainter class uses less memory.

yes - definitely the best way...

[anjohn]

My observations say that painting on a QPixmap has the lowest footprint. I had 80 MB/s of input, triangles of 1,4 GB were rendered. The program overall didn't occupy more than 100 MB of memory ...
So painting on a pixmap is very cool for rendering a static image using billions of triangles in a stream.

Didn't expect that. Here is the bird eye view of one square kilometer of the TU Dortmund, Germany: ;-)
9356

[rockdemon]

Awesome :) Result!