how to manipulate a very large data using QT?

[aurora]

Thanks a lot Chris...

• Is that one 1 gb blob of text or 100 million little bits of text? (War and Peace vs. a phone book?)
• How many "tables" of 1GB?
• What do you need to be able to do with that text?
• How quickly?
• If there are some kind of "records" in here how do they relate to each other?
• Does the data need to be shared between multiple users concurrently? How many?
• Is the data read-only or updatable?
• Does the data need to be controlled/protected by some sort of permissions system?
• Do you need to be able to distribute the database data or processing load between machines or sites?

.

• Its 100 million lines all with same length
• Hundreds of tables around 1gb
• I should able to search in those tables(basically)
• Speed should be reliable
• No concurancy
• All tables are read only
• Not distributed system...just for single stand alone machine

[wysota]

If you have text files, why don't you use those files directly? What's the benefit of moving the data into separate storage? What kind of "search" do you have in mind? Words? Patterns? Records? Is the content in each file ordered somehow?

[aurora]

If you have text files, why don't you use those files directly? What's the benefit of moving the data into separate storage? What kind of "search" do you have in mind? Words? Patterns? Records? Is the content in each file ordered somehow?

my target was to load a 1gb text file in table widget and to perform searching operation on those files, but i was unable to perform that operation.
So i thought, if i can store it in a database i can show part of the file(or the user requested part ) in the table view..
Is the way i have choosen is wrong? is there any alternative to perform this?

[ChrisW67]

my target was to load a 1gb text file in table widget and to perform searching operation on those files, but i was unable to perform that operation.

It was always a stretch to get a 1GB collection of lines into memory for a single widget on a 32-bit machine. Even less likely if there are "Hundreds of tables around 1gb," you tried to load.

So i thought, if i can store it in a database i can show part of the file(or the user requested part ) in the table view..

Possibly, we cannot tell. You still haven't told us anything about the structure of the file content, how the user might search it, what constitutes a "part of the file", etc. It might be something like:

• Open file, seek to a byte offset, read a 4kB "page", and display that, or
• Open file, seek to a particular line number, read a specified number of lines, and display that, or
• Open file, select all paragraphs matching a pattern, and display those, or
• Open file, select all lines matching a pattern, and display 5 lines either side of each, those, or
• Open file, select all lines matching a pattern, interpret the data in the lines to compute whether to display them, and display those, or
• Open file, select all lines matching a pattern, find a set of related lines, and display those,
• Open a file, open another, find the common/different lines, and display those, or...

I don't know how you expect us to know.

Is the way i have choosen is wrong? is there any alternative to perform this?

It is neither right nor wrong. There are plenty of options but only you have the information on which to assess suitability.

[wysota]

So i thought, if i can store it in a database i can show part of the file(or the user requested part ) in the table view..

I fail to see what a database would help you with here. A text file is also a sort of a database. If you want to visualize part of the file then do just that -- load the part you need and display whatever you need.