The best way to read big file ??

[jesse_mark]

Hello ,
Im trying to read a big file, but i want to find what is the best way to do so.
I ma not worry about the memory as it will be used on machines that have big memory enough to read the whole file at once.

right now im using textStream to read all in one time, but it still not fast and take time to do so, is there a way to read the file in parallel into QStringList and still keep the order of the lines ??

some are saying I can not get speed up as I am limited with the disk header to read, and having multiple threads reading from different location in the file will not benefit my performance much.

Please if anyone have an idea of what is the best way to read the file in memory in parallel or faster way help me.

Thank you.

[amleto]

there's a good answer on stack overflow

[MarekR22]

All depends on details.
If it is some kind of dictionary then and you are reading a text file, then conversion from file specific encoding to unicode may create overhead.
To reduce that you can disable detection of encoding. Allocation of large number of QStrings is also qute costly.

If yuo use this data in some kind of data model, then you could load file literally and do conversion to QString every time you are requesting some value (loading vary fast but access to data a bit slower).
There are lots of other tricks to tweak such things, but this depends on specific application.

[sulliwk06]

Have you considered reading the file in a separate thread and going ahead with whatever processing you plan to do before the entire file has been read?

[Braunstein]

Hi,

I read such files binary by using standard C++ streams. In order to improve speed i use my own stream buffer and reading in chunks (with buffer size). The interpretation of line ends can be done later directly on the buffer.
Up to now i don't use multithreading. It makes only sense for me if you need some further interpretation (parsing) of the data.

[anda_skoa]

You could experiement with QFile::map() to slide a window over the file, i.e. map a section of the file into memory, take whatever data you need, map the next section.

Mapping a file or parts of a file into memory allows the operating system to coordinate file system buffers, optimize read/writes, avoid copying (from FS buffer to application memory), etc.

Cheers,
_