Reading large unsorted file

[anda_skoa]

I'll give a try to the the QtConcurrent.

No point in doing that, QtConcurrent is for parallelizing multiple tasks, you only have one task.

As I see You don't reset map2. So after next line map2 contains all lines from begin to current.
I think that first line in whie loop should be :

map2.clear();

map2.clear();

To copy to clipboard, switch view to plain text mode 

No, every loop iteration is a single record.

The loop needs to work with map3 and only with map3.

Something like

map3[list1[0].insertMulti(list1[1], list1[2]);

map3[list1[0].insertMulti(list1[1], list1[2]);

To copy to clipboard, switch view to plain text mode 

i.e. retrieve or create the inner map for key "list1[0]" then insert the current inner pair, allowing multiple values for key "list1[1]"

Since the result is written back into files again, one could probably even avoid the encoding and decoding and just work with QByteArray instead of QString.

Is the machine's physical RAM large enough to do that without swapping?
Such a long time sounds like as if the machine started swapping.

Cheers,
_

[GSS]

As I see You don't reset map2. So after next line map2 contains all lines from begin to current.
I think that first line in whie loop should be :

map2.clear();

map2.clear();

To copy to clipboard, switch view to plain text mode 

Well spotted. This reduced the running time to 40 minutes.

The loop needs to work with map3 and only with map3.

Something like

map3[list1[0].insertMulti(list1[1], list1[2]);

map3[list1[0].insertMulti(list1[1], list1[2]);

To copy to clipboard, switch view to plain text mode 

i.e. retrieve or create the inner map for key "list1[0]" then insert the current inner pair, allowing multiple values for key "list1[1]"

Since the result is written back into files again, one could probably even avoid the encoding and decoding and just work with QByteArray instead of QString.

Is the machine's physical RAM large enough to do that without swapping?
Such a long time sounds like as if the machine started swapping.

Cheers,
_

I'm still running with your suggestions so I cannot tell if it will take less than the 40 minutes.

Regarding memory, it doesn't seem to be an issue. The file has almost 400 MB and the application uses less than 3 GB while running. Anyway, I'm not familiar with the concepts of "swap memory".

Using QByteArray seems to be interesting but how could I then sort the entries and put everything back to files?

Thanks!

EDIT: Maybe memory is an issue. I just got a "Segmentation fault" and only half of the lines were read. Perhaps this happened now and not with the previous attempts because I have other applications running now.

EDIT2:

The loop needs to work with map3 and only with map3.

Something like

map3[list1[0].insertMulti(list1[1], list1[2]);

map3[list1[0].insertMulti(list1[1], list1[2]);

To copy to clipboard, switch view to plain text mode 

I run for only 1/4 of the lines and it took 10 minutes, which means 40 minutes as the correction suggested by Lesiok.

[anda_skoa]

Sorry anda_skoa, you're wrong.

Maybe, but I don't think so :-)

map2 is declared outside the loop thus every iteration appends next elements.

yes

Adding a clear shortened significantly duration of process.

Yes, and it also makes map3 only contain the last map2 for each key, which will alway be only one entry since it is cleared for every line in the file.

My understanding was that the programs should keep all entries for a "map3" key, not just the last pair.

Regarding memory, it doesn't seem to be an issue. The file has almost 400 MB and the application uses less than 3 GB while running. Anyway, I'm not familiar with the concepts of "swap memory".

Swap is a file or partition on disk to which the operating system "swaps" memory that is currently not used by the running application, e.g. memory used by another application which is currently not running.
See https://en.wikipedia.org/wiki/Swap_memory

Getting a system into a state where it starts swapping heavily wil slow it "to its knees", since there is a lot of I/O overhead with a slow (compared to RAM) memory device.

Using QByteArray seems to be interesting but how could I then sort the entries and put everything back to files?

I am not sure what you mean, QFile is a QIODevice subclass, it has read/write functions for QByteArray.

Cheers,
_

[GSS]

Yes, and it also makes map3 only contain the last map2 for each key, which will alway be only one entry since it is cleared for every line in the file.

My understanding was that the programs should keep all entries for a "map3" key, not just the last pair.

You are absolutely right. It has to be as you say or it will replace and not append.

I am not sure what you mean, QFile is a QIODevice subclass, it has read/write functions for QByteArray.

Sorry, I though you were referring to QFile::readAll() and work from there.

I will explore the use of QByteArray to avoid the conversion to QString.

Thank you very much!

[Lesiok]

No, every loop iteration is a single record._

Sorry anda_skoa, you're wrong. map2 is declared outside the loop thus every iteration appends next elements. Adding a clear shortened significantly duration of process.