QList<QBbyteArray> & QHash memory usage.

[jesse_mark]

Hello everyone,


I am trying to read a text file and process its content.
the file i am using has 10,000,000 lines each line is 80 byte in lenght, so its about 800M in size.

The first step
I read the file line by line and push it into a QList <QByteArray>. <<== declared private inside the a method.

Then
I itarate over the QList parse each line and get from it Key and lineInfo to push into a QHash <QString, lineInfo> <=== declared public in header
The key is a QString====> the key lenght 10 bytes in this case.
the lineInfo is a class with eight private members all are "int"

when this process is done i check the memory using "top" command from a terimnal the application is using about 3g of memory.
any Idea why its using that much of memory and how to reduce that ??


best regards,

ps:
Enviroments:
Qt 4.8
linux redhat 6.3

[ChrisW67]

Well, for a start you should not store the entire file in RAM (That is easily a gigabyte on its own). It sounds like you can parse each line as it is read from file and drop the relevant bits into your QHash in one motion.

Have you considered using a disk-based store instead of the in-memory hash? Hash key lookups are something that the dbm (http://en.m.wikipedia.org/wiki/Dbm) family of libraries do for a living, but any RDBMS (including the zero cost Sqlite) will make decent job of it.

[jesse_mark]

storing the file in memory is ok in my case as i have a big ram and i dont mind that if it stays the same size as the file size,
but after loading the file the memory used is bigger than the file size its 1400M and my file is only 800M. any ideas we have this extra memory.
my other cncern is why the QHash is using way more memory, how can i compute or estimate how much the Qhash will use memory assuming the current case.
beside that, I want to parse my file in prallel using opemMP so i need it in memory so i dont get restraind by the disk access.
also after the parse the file and push it into the Qhash, I clear the loaded file and its scope ends after as its declared as local in the that function, but I dont see the memory is freed or at least the OS does not see it as it was freed.

using dbm is a good idea actully and I will consider using it for sure. I just need to do more sreach on how to use it as im not familier with it.
if you have any simple example of how to use it in such case it would be a great help.

Thanks for your help and advices.