PDA

View Full Version : Memory used to store text in QStringList.



jesse_mark
20th August 2014, 23:19
Hello,

I read a huge text file which could reach to GBs, In this particulare file its 5GB.
I read the whole file in memory, which my machine can handle with no problem. I use QTextStreamer to read line by line and store it in QStringList.
but I see the memery used to read the whole file is 11GB. Why I needed more than double the size of the file to be stored in QStringList.
Is there other way to reduce the memory being used ?? or what coulde be the best and fastes way to read a huge text files ??

Thanks for help,

Jesse.

Infinity
21st August 2014, 00:48
Show your code. Maybe you're not deleting everything properly.

anda_skoa
21st August 2014, 10:36
Maybe your text file is ASCII, one byte per character. QString is 16-bit unicode, 2 bytes per character.

Cheers,
_

jesse_mark
21st August 2014, 17:38
@Infinity deleting what ??? all I am doing is loading text file.


@anda_skoa, that is exactlly what it is. SO is there some work round it ? to force QString to use 1 byte per character ??

Thanks.

wysota
21st August 2014, 17:57
@anda_skoa, that is exactlly what it is. SO is there some work round it ? to force QString to use 1 byte per character ??

There are two workarounds:
1. don't use QStringList but rather QList<QByteArray>
2. don't read the whole file into memory but rather index the things you need and keep the index in memory only.

faldzip
21st August 2014, 18:00
why do you need to store it in memory as QString? Maybe just std::string, char * or QLatin1String would be sufficient, and then you can for example make QString with 1 line data at once if it is enough for current processing you need.

elcuco
25th August 2014, 08:46
Read wysota comment again and follow him. I remember that QString does not allocate exactly the amount of memory you request but other bunch sizes.

It was explained on some talk (which I cannot remember its name). For such use-cases vanilla Qt will not help you.