Memory used to store text in QStringList.

[jesse_mark]

Hello,

I read a huge text file which could reach to GBs, In this particulare file its 5GB.
I read the whole file in memory, which my machine can handle with no problem. I use QTextStreamer to read line by line and store it in QStringList.
but I see the memery used to read the whole file is 11GB. Why I needed more than double the size of the file to be stored in QStringList.
Is there other way to reduce the memory being used ?? or what coulde be the best and fastes way to read a huge text files ??

Thanks for help,

Jesse.

[Infinity]

Show your code. Maybe you're not deleting everything properly.

[anda_skoa]

Maybe your text file is ASCII, one byte per character. QString is 16-bit unicode, 2 bytes per character.

Cheers,
_

[jesse_mark]

@Infinity deleting what ??? all I am doing is loading text file.


@anda_skoa, that is exactlly what it is. SO is there some work round it ? to force QString to use 1 byte per character ??

Thanks.

[wysota]

@anda_skoa, that is exactlly what it is. SO is there some work round it ? to force QString to use 1 byte per character ??

There are two workarounds:
1. don't use QStringList but rather QList<QByteArray>
2. don't read the whole file into memory but rather index the things you need and keep the index in memory only.

[faldzip]

why do you need to store it in memory as QString? Maybe just std::string, char * or QLatin1String would be sufficient, and then you can for example make QString with 1 line data at once if it is enough for current processing you need.

[elcuco]

Read wysota comment again and follow him. I remember that QString does not allocate exactly the amount of memory you request but other bunch sizes.

It was explained on some talk (which I cannot remember its name). For such use-cases vanilla Qt will not help you.