Results 1 to 7 of 7

Thread: Memory used to store text in QStringList.

  1. #1
    Join Date
    Jun 2012
    Posts
    173
    Thanks
    48
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Memory used to store text in QStringList.

    Hello,

    I read a huge text file which could reach to GBs, In this particulare file its 5GB.
    I read the whole file in memory, which my machine can handle with no problem. I use QTextStreamer to read line by line and store it in QStringList.
    but I see the memery used to read the whole file is 11GB. Why I needed more than double the size of the file to be stored in QStringList.
    Is there other way to reduce the memory being used ?? or what coulde be the best and fastes way to read a huge text files ??

    Thanks for help,

    Jesse.

  2. #2
    Join Date
    Oct 2012
    Posts
    132
    Thanks
    10
    Thanked 21 Times in 21 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows Android

    Default Re: Memory used to store text in QStringList.

    Show your code. Maybe you're not deleting everything properly.

  3. #3
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Memory used to store text in QStringList.

    Maybe your text file is ASCII, one byte per character. QString is 16-bit unicode, 2 bytes per character.

    Cheers,
    _

  4. The following user says thank you to anda_skoa for this useful post:

    jesse_mark (22nd August 2014)

  5. #4
    Join Date
    Jun 2012
    Posts
    173
    Thanks
    48
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Memory used to store text in QStringList.

    @Infinity deleting what ??? all I am doing is loading text file.


    @anda_skoa, that is exactlly what it is. SO is there some work round it ? to force QString to use 1 byte per character ??

    Thanks.

  6. #5
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Memory used to store text in QStringList.

    Quote Originally Posted by jesse_mark View Post
    @anda_skoa, that is exactlly what it is. SO is there some work round it ? to force QString to use 1 byte per character ??
    There are two workarounds:
    1. don't use QStringList but rather QList<QByteArray>
    2. don't read the whole file into memory but rather index the things you need and keep the index in memory only.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  7. The following user says thank you to wysota for this useful post:

    jesse_mark (22nd August 2014)

  8. #6
    Join Date
    Jan 2008
    Location
    Poland
    Posts
    687
    Thanks
    4
    Thanked 140 Times in 132 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Memory used to store text in QStringList.

    why do you need to store it in memory as QString? Maybe just std::string, char * or QLatin1String would be sufficient, and then you can for example make QString with 1 line data at once if it is enough for current processing you need.
    I would like to be a "Guru"

    Useful hints (try them before asking):
    1. Use Qt Assistant
    2. Search the forum

    If you haven't found solution yet then create new topic with smart question.

  9. The following user says thank you to faldzip for this useful post:

    jesse_mark (22nd August 2014)

  10. #7
    Join Date
    Jan 2006
    Posts
    368
    Thanks
    14
    Thanked 18 Times in 17 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Memory used to store text in QStringList.

    Read wysota comment again and follow him. I remember that QString does not allocate exactly the amount of memory you request but other bunch sizes.

    It was explained on some talk (which I cannot remember its name). For such use-cases vanilla Qt will not help you.

Similar Threads

  1. How does QTextEdit store the text?
    By Fox196 in forum Newbie
    Replies: 2
    Last Post: 18th August 2011, 05:47
  2. How to store the lineedit contents in memory
    By grsandeep85 in forum Qt Programming
    Replies: 4
    Last Post: 3rd August 2009, 14:16
  3. Replies: 5
    Last Post: 9th April 2008, 00:10
  4. what is free store in C++ memory?
    By Masih in forum General Programming
    Replies: 6
    Last Post: 2nd July 2007, 23:25
  5. Replies: 7
    Last Post: 2nd June 2006, 13:48

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.