Results 1 to 3 of 3

Thread: QList<QBbyteArray> & QHash memory usage.

  1. #1
    Join Date
    Jun 2012
    Posts
    173
    Thanks
    48
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default QList<QBbyteArray> & QHash memory usage.

    Hello everyone,


    I am trying to read a text file and process its content.
    the file i am using has 10,000,000 lines each line is 80 byte in lenght, so its about 800M in size.

    The first step
    I read the file line by line and push it into a QList <QByteArray>. <<== declared private inside the a method.

    Then
    I itarate over the QList parse each line and get from it Key and lineInfo to push into a QHash <QString, lineInfo> <=== declared public in header
    The key is a QString====> the key lenght 10 bytes in this case.
    the lineInfo is a class with eight private members all are "int"

    when this process is done i check the memory using "top" command from a terimnal the application is using about 3g of memory.
    any Idea why its using that much of memory and how to reduce that ??


    best regards,

    ps:
    Enviroments:
    Qt 4.8
    linux redhat 6.3

  2. #2
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,677
    Thanks
    13
    Thanked 1,596 Times in 1,524 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: QList<QBbyteArray> & QHash memory usage.

    Well, for a start you should not store the entire file in RAM (That is easily a gigabyte on its own). It sounds like you can parse each line as it is read from file and drop the relevant bits into your QHash in one motion.

    Have you considered using a disk-based store instead of the in-memory hash? Hash key lookups are something that the dbm family of libraries do for a living, but any RDBMS (including the zero cost Sqlite) will make decent job of it.

  3. #3
    Join Date
    Jun 2012
    Posts
    173
    Thanks
    48
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QList<QBbyteArray> & QHash memory usage.

    storing the file in memory is ok in my case as i have a big ram and i dont mind that if it stays the same size as the file size,
    but after loading the file the memory used is bigger than the file size its 1400M and my file is only 800M. any ideas we have this extra memory.
    my other cncern is why the QHash is using way more memory, how can i compute or estimate how much the Qhash will use memory assuming the current case.
    beside that, I want to parse my file in prallel using opemMP so i need it in memory so i dont get restraind by the disk access.
    also after the parse the file and push it into the Qhash, I clear the loaded file and its scope ends after as its declared as local in the that function, but I dont see the memory is freed or at least the OS does not see it as it was freed.

    using dbm is a good idea actully and I will consider using it for sure. I just need to do more sreach on how to use it as im not familier with it.
    if you have any simple example of how to use it in such case it would be a great help.

    Thanks for your help and advices.

Similar Threads

  1. How to disable individual members of a QHash and QList?
    By in_dbasu in forum Qt Programming
    Replies: 3
    Last Post: 22nd July 2011, 06:55
  2. Replies: 6
    Last Post: 22nd March 2011, 16:17
  3. QList < QHash < int, QString> > ... clear() is slow
    By JoZCaVaLLo in forum Qt Programming
    Replies: 8
    Last Post: 15th March 2011, 12:07
  4. QHash memory usage
    By helloworld in forum Qt Programming
    Replies: 1
    Last Post: 10th December 2010, 22:08
  5. Query about QHash , QList
    By aamer4yu in forum Qt Programming
    Replies: 3
    Last Post: 14th December 2006, 10:04

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.