Results 1 to 5 of 5

Thread: QProcess and memory usage

  1. #1
    Join Date
    May 2011
    Posts
    11
    Thanks
    3
    Qt products
    Qt4

    Default QProcess and memory usage

    Hello everybody,

    I am just working on program that turns out to have a huge memory leak. In this program I use QProcess to start an external program which is generating data that I process in my program. The external programm writes its output to stdout from where I read it with QProcess::readLine(). Reading is triggered by the QProcess::readyReadStandardOutput() signal. The external program creates up to a few hundred MB of data every second. It looks like QProcess never throws that data away after I read the data from stdout, eating up my systems memory.

    Is there a way to free that memory used by QProcess again? But the QProcess has to be active all the time.

    Or is there a fundamental flaw with my concept? Did I make obvious mistakes? I want to move to sockets to get the data in the future, but this is not a option now. I have to wait until the developer of the external program is able to do it. Until then I would like to have my program work with passing data via stdout.

    Thanks in advance.

    (Edit: I am developing on a Linux machine, but I want my program to work on other systems at a later point)
    Last edited by ts66; 8th December 2011 at 19:04.

  2. #2
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,592
    Thanks
    13
    Thanked 1,587 Times in 1,516 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: QProcess and memory usage

    Are you reading a few hundred megabytes per second or is QProcess buffering it because you cannot keep up? Is there a memory leak in your reading code that has nothing to do with QProcess?

    In general, once memory is allocated for use in a program it is not freed to the operating system until program termination, or until the OS explicitly reclaims unused memory that is still allocated to a process (which will happen if memory is exhausted). Between QProcess buffering data and you doing something with that data that probably involves creating copies it is easy to see a memory consumption that rapidly climbs to a high, but reasonable stable level even if there is no leak.

  3. #3
    Join Date
    May 2011
    Posts
    11
    Thanks
    3
    Qt products
    Qt4

    Default Re: QProcess and memory usage

    I did some more testing and it probably is as you suggested.

    I reduced the amount of data the external program generates (which I run with QProcess) and now the memory consumption stays on a constant, reasonable level.

    So, maybe the old external program just created more data than I could handle with my Qt Application, and that's why the data got buffered and the memory usage skyrocketed. That's a bummer, I will have to rethink much of the concept I currently want to implement and do some more testing.

    Whats your opinion in general on this: If a QProcess creates up to 1000 sets of data per second, each data set basically a oneliner on stdout, and I fetch the data with QProcess::readLine(), every reading triggered by a readyreadLine signal, should Qt be able to handle his?

    And thanks for giving me the hint.

  4. #4
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,592
    Thanks
    13
    Thanked 1,587 Times in 1,516 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: QProcess and memory usage

    What is producing 100+MB of data per second in a text form?

    As to the performance... maybe. If it is a 1000 lines of 10 bytes per second that's a vastly different proposition to 1000 lines of 100000+ bytes, which is what you started with. Since a QIODevice does not have a signal to tell you that a whole line is present, only that some new data is available (QIODevice::readyRead() or the stdout/stderr versions in QProcess) you must at least be doing some buffering and construction of whole lines from fragments. Then, of course, you will be doing something to the data that takes time, which may be trivial or time expensive.

  5. #5
    Join Date
    May 2011
    Posts
    11
    Thanks
    3
    Qt products
    Qt4

    Default Re: QProcess and memory usage

    Sorry for the long delay, but I was busy with other things. To give some more precise numbers: I tried to process about 20MB of data per second, cominig in about 3.000.000 lines. Each line triggers a readline event. Thats aparently too much data. When I reduce the data input to about one third of that number my program works fine without excessive memory consumption.

Similar Threads

  1. Bad memory usage on QWebView I think
    By jiturra in forum Qt Programming
    Replies: 15
    Last Post: 21st January 2014, 20:35
  2. QHash memory usage
    By helloworld in forum Qt Programming
    Replies: 1
    Last Post: 10th December 2010, 21:08
  3. Memory usage of simple Qt app
    By DiamonDogX in forum Qt Programming
    Replies: 1
    Last Post: 23rd September 2009, 06:53
  4. CPU Time & Memory Usage in QThread or QProcess
    By Davidaino in forum Qt Programming
    Replies: 0
    Last Post: 11th July 2008, 19:15
  5. CPU and Memory Usage
    By philwinder in forum Qt Programming
    Replies: 16
    Last Post: 17th May 2008, 22:25

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.