Results 1 to 9 of 9

Thread: QTcpSocket leaking? What am I doing wrong?

  1. #1
    Join Date
    Feb 2007
    Location
    Karlsruhe, Germany
    Posts
    469
    Thanks
    17
    Thanked 90 Times in 88 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default QTcpSocket leaking? What am I doing wrong?

    Hello all!

    I have been trying to pinpoint a heavy memory leak in a realtime network library I wrote, without success. As I transmit quite a lot of data (5-10MB/sec) the leak quickly crashes the application. The weird thing is that in about half of the program starts (without recompiling, just restarting!) no leaking occurs, at all! This happens both under windows (mingw/Qt4.71, msvc2008/Qt4.73) and under linux (g++/4.70).

    After a lot of code rechecking and doing most of the memory allocations directly, my programm shows the following valgrind output:

    Qt Code:
    1. ==2784== 327,763,678 bytes in 398 blocks are possibly lost in loss record 284 of 284
    2. ==2784== at 0x4025BD3: malloc (vg_replace_malloc.c:236)
    3. ==2784== by 0x4D6E05C: qMalloc(unsigned int) (qmalloc.cpp:55)
    4. ==2784== by 0x4D76A34: QByteArray::resize(int) (qbytearray.cpp:1379)
    5. ==2784== by 0x4C8D5B9: QRingBuffer::reserve(int) (qringbuffer_p.h:187)
    6. ==2784== by 0x4C8793C: QAbstractSocketPrivate::readFromSocket() (qabstractsocket.cpp:1158)
    7. ==2784== by 0x4C87F4B: QAbstractSocketPrivate::canReadNotification() (qabstractsocket.cpp:614)
    8. ==2784== by 0x4C74A7A: QAbstractSocketEngine::readNotification() (qabstractsocketengine.cpp:154)
    9. ==2784== by 0x4C75CEE: QReadNotifier::event(QEvent*) (qnativesocketengine.cpp:1103)
    10. ==2784== by 0x41B7FFB: QApplicationPrivate::notify_helper(QObject*, QEvent*) (qapplication.cpp:4396)
    11. ==2784== by 0x41BED92: QApplication::notify(QObject*, QEvent*) (qapplication.cpp:3798)
    12. ==2784== by 0x4E8D68A: QCoreApplication::notifyInternal(QObject*, QEvent*) (qcoreapplication.cpp:732)
    13. ==2784== by 0x4EBDA12: socketNotifierSourceDispatch(_GSource*, int (*)(void*), void*) (qcoreapplication.h:215)
    To copy to clipboard, switch view to plain text mode 
    I'm using a standard QTcpSocket.

    Qt Code:
    1. void QntsBlockStreamReader::setupDataTCP()
    2. {
    3. nextBlockSize = 0;
    4. dataPortTCP = new QTcpSocket();
    5. connect(dataPortTCP,SIGNAL(connected()),this,SLOT(dataPortConnectedTCP()));
    6. connect(dataPortTCP,SIGNAL(readyRead()),this,SLOT(dataIncomingTCP()));
    7. dataPortTCP->connectToHost(writerAddress,dataPortNo);
    8. }
    To copy to clipboard, switch view to plain text mode 
    In it's readyRead slot, I create block objects - which I checked are properly released after usage.

    Qt Code:
    1. void QntsBlockStreamReader::dataIncomingTCP()
    2. {
    3. while (dataPortTCP->bytesAvailable() > 0)
    4. {
    5. if (nextBlockSize == 0) {
    6. if (dataPortTCP->bytesAvailable() < (sizeof(quint8)+sizeof(TBlockSize))) return;
    7. // BlockSize contains the blockSize value itself
    8. QByteArray b = dataPortTCP->peek(sizeof(quint8)+sizeof(TBlockSize));
    9. QntsDataStream ds(&b,QIODevice::ReadOnly);
    10. quint8 protocolVersion;
    11. ds >> protocolVersion >> nextBlockSize;
    12. }
    13. if ((TBlockSize)dataPortTCP->bytesAvailable() < nextBlockSize) return;
    14. logMsg(QString("Bytes Available: %1 - %2").arg(dataPortTCP->bytesAvailable()).arg(QntsBlock::allocatedBlocks));
    15. QByteArray newData = dataPortTCP->read(nextBlockSize);
    16. blockArrived(new QntsBlock(newData),dataPortTCP->peerAddress(),dataPortTCP->peerPort());
    17. nextBlockSize = 0;
    18. }
    19. }
    20.  
    21. QntsBlock::QntsBlock(const QByteArray& ba)
    22. {
    23. m_size = ba.size();
    24. m_data = (char*)malloc(m_size);
    25. memcpy(&m_data[0],(void*)ba.constData(),m_size);
    26. allocatedBlocks += 1;
    27. }
    28.  
    29. QntsBlock::~QntsBlock()
    30. {
    31. allocatedBlocks -= 1;
    32. free(m_data);
    33. }
    To copy to clipboard, switch view to plain text mode 
    I would greatly appreciate any clues or hints how to proceed! First of all I would like to make sure, that I am not doing something plainly stupid, before investing even more time boiling this down, by creating a small test app, that shows the same symptoms.

    Thx a lot in advance!

    Johannes

  2. #2
    Join Date
    Jun 2010
    Location
    Germany, Munich
    Posts
    16
    Thanks
    1
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QTcpSocket leaking? What am I doing wrong?

    I didn`t get your code exactly but once I had the same problem with
    mismatch the blocking functions with the non-blocking signals & slots.

    If you create some Sockets within a exec() event loop you should use the
    non blocking way.

    read carefully the QAbstractSocket
    int http://doc.qt.nokia.com/4.1/qabstrac...ected-function

  3. #3
    Join Date
    Feb 2007
    Location
    Karlsruhe, Germany
    Posts
    469
    Thanks
    17
    Thanked 90 Times in 88 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QTcpSocket leaking? What am I doing wrong?

    Thank you for your time.. but, as you can see in the code I provided, I am using the readyRead-signal and direct reading of the blockbuffer, without any blocking calls to the socket.

    Furthermore, I don't think that swaping to the blocking functions will change the memory allocation behaviour of the socket, which seems to leak.

    The code valgrind points out is purely Qt. Am I inadvertently holding a reference to an implicitly shared byte array? I can't see it!

    Thx a lot in advance for any further hints!

    Johannes

  4. #4
    Join Date
    Jun 2010
    Location
    Germany, Munich
    Posts
    16
    Thanks
    1
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QTcpSocket leaking? What am I doing wrong?

    So does QntsBlockStreamReader lives in a thread inside a event loop?

    I this Class created in the GUI thread do you have some threads like this?

    Qt Code:
    1. myThread::start()
    2. while ( m_running)
    3. {
    4.  
    5. }
    To copy to clipboard, switch view to plain text mode 

  5. #5
    Join Date
    Feb 2007
    Location
    Karlsruhe, Germany
    Posts
    469
    Thanks
    17
    Thanked 90 Times in 88 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QTcpSocket leaking? What am I doing wrong?

    If that would be the case, then my socket could not emit its readyRead signal. But it does so beautifully.

    Everything works. But sporadically leaks huge amounts of memory!

    The reader and thus the tcpsocket lives in a thread like this:

    Qt Code:
    1. QntsBlockStreamReader::QntsBlockStreamReader(QString pName,QThread* pThread)
    2. {
    3. thread = pThread;
    4. if (thread == 0)
    5. {
    6. thread = new QThread(this);
    7. thread->start();
    8. }
    9.  
    10. this->moveToThread(thread);
    11.  
    12. QMetaObject::invokeMethod(this,"setupDataTCP");
    13. }
    14.  
    15. void QntsBlockStreamReader::setupDataTCP()
    16. {
    17. nextBlockSize = 0;
    18. dataPortTCP = new QTcpSocket();
    19. connect(dataPortTCP,SIGNAL(connected()),this,SLOT(dataPortConnectedTCP()));
    20. connect(dataPortTCP,SIGNAL(readyRead()),this,SLOT(dataIncomingTCP()));
    21. dataPortTCP->connectToHost(writerAddress,dataPortNo);
    22. }
    To copy to clipboard, switch view to plain text mode 

    I don't think that we are close to a solution here, because this would be a fundamental flaw and not a sporadic leak.

    Joh

  6. #6
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,363
    Thanks
    3
    Thanked 5,012 Times in 4,791 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QTcpSocket leaking? What am I doing wrong?

    Please provide a minimal compilable example reproducing the problem.

    By the way, creating a thread with 'this' as a parent and moving 'this' object to the thread effectively moves the thread object to itself which is wrong.

    You might be leaking memory because of an incorrect use of threads and sockets. Some cleanup routine might not be firing if some piece of code gets confused by incorrect thread affinity.
    Last edited by wysota; 22nd June 2011 at 02:18.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  7. #7
    Join Date
    Feb 2007
    Location
    Karlsruhe, Germany
    Posts
    469
    Thanks
    17
    Thanked 90 Times in 88 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QTcpSocket leaking? What am I doing wrong?

    Wysota, thanks for your input! I quickly tested your hypothesis. Inside the MainForm constructor, thus from the mainthread:

    Qt Code:
    1. reader = new QntsBlockStreamReader("COVE BXPD",QThread::currentThread());
    To copy to clipboard, switch view to plain text mode 
    ... and it still leaks! I followed your discussions concerning sockets and threading and I agree that in most cases threading a socket is plainly unnecessary. In this case I have the additional special requirement of minimal latency (~1ms) for my working code, and I can't take the chance of mainthread-stuff interfering. Furthermore this library is also intended to be used from a non qt-app, and I want the socket to handle its data even if the calling thread messes up.

    I am inclined to implement the socket handling myself, without any unnecessary intermediate ringbuffering. On the other hand I am still convinced that QTCPSocket is error free and its just my usage of it that screws it up! I will try to investigate how usually its ringbuffer is released. If that does not reveal any new clues, I will try to reproduce the problem in a small test application!

    Johannes
    Last edited by JohannesMunk; 22nd June 2011 at 10:31.

  8. #8
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,363
    Thanks
    3
    Thanked 5,012 Times in 4,791 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QTcpSocket leaking? What am I doing wrong?

    Try getting rid of threads completely and check if the leak is still there. And explain why you think there is a leak in the first place. Valgrind reports your memory as reachable so there is a pointer somewhere that points to those memory blocks. Maybe you're just not freeing memory yourself somewhere? That's why I asked for a minimal example reproducing the problem. BTW. I don't see how putting each socket in a separate thread helps achieving <1ms latency.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  9. #9
    Join Date
    Feb 2007
    Location
    Karlsruhe, Germany
    Posts
    469
    Thanks
    17
    Thanked 90 Times in 88 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QTcpSocket leaking? What am I doing wrong?

    I did test it entirely without threads! It must be the internal ringbuffer that has some dangling pointers left. I don't know what is going on.

    But I've had enough and implemented a native windows socket implementation, and instant reward: no leaks anymore!

    I will hunt down this usage pattern incompatibility / bug, but not this week. The project comes first!

    I will be back with this!

    Johannes

Similar Threads

  1. What do I do wrong
    By Arif Bilgin in forum Newbie
    Replies: 12
    Last Post: 20th October 2010, 20:03
  2. i don't know what i do wrong..
    By Hardstyle in forum Newbie
    Replies: 2
    Last Post: 27th June 2010, 17:33
  3. what is wrong with my qt ?
    By lwb422 in forum Qt Programming
    Replies: 5
    Last Post: 12th April 2010, 14:07
  4. wrong syntax or QTcpSocket problem?
    By vito49 in forum Newbie
    Replies: 3
    Last Post: 8th October 2008, 08:12
  5. Help please - what am I doing wrong?
    By Jimmy2775 in forum Qt Programming
    Replies: 6
    Last Post: 6th March 2006, 22:06

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.