gzh
1st March 2019, 14:02
I encountered a strange problem with my Qt-based multi-thread application. After several days running, the application will freeze without any response.
After freeze occurred, I can confirm that several threads, including the main thread, are in futex_wait_queue_me status. When I attach to that application to investigate thread status by GDB, the backtrace of those threads
shows that they all stopped at the following function with the same argument i.e. futex=0x45a2f8b8 <main_arena>.
__lll_lock_wait_private (futex=0x45a2f8b8 <main_arena>)
I know that on Linux, using non-asynchronous-safe functions within signal handlers is one of possible reasons for this status, i.e. several threads wait for the same mutex, (I can confirm from backtrace that they all stopped at malloc()/free() related function calls), but after I confirmed my Qt application, I can not find implementations related to Linux signal handlers. (but I am not sure whether Qt core library is using Linux signal handlers in its signal/slot mechanism.)
I am sorry that I can not provide source code for this question because it is a huge project. Would you like tell me some possible reasons for this phenomenon, or some advises on how to debug it?
Thanks in advance.
I can provide backtrace, but sorry I have to delete some sensible information.
Backtrace of sub thread:
#0 in __lll_lock_wait_private (futex=0x4ad078b8 <main_arena>)
#1 in __GI___libc_malloc (bytes=32) at malloc.c:2918
#11 in SystemEventImp::event(QEvent*) ()
#12 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#13 in QApplication::notify(QObject*, QEvent*) ()
#14 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#15 in QCoreApplicationPrivate::sendPostedEvents(QObject* , int, QThreadData*) ()
#16 in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1329
#17 in QWindowSystemInterface::sendWindowSystemEvents (flags=...) at kernel/qwindowsysteminterface.cpp:560
#18 in QUnixEventDispatcherQPA::processEvents (this=0x8079958, flags=...) at eventdispatchers/qunixeventdispatcher.cpp:70
#19 in QEventLoop::processEvents (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:136
#20 in QEventLoop::exec (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:212
#21 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1120
#22 in QGuiApplication::exec () at kernel/qguiapplication.cpp:1220
#23 in QApplication::exec () at kernel/qapplication.cpp:2689
#24 in main(argc=2, argv=0xbffff294)
Backtrace of main thread:
#0 in __lll_lock_wait_private (futex=0x4ad078b8 <main_arena>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:32
#1 in __GI___libc_malloc (bytes=8) at malloc.c:2918
... ...
#15 in QGraphicsView::paintEvent(QPaintEvent*) ()
#16 in QWidget::event(QEvent*) ()
#17 in QFrame::event(QEvent*) ()
#18 in QGraphicsView::viewportEvent(QEvent*) ()
#19 in Platform::Drawing::GraphicsView::viewportEvent(QEv ent*) ()
#20 in QAbstractScrollAreaFilter::eventFilter(QObject*, QEvent*) ()
#21 in QCoreApplicationPrivate::cancel_handler(QObject*, QEvent*) ()
#22 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#23 in QApplication::notify(QObject*, QEvent*) ()
#24 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#25 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, int, QPainter*, QWidgetBackingStore*) [clone .part.175] ()
#26 in QWidgetBackingStore::sync() ()
#27 in QWidgetPrivate::syncBackingStore() ()
#28 in QWidget::event(QEvent*) ()
#29 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#30 in QApplication::notify(QObject*, QEvent*) ()
#31 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#32 in QCoreApplicationPrivate::sendPostedEvents(QObject* , int, QThreadData*) ()
#33 in QCoreApplication::sendPostedEvents (receiver=0x809ea50, event_type=77)
#34 in QGraphicsViewPrivate::dispatchPendingUpdateRequest s (this=0x80e4418)
#35 in QGraphicsScenePrivate::_q_processDirtyItems (this=0x80de238) at graphicsview/qgraphicsscene.cpp:508
#36 in QGraphicsScene::qt_static_metacall (_o=0x80d1a80, _c=QMetaObject::InvokeMetaMethod, _id=15, _a=0x865e238)
#37 in QMetaCallEvent::placeMetaCall (this=0x898d020, object=0x80d1a80)
#38 in QObject::event (this=0x80d1a80, e=0x898d020) at kernel/qobject.cpp:1070
#39 in QGraphicsScene::event (this=0x80d1a80, event=0x898d020) at graphicsview/qgraphicsscene.cpp:3478
#40 in QApplicationPrivate::notify_helper (this=0x8077ba0, receiver=0x80d1a80, e=0x898d020) at kernel/qapplication.cpp:3457
#41 in QApplication::notify (this=0x8077970, receiver=0x80d1a80, e=0x898d020) at kernel/qapplication.cpp:2878
#42 in QCoreApplication::notifyInternal (this=0x8077970, receiver=0x80d1a80, event=0x898d020) at kernel/qcoreapplication.cpp:867
#43 in QCoreApplication::sendEvent (receiver=0x80d1a80, event=0x898d020) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:232
#44 in QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0, data=0x8073318) at kernel/qcoreapplication.cpp:1471
#45 in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1329
#46 in QWindowSystemInterface::sendWindowSystemEvents (flags=...) at kernel/qwindowsysteminterface.cpp:560
#47 in QUnixEventDispatcherQPA::processEvents (this=0x8079958, flags=...) at eventdispatchers/qunixeventdispatcher.cpp:70
#48 in QEventLoop::processEvents (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:136
#49 in QEventLoop::exec (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:212
#50 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1120
#51 in QGuiApplication::exec () at kernel/qguiapplication.cpp:1220
#52 in QApplication::exec () at kernel/qapplication.cpp:2689
#53 in main(argc=2, argv=0xbffff294)
After freeze occurred, I can confirm that several threads, including the main thread, are in futex_wait_queue_me status. When I attach to that application to investigate thread status by GDB, the backtrace of those threads
shows that they all stopped at the following function with the same argument i.e. futex=0x45a2f8b8 <main_arena>.
__lll_lock_wait_private (futex=0x45a2f8b8 <main_arena>)
I know that on Linux, using non-asynchronous-safe functions within signal handlers is one of possible reasons for this status, i.e. several threads wait for the same mutex, (I can confirm from backtrace that they all stopped at malloc()/free() related function calls), but after I confirmed my Qt application, I can not find implementations related to Linux signal handlers. (but I am not sure whether Qt core library is using Linux signal handlers in its signal/slot mechanism.)
I am sorry that I can not provide source code for this question because it is a huge project. Would you like tell me some possible reasons for this phenomenon, or some advises on how to debug it?
Thanks in advance.
I can provide backtrace, but sorry I have to delete some sensible information.
Backtrace of sub thread:
#0 in __lll_lock_wait_private (futex=0x4ad078b8 <main_arena>)
#1 in __GI___libc_malloc (bytes=32) at malloc.c:2918
#11 in SystemEventImp::event(QEvent*) ()
#12 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#13 in QApplication::notify(QObject*, QEvent*) ()
#14 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#15 in QCoreApplicationPrivate::sendPostedEvents(QObject* , int, QThreadData*) ()
#16 in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1329
#17 in QWindowSystemInterface::sendWindowSystemEvents (flags=...) at kernel/qwindowsysteminterface.cpp:560
#18 in QUnixEventDispatcherQPA::processEvents (this=0x8079958, flags=...) at eventdispatchers/qunixeventdispatcher.cpp:70
#19 in QEventLoop::processEvents (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:136
#20 in QEventLoop::exec (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:212
#21 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1120
#22 in QGuiApplication::exec () at kernel/qguiapplication.cpp:1220
#23 in QApplication::exec () at kernel/qapplication.cpp:2689
#24 in main(argc=2, argv=0xbffff294)
Backtrace of main thread:
#0 in __lll_lock_wait_private (futex=0x4ad078b8 <main_arena>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:32
#1 in __GI___libc_malloc (bytes=8) at malloc.c:2918
... ...
#15 in QGraphicsView::paintEvent(QPaintEvent*) ()
#16 in QWidget::event(QEvent*) ()
#17 in QFrame::event(QEvent*) ()
#18 in QGraphicsView::viewportEvent(QEvent*) ()
#19 in Platform::Drawing::GraphicsView::viewportEvent(QEv ent*) ()
#20 in QAbstractScrollAreaFilter::eventFilter(QObject*, QEvent*) ()
#21 in QCoreApplicationPrivate::cancel_handler(QObject*, QEvent*) ()
#22 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#23 in QApplication::notify(QObject*, QEvent*) ()
#24 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#25 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, int, QPainter*, QWidgetBackingStore*) [clone .part.175] ()
#26 in QWidgetBackingStore::sync() ()
#27 in QWidgetPrivate::syncBackingStore() ()
#28 in QWidget::event(QEvent*) ()
#29 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#30 in QApplication::notify(QObject*, QEvent*) ()
#31 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#32 in QCoreApplicationPrivate::sendPostedEvents(QObject* , int, QThreadData*) ()
#33 in QCoreApplication::sendPostedEvents (receiver=0x809ea50, event_type=77)
#34 in QGraphicsViewPrivate::dispatchPendingUpdateRequest s (this=0x80e4418)
#35 in QGraphicsScenePrivate::_q_processDirtyItems (this=0x80de238) at graphicsview/qgraphicsscene.cpp:508
#36 in QGraphicsScene::qt_static_metacall (_o=0x80d1a80, _c=QMetaObject::InvokeMetaMethod, _id=15, _a=0x865e238)
#37 in QMetaCallEvent::placeMetaCall (this=0x898d020, object=0x80d1a80)
#38 in QObject::event (this=0x80d1a80, e=0x898d020) at kernel/qobject.cpp:1070
#39 in QGraphicsScene::event (this=0x80d1a80, event=0x898d020) at graphicsview/qgraphicsscene.cpp:3478
#40 in QApplicationPrivate::notify_helper (this=0x8077ba0, receiver=0x80d1a80, e=0x898d020) at kernel/qapplication.cpp:3457
#41 in QApplication::notify (this=0x8077970, receiver=0x80d1a80, e=0x898d020) at kernel/qapplication.cpp:2878
#42 in QCoreApplication::notifyInternal (this=0x8077970, receiver=0x80d1a80, event=0x898d020) at kernel/qcoreapplication.cpp:867
#43 in QCoreApplication::sendEvent (receiver=0x80d1a80, event=0x898d020) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:232
#44 in QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0, data=0x8073318) at kernel/qcoreapplication.cpp:1471
#45 in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1329
#46 in QWindowSystemInterface::sendWindowSystemEvents (flags=...) at kernel/qwindowsysteminterface.cpp:560
#47 in QUnixEventDispatcherQPA::processEvents (this=0x8079958, flags=...) at eventdispatchers/qunixeventdispatcher.cpp:70
#48 in QEventLoop::processEvents (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:136
#49 in QEventLoop::exec (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:212
#50 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1120
#51 in QGuiApplication::exec () at kernel/qguiapplication.cpp:1220
#52 in QApplication::exec () at kernel/qapplication.cpp:2689
#53 in main(argc=2, argv=0xbffff294)