PDA

View Full Version : Qt multi-thread application freezes and several threads wait for the same mutex


gzh
1st March 2019, 14:02
I encountered a strange problem with my Qt-based multi-thread application. After several days running, the application will freeze without any response.

After freeze occurred, I can confirm that several threads, including the main thread, are in futex_wait_queue_me status. When I attach to that application to investigate thread status by GDB, the backtrace of those threads
shows that they all stopped at the following function with the same argument i.e. futex=0x45a2f8b8 <main_arena>.

__lll_lock_wait_private (futex=0x45a2f8b8 <main_arena>)

I know that on Linux, using non-asynchronous-safe functions within signal handlers is one of possible reasons for this status, i.e. several threads wait for the same mutex, (I can confirm from backtrace that they all stopped at malloc()/free() related function calls), but after I confirmed my Qt application, I can not find implementations related to Linux signal handlers. (but I am not sure whether Qt core library is using Linux signal handlers in its signal/slot mechanism.)

I am sorry that I can not provide source code for this question because it is a huge project. Would you like tell me some possible reasons for this phenomenon, or some advises on how to debug it?

Thanks in advance.

I can provide backtrace, but sorry I have to delete some sensible information.

Backtrace of sub thread:


#0 in __lll_lock_wait_private (futex=0x4ad078b8 <main_arena>)
#1 in __GI___libc_malloc (bytes=32) at malloc.c:2918

#11 in SystemEventImp::event(QEvent*) ()
#12 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#13 in QApplication::notify(QObject*, QEvent*) ()
#14 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#15 in QCoreApplicationPrivate::sendPostedEvents(QObject* , int, QThreadData*) ()
#16 in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1329
#17 in QWindowSystemInterface::sendWindowSystemEvents (flags=...) at kernel/qwindowsysteminterface.cpp:560
#18 in QUnixEventDispatcherQPA::processEvents (this=0x8079958, flags=...) at eventdispatchers/qunixeventdispatcher.cpp:70
#19 in QEventLoop::processEvents (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:136
#20 in QEventLoop::exec (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:212
#21 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1120
#22 in QGuiApplication::exec () at kernel/qguiapplication.cpp:1220
#23 in QApplication::exec () at kernel/qapplication.cpp:2689
#24 in main(argc=2, argv=0xbffff294)


Backtrace of main thread:


#0 in __lll_lock_wait_private (futex=0x4ad078b8 <main_arena>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:32
#1 in __GI___libc_malloc (bytes=8) at malloc.c:2918
... ...
#15 in QGraphicsView::paintEvent(QPaintEvent*) ()
#16 in QWidget::event(QEvent*) ()
#17 in QFrame::event(QEvent*) ()
#18 in QGraphicsView::viewportEvent(QEvent*) ()
#19 in Platform::Drawing::GraphicsView::viewportEvent(QEv ent*) ()
#20 in QAbstractScrollAreaFilter::eventFilter(QObject*, QEvent*) ()
#21 in QCoreApplicationPrivate::cancel_handler(QObject*, QEvent*) ()
#22 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#23 in QApplication::notify(QObject*, QEvent*) ()
#24 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#25 in QWidgetPrivate::drawWidget(QPaintDevice*, QRegion const&, QPoint const&, int, QPainter*, QWidgetBackingStore*) [clone .part.175] ()
#26 in QWidgetBackingStore::sync() ()
#27 in QWidgetPrivate::syncBackingStore() ()
#28 in QWidget::event(QEvent*) ()
#29 in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
#30 in QApplication::notify(QObject*, QEvent*) ()
#31 in QCoreApplication::notifyInternal(QObject*, QEvent*) ()
#32 in QCoreApplicationPrivate::sendPostedEvents(QObject* , int, QThreadData*) ()
#33 in QCoreApplication::sendPostedEvents (receiver=0x809ea50, event_type=77)
#34 in QGraphicsViewPrivate::dispatchPendingUpdateRequest s (this=0x80e4418)
#35 in QGraphicsScenePrivate::_q_processDirtyItems (this=0x80de238) at graphicsview/qgraphicsscene.cpp:508
#36 in QGraphicsScene::qt_static_metacall (_o=0x80d1a80, _c=QMetaObject::InvokeMetaMethod, _id=15, _a=0x865e238)
#37 in QMetaCallEvent::placeMetaCall (this=0x898d020, object=0x80d1a80)
#38 in QObject::event (this=0x80d1a80, e=0x898d020) at kernel/qobject.cpp:1070
#39 in QGraphicsScene::event (this=0x80d1a80, event=0x898d020) at graphicsview/qgraphicsscene.cpp:3478
#40 in QApplicationPrivate::notify_helper (this=0x8077ba0, receiver=0x80d1a80, e=0x898d020) at kernel/qapplication.cpp:3457
#41 in QApplication::notify (this=0x8077970, receiver=0x80d1a80, e=0x898d020) at kernel/qapplication.cpp:2878
#42 in QCoreApplication::notifyInternal (this=0x8077970, receiver=0x80d1a80, event=0x898d020) at kernel/qcoreapplication.cpp:867
#43 in QCoreApplication::sendEvent (receiver=0x80d1a80, event=0x898d020) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:232
#44 in QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0, data=0x8073318) at kernel/qcoreapplication.cpp:1471
#45 in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1329
#46 in QWindowSystemInterface::sendWindowSystemEvents (flags=...) at kernel/qwindowsysteminterface.cpp:560
#47 in QUnixEventDispatcherQPA::processEvents (this=0x8079958, flags=...) at eventdispatchers/qunixeventdispatcher.cpp:70
#48 in QEventLoop::processEvents (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:136
#49 in QEventLoop::exec (this=0xbfffef50, flags=...) at kernel/qeventloop.cpp:212
#50 in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1120
#51 in QGuiApplication::exec () at kernel/qguiapplication.cpp:1220
#52 in QApplication::exec () at kernel/qapplication.cpp:2689
#53 in main(argc=2, argv=0xbffff294)
anda_skoa
1st March 2019, 15:08
It is strange that both thread's stack claim that the execution started in main() and that both are inside the main application event processing.

Are those additional threads using QThread?
If yes, are you using the method of reimplementing run(), per-thread event loop or both?

Cheers,
_
gzh
1st March 2019, 23:33
It is strange that both thread's stack claim that the execution started in main() and that both are inside the main application event processing.

Are those additional threads using QThread?
If yes, are you using the method of reimplementing run(), per-thread event loop or both?

Cheers,
_

Thanks.

I create a QObject( show as SystemEventImp class in backtrace), and use movetoThread() to run it in an independent thread created by QThread.
In SystemEventImp, I register some event handler and maintain an event queue.

I did not reimplementing run(). Besides SystemEventImp Object, I created another Object named EventManager, which will call movetoThread()
to move to the same QThread object SystemEventImp are running. EventManager::ProcessQueue() will be called to start a infinite loop to process
those events maintained by SystemEventImp.
anda_skoa
2nd March 2019, 12:09
Ok, so kind of a hybrid approach: using moveToThread() like in the case of an event loop based thread but running your own loop like in a subclassed thread.

How to you start your own loop?
Do you register your event handlers in the secondary thread context?

Cheers,
_
gzh
2nd March 2019, 15:10
Hi, anda_skoa

> Do you register your event handlers in the secondary thread context?
When I create SystemEventImp Object, I register my event handlers, indeed, SystemEventImp have a member variable of map type to record <event, handler> pairs.
The registration was done before moveToThread().

>How to you start your own loop?
It seems that the loop started by calling EventManager::ProcessQueue() directly from main thread.
It is a legacy source code, I am not very sure about it.
Can this can explain your first question, i.e. "It is strange that both thread's stack claim that the execution started in main() "?
anda_skoa
2nd March 2019, 15:55
It seems that the loop started by calling EventManager::ProcessQueue() directly from main thread.

Hmm, what does that method do?
Its name sounds like it would do the actual event processing, i.e. the loop you mentioned earlier.
But it doesn't make sense to move the object to another thread and then still let the main thread to the work.

Do you call start() on the thread object?

What kind of technique are you using to make the secondary thread call the worker objects methods?



Can this can explain your first question, i.e. "It is strange that both thread's stack claim that the execution started in main() "?

No, not yet.

I don't quite yet understand the setup you are using.
Very uncommon to call a method on an object that has been moved to another thread.

Cheers,
_
gzh
4th March 2019, 09:44
Hi, anda_skoa,

Sorry, I have to revise what I said about event processing in my application. Indeed, It works like the followings.

1) I did not create event loop for those threads besides main thread, all threads are using the eventloop in main thread.

2) When SystemEventImp object created, map of <event, handler> pairs will be initialized. there are many types of event, but handler
is only SystemEventImp object, one of its member variable is a list of handler.

3) I create QThread instance, e.x. named subThread, Besides call SystemEventImp->moveToThread(subThread), subThread::start() will launch a
periodic timer, that timer will create timer event to force screen redraw periodically in main thread.

4) Besides those two threads I mentioned, there are other threads that will use call QCoreApplication::postEvent() to append event to main event queue.
when the event's handler is SystemEventImp object, it will be handled in the subThread I created.

I am not familiar with QThread and its scheduling policy, I want know whether my event processing method will run into a race condition and incur that freeze as I showed on my question.
Cruz
9th March 2019, 09:22
Hi,

I don't understand the details of the constellation gzh described, but the most common reason for this deadlock to occur is when a mutex is never unlocked. After several days of running, this could happen imho when a thread segfaults or throws an exception, or is coded in a way that it is possible for it to bypass the thread unlock. Wouldn't it be easier to approach this problem from a code analysis perspective? So far, no source has been shown...
anda_skoa
9th March 2019, 09:29
3) I create QThread instance, e.x. named subThread, Besides call SystemEventImp->moveToThread(subThread), subThread::start() will launch a
periodic timer, that timer will create timer event to force screen redraw periodically in main thread.

How is it launching that timer and why is the subThread doing that when the timer needs to trigger work on the main thread?

Which method of the SystemEventImp is being called by the subThread and how?



I am not familiar with QThread and its scheduling policy, I want know whether my event processing method will run into a race condition and incur that freeze as I showed on my question.

I doubt this is a matter of scheduling policy and that is handled by the operating system anyway.

It is far more likely that you are dealing with a situation where things are not executed by the thread you think they are.

For example you say that the main thread is executing EventManager::ProcessQueue() but you also say you have moved the object to another thread.

What does the other thread use the object for? Which of its methods does it call?

Cheers,
_