Qt Multithreaded Design Best Practices

[bob2oneil]

I am designing a multithreaded Qt application for embedded Linux that will consist of the main GUI thread along with
several long lived secondary threads. I am currently not deriving from QThread, based on the ongoing discusssion on Qt threading
best practices, and the lack of need to derive from QThread directly.

This is discussed in the following URLs on thread affinity.

http://labs.qt.nokia.com/2010/06/17/youre-doing-it-wrong/
http://labs.qt.nokia.com/2006/12/04/threading-without-the-headache/

I currently have implemented just two threads, the main GUI thread and a network thread, but this will be expanded as the design
matures to several secondary threads. The secondary network thread is long lived and responsible for dispatching received UDP
frames to corresponding secondary threads. In my current implementation, I create a packet on the heap, and then emit a Qt signal
with this packet to the main GUI thread. The Main GUI thread has a slot connection to receive the signal from the UDP secondary
thread (producer). The main GUI thread would be responsible for processing the content of the received packet and freeing the packet allocation
upon consumption. The network dispatcher would use a similar technique to provide data to the other secondary threads.

In my worker threads, I would like to effectively implement a pending queue, such that the secondary threads pend until they have a message
to process. The secondary threads would process the contents of the message, and then pend until another message is received.

My first attempt at this was to use a QWaitCondition to pend the Main GUI thead, where it would pend after transmitting content over the
network, and then wait on a response or a timeout.

Qt examples of the producer/consumer pattern uses global variables of a shared mutex and QWaitConditions to synchronize thread activity,
but this seems to break object oriented design principles.

I was using a QWaitCondition that would be signalled from the secondary thread upon the reception of a UDP frame. What I observed is that
the Main GUI thread did not receive the data signal from the UDP secondary thread while it was in the pending state on QWaitCondition. Perhaps
this makes sense as the event loop was not running in the main GUI thread, and without the event loop running, it would seem that the
thread is not receptive to a signal/event generated from a secondary thread. Assuming that the producer thread was to first awaken
the consumer thread via QWaitCondition.Wake(), it would seem there would be a race condition between awakening the thread, and then
sending out a signal containing the UDP packet.

It would appear that the GUI thread has to be running in its event loop to be able to receive a signal from a secondary thread, which under
the covers is implemented in Qt as an thread safe event.

I want to use the cross thread safe signal/slot mechanism to communicate between worker threads. This would seem to require that
the long lived secondary threads call the exec() function on their run() routines (when deriving from QThread). The event loop
exec() function is called by default in a QThread's run() routine.

Does exec() in a secondary thread actually pend the thread until activity is required such as in response to a slot being called or other signals received by the thread? I want to minimizing CPU utilization in the overall system such as if a secondary thread
has no data to process, it is in a pending state. Is this not possible when using a thread running an event loop and hence exec()?

Does anyone have an suggestions on implementing a multithreaded application based on signals and slots with long lived pending threads?

The following code snipnet illustrates my current secondary thread creation and signal/slot connections.



// 1. Create secondary threads (TODO: add others here)
QThread socketThread;

// 2. Connect objects that require threading to the thread start signal (TODO: add others)
Q_ASSERT(w.m_socket.connect(&socketThread, SIGNAL(started()), SLOT(start())) == true);

// 3. Connect object signals and slots together (TODO: add others)
Q_ASSERT(QObject::connect(&w.m_socket, SIGNAL(receiveGuiPacket(QByteArray *)), &w, SLOT(processPacket(QByteArray *))) == true);

// 4. Move objects that require threading to threads (TODO: add others)
// Tell Qt to ensure that event handlers, and by extension signals and slots, are called from the specified thread context
w.m_socket.moveToThread(&socketThread);

// 5. Start the threads (TODO: add others her)
socketThread.start();

// 6. Enter the main event loop and start event handling
int result = a.exec();

// 7. Tell all threads to shut down (TODO: add others here)
const int StopWait = 1000;
socketThread.quit();

// 8. Wait for all secondary threads to finish (TODO: add others here)
socketThread.wait(StopWait);

// Done
return result;

[wysota]

Wow... another thread about doing multi-threaded networking...

[bob2oneil]

Hi wysota, this content is currently about multithreaded networking, but it will expand to include several worker threads (such as USB media writers, etc.), many of which are feed data over the network.

So what are your thoughts?

I did read your response to a similar question referencing the following:

http://doc.trolltech.com/qq/qq27-responsive-guis.html

This article provides some insight, but does not address some of the questions I described.

[TemporalBeing]

Well, to answer your questions...

recent default implementation of QThread::run() I think from Qt4.3 and later, simply calls the exec() to use the signal/slots. So all you should have to do is move your various objects to the various threads you want to use.

That said, I highly suggest looking into using the more advanced pointer calsses Qt offers, namely QSharedPointer and QSharedDataPointer, as they would alleviate the need for any one place in your program to officially delete data.

HTH,

Ben

[bob2oneil]

Thanks Ben, I have been using QPointer<> throughout my code, and will consider your suggestions for the others to have "safe pointers" that become zero upon deletion.

What I do not fully understand is whether or not a thread whose run() method (which simply calls exec()) actually pends? I want my secondary threads to pend until they have data to process, hence
the pending queue requirement.

It would seem that the event loop has to be somewhat "running" in the thread for it to both generate signals and respond to invoked slots. This is atypical of nearly all embedded systems that I have
worked on, where tasks lay dormant until necessary for them to perform work.

There has to be some method to have Qt act more like an embedded system of cooperatives tasks, with atomic queues, pending queues, separate task priorities, etc.

My target hardware is a 1.6 GHz Atom processor, single core, so code techniques that utilize threads running on separate cores does not apply.

[wysota]

Hi wysota, this content is currently about multithreaded networking, but it will expand to include several worker threads (such as USB media writers, etc.), many of which are feed data over the network.

So what are your thoughts?
My thoughts are there is a very good chance you don't need threads for "communication" at all. Threads should be used when they have something to do, not when you have some system resources lying unutilized. Communication is by definition asynchronous and making it synchronous is an artificial thing to do. Using threads to counter this artificiality is another artificial thing to do. So why add overhead and complexity if you can do it the easy and direct way.

Bottom line: Use threads if you intend to keep them busy, use signals, slots and events when you want to be notified when something happens. If you can't use asynchronous communication then provide as little code as possible to emulate it using synchronous communication (i.e. use a thread with blocking calls and emit signals or post events from the thread when something interesting happens).

http://libusb.sourceforge.net/api-1.0/group__asyncio.html

[bob2oneil]

Thanks wysota.

I am a big believer in the KISS principle (keep it simple stupid), and would only use threads if absolutely necessary. Therefore, in general, I agree with your advice. I will have to see if I can achieve our design goals
without threading. I have used STL quite a bit in the past to achieve cross platform C++ support for common multitasking concepts such as semaphores, queues, threads, etc. WxWidgets (not to be confused with your
wwWidgets) provides a nice cross platform wrapper on typical threading concepts and communications. I do not have as clear a vision of the Qt equivalence of these per my questions above. I guess I need some more "learning" on Qt.

[wysota]

I am a big believer in the KISS principle (keep it simple stupid), and would only use threads if absolutely necessary.
I would use threads if they are useful. For networking they are not if the technology you use favours asynchronous approach.


I do not have as clear a vision of the Qt equivalence of these per my questions above.
You have QMutex, QSemaphore, QWaitCondition, QSystemSemaphore and a bunch of auxiliary classes (like QMutexLocker).

[bob2oneil]

Hi wysota.

I have looked at the typical multithreaded primitives in Qt, but where do I find say a semaphore with a timeout, or a waiting queue, or would I build these with other primitives such as mutex/semaphore in combination with a QQueue or timers?
As I mentioned, it would seem from the Qt examples, that you have to share QWaitConditions and mutexes between threads, which seems to break the object oriented design metaphor -- but that is ok if that is the way it has to be done.

I am used to having tasks/threads suspend on a waiting queue until work is available. It would seem to me based on my limited understanding an empirical results, that if you intend to have a thread that uses signals and slots, then you implicitly have to invoke
exec() to have the event loop called. It is not clear to me that within the bowels of exec() whether or not this thread actually pends or not? From my limited testing, having a secondary thread attempt to signal the GUI thread, that the GUI
thread has to be active and its event loop running for it to actually receive a signal. This would seem to imply that threads that need to be provided data via signals from secondary threads can not really pend, but instead have to be active, and therefore the benefit
of threading these is of little value. Perhaps that was your point in your previous message.

Assuming I wanted to construct a thread that does behave more like a typcial task in an embedded system design based on an RTOS, one that pends entirely until it has work to perform, maybe that means I can not use one based on an event loop, and therefore signals and
slots can not be used. That is ok, I just need to determine the rules of the game.

P.S.> By the way, I looked at your wwWidgets site, looks like some real nice UI components, thanks for the fine toolkit. I hope to be able to use them sometime in the future.

[wysota]

I have looked at the typical multithreaded primitives in Qt, but where do I find say a semaphore with a timeout,
What is a 'semaphore with a timeout'? Never heard of such thing.


or a waiting queue,
What's a waiting queue? There is a 'wait condition' primitive I'm aware of but not a "waiting queue".

[bob2oneil]

A semaphore with a timeout is a semaphore wait where you can also specify a 2nd condition, which is the maximum time to wait. It is implemented under Unix/Linux using the sem_timedwait() API.
It is particular useful when you have a task blocking on a semaphore waiting on some data, but you also need the task to do something periodically.

http://linux.die.net/man/3/sem_timedwait

In a previous job, we created a standard template library based cross platform library with support for Windows, Linux, and embedded (Texas Instruments TI DSP BIOS RTOS). There were Windows equivalence for Linux sem_timedwait().

A waiting queue is a queue implementation such that it pends the task until data has been received in the queue. These are very common in the embedded world.

I am not aware of any direct equivalent under Qt, but perhaps similar behavior can be built using primitives.

[JohannesMunk]

Am I missing something?

http://doc.trolltech.com/latest/qsemaphore.html#tryAcquire-2

HIH

Joh

[wysota]

A waiting queue is a queue implementation such that it pends the task until data has been received in the queue. These are very common in the embedded world.
That's what you use wait conditions or semaphores for. A primitive is called a primitive because it's primitive. A 'waiting queue' as you call it is a data structure (in your example a queue) protected with a synchronization primitive (semaphore or wait condition).

And a "semaphore with a timeout" is still a regular semaphore. Unfortunately it's a completely useless (in terms of reliability) mechanism. You can get a timeout at the exact moment the semaphore is raised which prevents you from doing the task you want and can lead to a live-lock.

[bob2oneil]

Good input, and the tryAcquire seems to be equivalent to what I have used in the past. For the semaphore with timeout and the live lock condition, when the task is enabled, I examine the queue condition first and handle this in addition to code associated with the timeout.

Does anyone have a feel for my principle question, which is whether or not a thread, using signals and slots and running its own event loop via exec(), actually pends and it awaken by say a signal invocation?

If not, if I were trying to construct such a task, would I do so only by deriving a custom run() routine which had a wait condition or semaphore or the like (and therefore would need to derive from QThread)?

[wysota]

For the semaphore with timeout and the live lock condition, when the task is enabled, I examine the queue condition first and handle this in addition to code associated with the timeout.
Unless you do the examination within the same critical section as you suspend yourself on the semaphore, the result is unreliable as well (the state can change between the check and testing the result of the check).


Does anyone have a feel for my principle question, which is whether or not a thread, using signals and slots and running its own event loop via exec(), actually pends and it awaken by say a signal invocation?
In my opinion it depends on the event loop implementation used and since Qt uses different implementations on different platforms (plus you can use your own) it is probably impossible to answer this question without studying each implementation.

But if you are asking whether receiving the signal while being suspended on a semaphore wakes the thread then the answer is no, it doesn't.

[bob2oneil]

I thought that was probably the case, my question, to some extent was rhetorical. I could not envision how the event loop could be responsive to signals, or generate slots, without it running to some extent.

Therefore, it would seem that if I want to create a thread whose behavior mimics the "task" metaphor that is typical with nearly all embedded system developments I have been involved with, then I would need
to have my own run() routine, one where I would create my own pend criteria (QWaitCondition, or QSemaphore, or QMutex, etc.). It would also seem that since I can not have exec() invoked as my task's execution
loop, then I can not use signals and slots for these particular threads, and take advantage of the thread safe nature of signals and slots. This would seem to imply the need to derive from QThread, which contradicts
the two URLs referenced at the beginning of my message.

Is this summary correct, or am a missing something?

[wysota]

Yes, you are missing something. This something is (or are) called QtConcurrent and QRunnable.

[bob2oneil]

Just started re-reading about these in "Advanced Qt Programming" by Summerfield.
I think I understand now how to accomplish my design goals w.r.t to threading.