Concurrent execution of QThread or QRunnable (also how to terminate the latter?)

[Deusdies]

I have the following code:

from PySide.QtCore import *
import time
 
class GUI(object):
 
    IDLIST = [i for i in xrange(20)]
    UNUSEDIDS = [i for i in xrange(20)]
 
    def __init__(self):
        print "GUI CLASS INITIALIZED!"
 
        worker = Worker()
        worker2 = Worker2()
 
        threadpool = QThreadPool()
        threadpool.setMaxThreadCount(10)
 
        for i in xrange(5):
            #Alternate between the two
            #threadpool.start(worker)
            #worker2.start()
 
 
    @classmethod
    def delegator(self):
        """Irrelevant to the question, I need this method for something else"""
        USEDIDS = []
 
        toUse = self.UNUSEDIDS[0]
        USEDIDS.append(toUse)
        self.UNUSEDIDS.pop(0)
 
        return toUse
 
 
class Worker(QRunnable):
 
    def __init__(self, parent=None):
        super(Worker, self).__init__(parent)
 
    def run(self):
        #idInUse = getattr(GUI, "delegator")
        idInUse = GUI.delegator()
        print "Hello world from QRunnable", idInUse
        #time.sleep(5)
 
class Worker2(QThread):
 
    def __init__(self, parent=None):
        super(Worker2, self).__init__(parent)
 
    def run(self):
        idInUse = GUI.delegator()
        print "Hello world from QThread", idInUse    
 
 
s = time.time()
GUI()
print "Done in %s" % ((time.time()-s) * 1000)

from PySide.QtCore import *
import time

class GUI(object):

    IDLIST = [i for i in xrange(20)]
    UNUSEDIDS = [i for i in xrange(20)]

    def __init__(self):
        print "GUI CLASS INITIALIZED!"

        worker = Worker()
        worker2 = Worker2()

        threadpool = QThreadPool()
        threadpool.setMaxThreadCount(10)

        for i in xrange(5):
            #Alternate between the two
            #threadpool.start(worker)
            #worker2.start()


    @classmethod
    def delegator(self):
        """Irrelevant to the question, I need this method for something else"""
        USEDIDS = []

        toUse = self.UNUSEDIDS[0]
        USEDIDS.append(toUse)
        self.UNUSEDIDS.pop(0)

        return toUse


class Worker(QRunnable):

    def __init__(self, parent=None):
        super(Worker, self).__init__(parent)

    def run(self):
        #idInUse = getattr(GUI, "delegator")
        idInUse = GUI.delegator()
        print "Hello world from QRunnable", idInUse
        #time.sleep(5)

class Worker2(QThread):

    def __init__(self, parent=None):
        super(Worker2, self).__init__(parent)

    def run(self):
        idInUse = GUI.delegator()
        print "Hello world from QThread", idInUse    


s = time.time()
GUI()
print "Done in %s" % ((time.time()-s) * 1000)

To copy to clipboard, switch view to plain text mode 

I think the desired effect is obvious from the code. I want the "Hello world from QThread/QRunnable " to be shown. Since I am writing a multi-threaded application, in my GUI __init__ part I have the loop that starts concurrent threads.

The thing is that, with QRunnable it works just fine. All the 5 threads I specified get executed at once, concurrently. With QThread, however, that is not the case. Instead, I get the following error:

QThread: Destroyed while thread is still running

QThread: Destroyed while thread is still running

To copy to clipboard, switch view to plain text mode 

And it is not executed at all.

Normally I would not at all mind using the QRunnable, however, it does not derive from QObject (so I can't directly emit signals though I can construct a QObject() within it) and also it does not have the .stop() method which I badly need. Googling revealed that there is no way to stop a QRunnable from executing? On the other hand, QThread has both of these methods that I need.

So I guess my question is either how to make multiple same QThreads run concurrently, or how to terminate an execution of a QRunnable?

[amleto]

why cant you just make and use several QThread instances?

[Deusdies]

You mean why not make them all separate classes? Well, one because I need them all to be identical (except for some small parts that that Delegator() method decides) and two because I need to create hundreds of such threads.

[amleto]

no, I said instances, not classes.

for i in xrange(5):
            #Alternate between the two
            #threadpool.start(worker)
            #worker2.start() # no, this is wrong
            wrk2 = Worker2() # make a new Worker2.
            wrk2.start() # you cant expect 'start()' to work properly if the thread is till running...

for i in xrange(5):
            #Alternate between the two
            #threadpool.start(worker)
            #worker2.start() # no, this is wrong
            wrk2 = Worker2() # make a new Worker2.
            wrk2.start() # you cant expect 'start()' to work properly if the thread is till running...

To copy to clipboard, switch view to plain text mode 

You know you dont *need* to inherit from QThread in order to have the work done *in* the thread?

[Deusdies]

Oh sorry about that - yes I tried that too. The result is:

GUI CLASS INITIALIZED!!!
Done in 3.00002098083
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running

GUI CLASS INITIALIZED!!!
Done in 3.00002098083
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running

To copy to clipboard, switch view to plain text mode 

If I add a .wait() in that for loop, it works, but then it is not really concurrent.

And yes I am aware that I don't need to inherit from QThread, but I need to... I mean, the rest of the application requires me to (don't ask).

[wysota]

What if you change "worker2" to "self.worker2" (that is make it an object member)?

[Deusdies]

Same thing happens

[amleto]

Oh sorry about that - yes I tried that too. The result is:

GUI CLASS INITIALIZED!!!
Done in 3.00002098083
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running

GUI CLASS INITIALIZED!!!
Done in 3.00002098083
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running

To copy to clipboard, switch view to plain text mode 

If I add a .wait() in that for loop, it works, but then it is not really concurrent.

And yes I am aware that I don't need to inherit from QThread, but I need to... I mean, the rest of the application requires me to (don't ask).

please show your code that does this. If truly is different instances that are being destroyed then it must be the GC, which seems unlikely.

[Deusdies]

It isn't any different from the one I gave already, except for the instance creation part...

from PySide.QtCore import *
import time
 
class GUI(object):
 
    IDLIST = [i for i in xrange(20)]
    UNUSEDIDS = [i for i in xrange(20)]
 
    def __init__(self):
        print "GUI CLASS INITIALIZED!"
 
        worker = Worker()
        worker2 = Worker2()
 
        threadpool = QThreadPool()
        threadpool.setMaxThreadCount(10)
 
        for i in xrange(5):
            #Tried:
            #worker2 = Worker2()
            #worker2.start()
            #...and...
            #self.worker2 = Worker2()
            #self.worker2.start()
 
 
    @classmethod
    def delegator(self):
        """Irrelevant to the question, I need this method for something else"""
        USEDIDS = []
 
        toUse = self.UNUSEDIDS[0]
        USEDIDS.append(toUse)
        self.UNUSEDIDS.pop(0)
 
        return toUse
 
 
class Worker(QRunnable):
 
    def __init__(self, parent=None):
        super(Worker, self).__init__(parent)
 
    def run(self):
        #idInUse = getattr(GUI, "delegator")
        idInUse = GUI.delegator()
        print "Hello world from QRunnable", idInUse
        #time.sleep(5)
 
class Worker2(QThread):
 
    def __init__(self, parent=None):
        super(Worker2, self).__init__(parent)
 
    def run(self):
        idInUse = GUI.delegator()
        print "Hello world from QThread", idInUse    
 
 
s = time.time()
GUI()
print "Done in %s" % ((time.time()-s) * 1000)

from PySide.QtCore import *
import time

class GUI(object):

    IDLIST = [i for i in xrange(20)]
    UNUSEDIDS = [i for i in xrange(20)]

    def __init__(self):
        print "GUI CLASS INITIALIZED!"

        worker = Worker()
        worker2 = Worker2()

        threadpool = QThreadPool()
        threadpool.setMaxThreadCount(10)

        for i in xrange(5):
            #Tried:
            #worker2 = Worker2()
            #worker2.start()
            #...and...
            #self.worker2 = Worker2()
            #self.worker2.start()


    @classmethod
    def delegator(self):
        """Irrelevant to the question, I need this method for something else"""
        USEDIDS = []

        toUse = self.UNUSEDIDS[0]
        USEDIDS.append(toUse)
        self.UNUSEDIDS.pop(0)

        return toUse


class Worker(QRunnable):

    def __init__(self, parent=None):
        super(Worker, self).__init__(parent)

    def run(self):
        #idInUse = getattr(GUI, "delegator")
        idInUse = GUI.delegator()
        print "Hello world from QRunnable", idInUse
        #time.sleep(5)

class Worker2(QThread):

    def __init__(self, parent=None):
        super(Worker2, self).__init__(parent)

    def run(self):
        idInUse = GUI.delegator()
        print "Hello world from QThread", idInUse    


s = time.time()
GUI()
print "Done in %s" % ((time.time()-s) * 1000)

To copy to clipboard, switch view to plain text mode 

[amleto]

oh, the problem is the way you close the threads! (I think)


close them more politely (call quit() / wait()) instead of ignoring them and letting end of program clean them up.

[Deusdies]

Now I'm getting some very interesting results.

First run output:

GUI CLASS INITIALIZED!!!
Done in 2.99009998083
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running

GUI CLASS INITIALIZED!!!
Done in 2.99009998083
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running

To copy to clipboard, switch view to plain text mode 

Second run output:

GUI CLASS INITIALIZED!!!
UNUSEDIDS [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 0
UNUSEDIDS [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 1
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
UNUSEDIDS [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 2
Done in 13.9999389648

GUI CLASS INITIALIZED!!!
UNUSEDIDS [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 0
UNUSEDIDS [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 1
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
UNUSEDIDS [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 2
Done in 13.9999389648

To copy to clipboard, switch view to plain text mode 

Third run output

GUI CLASS INITIALIZED!!!
UNUSEDIDS UNUSEDIDS[ UNUSEDIDS0[ , 0[1, 0, 1, 2, 1, 2, 3UNUSEDIDS, 2,  3, 4[, 3, 04, 5UNUSEDIDS, , 4, Done in 2.00009346008 15, 6
[, , 5, 026, 7, , , 6,

GUI CLASS INITIALIZED!!!
UNUSEDIDS UNUSEDIDS[ UNUSEDIDS0[ , 0[1, 0, 1, 2, 1, 2, 3UNUSEDIDS, 2,  3, 4[, 3, 04, 5UNUSEDIDS, , 4, Done in 2.00009346008 15, 6
[, , 5, 026, 7, , , 6,

To copy to clipboard, switch view to plain text mode 

Fourth run output

GUI CLASS INITIALIZED!!!
UNUSEDIDS [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 0
UNUSEDIDS [1, 2, 3, 4, 5, 6, UNUSEDIDS7 , [Done in 3.9999485015981
, , QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
92, , 10

GUI CLASS INITIALIZED!!!
UNUSEDIDS [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 0
UNUSEDIDS [1, 2, 3, 4, 5, 6, UNUSEDIDS7 , [Done in 3.9999485015981
, , QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
92, , 10

To copy to clipboard, switch view to plain text mode 

Fifth run output the python.exe crashed (Windows displayed the "not responding" message).

What's obvious is that there are different results from every single run. This prompted me to believe that either there's a problem with GC, or that there's an actual bug in Qt.

It crossed my mind to do the following:

self.workers = [Worker2(),Worker2(),Worker2(),Worker2(),Worker2()]
        for i in xrange(5):
            self.workers[i].start()
            time.sleep(0.01)

self.workers = [Worker2(),Worker2(),Worker2(),Worker2(),Worker2()]
        for i in xrange(5):
            self.workers[i].start()
            time.sleep(0.01)

To copy to clipboard, switch view to plain text mode 

In other words, sleep for 10 miliseconds after every thread start. Interestingly enough, this (sort-of) worked. Here's the result I got every time after 5 runs:

GUI CLASS INITIALIZED!!!
UNUSEDIDS [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 0
UNUSEDIDS [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 1
UNUSEDIDS [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 2
UNUSEDIDS [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 3
UNUSEDIDS [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 4
Done in 49.9999523163

GUI CLASS INITIALIZED!!!
UNUSEDIDS [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 0
UNUSEDIDS [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 1
UNUSEDIDS [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 2
UNUSEDIDS [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 3
UNUSEDIDS [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Hello world from QThread 4
Done in 49.9999523163

To copy to clipboard, switch view to plain text mode 

While not quite as efficient as the QRunnable equivalent (which gets done in 10 ms but this is because the QRunnable's for loop does not have the time.sleep at the end), it's better than nothing. Since my application will be limited to 20 (30) threads at most, the performance impact should be negligible.

I've also noticed that if I reduce the sleep() period to 1 milisecond, the code becomes much more unstable; while it worked fine the first 3 runs, it crashed the interpreter on the 4th run. I am assuming that the higher the value of the sleep is, the more stable the code will be.

[amleto]

no, you should not be hacking a fix that only addresses the symptoms.

As I say, the problem is at close down - you need to save a handle to all QThreads, and then call wait/quit/terminate on them when you have done everything else in the main thread:

class GUI(object):
  def join(self):
    for w in self.workers:
      w.wait()
...
 
s = time.time()
g = GUI()
g.join()
print "Done in %s" % ((time.time()-s) * 1000)

class GUI(object):
  def join(self):
    for w in self.workers:
      w.wait()
...

s = time.time()
g = GUI()
g.join()
print "Done in %s" % ((time.time()-s) * 1000)

To copy to clipboard, switch view to plain text mode