QThread with Python Popen freezing on sizable external program

[jkrienert]

As it stands, a QThread worker class enables the external call of an *.exe with Popen. While this external process is running, a QProgressBar within the governing class needs to be periodically updated.

The sample below works for a few percentage increments, then it suddenly freezes until the process is complete.


class workerThread(QThread):
progress = pyqtSignal(int)

def runProgram(self,directory):
file = directory+'fileName.txt'
program = sys.path[4] + "/Root/GIS_Pre_Processor.exe"
runProgram = Popen([program,file],shell=True,cwd=directory)
currentProgress = 45
while runProgram.poll() is None:
time.sleep(1.5) #<=== makes it through a couple of these loops
currentProgress = currentProgress + 5
self.progress.emit(currentProgress)
self.progress.emit(100)

Does this suggest an improper implementation of QThread?

[wysota]

What is the reason for the while loop? Why don't you use a timer or QTimeLine? And QProcess for handling the worker process.

[jkrienert]

I had not yet investigated QProcess, but currently I am having trouble implementing it to any functioning result.


fileName = someDirectory+'anInterestingFile.txt'
Program = sys.path[4] + "/someDirectory/console.exe"
runTheProgram = QProcess()
runTheProgram.setWorkingDirectory(outsideDirectory )
runTheProgram.finished.connect(self.parseResults)
runTheProgram.execute(Program,fileName)

Where 'fileName' is the first and only argument to be entered into the console 'Program'.
The program itself closes automatically wether successful or not, but I would like to somehow check for assurance that it indeed has
ran and executed the singular argument 'fileName' sent.

Currently it closes without executing the 'fileName' argument.

[jkrienert]

The implementation of QProcess must be off-tilt, as with the following example, the Ui hangs just as with the initial postings use of POpen...
Granted, the program does run - but eventually crashes in the process (after ~20 sec.) without any progressBar updates sent where as with the identical 'NAMfile' command used with POpen
results in success.




NAMfile = str(self.modDirectory)+'mfnam.txt'
MODFLOW = sys.path[4] + "/GraphicGroundwater/mf2005.exe"

qMODFLOW = QProcess()
qMODFLOW.setWorkingDirectory(self.modDirectory)
qMODFLOW.finished.connect(self.returnResultStats)
qMODFLOW.start(MODFLOW,[r"%s"%(NAMfile)])

qMODFLOW.waitForFinished()
while not qMODFLOW.ExitStatus():
time.sleep(0.5)
currentProgress = currentProgress + 2
self.progress.emit(currentProgress)

[jefftee]

I'm not entirely sure I understand what you're doing, but a couple of observations:

- QProcess:waitForFinished() will block for 30,000 milliseconds (30 seconds). If your process ends before 30 seconds is up, it will return. It will also return after 30 seconds expires even if your QProcess is still running. Just used the QProcess signals to get state changes and/or finished notification, etc.

- Not sure what your while loop is trying to accomplish. You will get the process exit code and exit status in the finished signal. Use that to determine whether your process ran successfully.

- Your progress bar updates are likely not reflecting the completion status since you are arbitrarily updating your progress every .5 seconds. How many seconds will the process run to completion, does it vary?

- You can read output of your QProcess, so if it's outputing any information while running, you can consume that output to get a better completion status if your process normally displays this. I have used QProcess like this in the past where i read the stdout and stderr of the QProcess to determine progress, etc.

Good luck.

[jkrienert]

Alright, inexperience clearly demands starting from scratch.

The initial postings use of POpen worked, in that the external program was called, ran, and completed its processes. Aside from freezing shortly after trying to update the progress bar, it was nominal.

The following code calls the program, and it starts - but for some reason is terminated prematurely. In that the program itself has a text output file which abruptly stops midway without any context of cause.
Since no other parameters for the inputs to the actual program changed (ergo - the argument string), I am under the assumption that something is a miss in my implementation of QProcess.


NAMfile = self.modDirectory+'mfnam.txt'
MODFLOW = sys.path[4] + "/GraphicGroundwater/mf2005.exe"
arguments = [r"%s"%(NAMfile)]
qMODFLOW = QProcess()
qMODFLOW.setWorkingDirectory(self.modDirectory)
qMODFLOW.finished.connect(self.returnResultStats)
qMODFLOW.start(MODFLOW,arguments)

Ideally, the goal is to:
-open external program
-send program singular argument
-listen for the (varied amount of) lines written in the program (as displayed in console if not running as shell)
wherein each on of these lines would help to gauge update of the progress bar
-catch signal when the external program has concluded
! All while maintaining a non-freezing Gui

Added after 1 33 minutes:

Reversion back to the (functioning) POpen might be ideal at this point in time.

I have narrowed down the problem why the external program wasn't able to conclude, in that QProcess was killing it before it finished.

Added the waitForFinished() command with the default 30 sec allowed more of the programs output file to be filled, yet since the process has a variable run length (sometimes well over 30 sec),
another means to working QProcess would have to be sought. Setting the waitForFinished() to 40 seconds allowed the process to complete fully, yet the Ui still froze.
Having to hard set this wait time seems non-sensical, so there is obviously something I am missing.

[jefftee]

I have narrowed down the problem why the external program wasn't able to conclude, in that QProcess was killing it before it finished.

Added the waitForFinished() command with the default 30 sec allowed more of the programs output file to be filled, yet since the process has a variable run length (sometimes well over 30 sec),
another means to working QProcess would have to be sought. Setting the waitForFinished() to 40 seconds allowed the process to complete fully, yet the Ui still froze.
Having to hard set this wait time seems non-sensical, so there is obviously something I am missing.
Yes, you are missing the point completely.

First, using QProcess::waitFor* methods will cause the calling thread to hang, no doubt about it. So, if you don't want your program to hang, don't use the QProcess::waitFor* methods!

If you bothered to read the documentation at all, you'd see that the default timeout for QProcess::waitForFinished is 30 seconds and you can specify a different time or use -1 for no timeout. So even if you were hell bent and insisted on using the waitForFinished method, you wouldn't have to guess the exact run time at all.

That said, I would bet that you are creating your QProcess on the stack and it goes out of scope when your function ends, which causes QProcess to go out of scope and get deleted, etc.

Please post your most recent attempt so I don't have to guess what you're doing.

Added after 16 minutes:

One more thing after reviewing this entire thread, I don't see where you have properly created a new thread, so while you think you're doing all of this in a separate thread, you are likely still executing in the same thread.

Where do you start your new QThread after it has been created?

Where is the QThread::run method?

Just creating a QThread or an object that inherits QThread does not result in a new thread being executed. You must start the thread, which causes the QThread run virtual function to be executed. I don't see where you have done either from the code snippets you have posted.

[jkrienert]

If you bothered to read the documentation at all...
Well, I understand that you hold superior experience with Qt - but are claims like this necessary?

I have read through the doc's somewhat roughly and not every single aspect. One instance I did notice is
the default timeout of 30 seconds - which is what I iterated in my last posting, followed by voicing that
I then bumped it to 40 seconds to see if I was doing something else wrong, or in fact the external process
was functioning 'properly'.

Indeed my experience limits my knowledge of some terminology and/or conventions, but your description of
'the stack' being the place where this QProcess currently is sounds accurate (?).

The snips I have posed all originate from inside a QThread called upon from another class:


class runMODFLOW(QDialog, Ui_ModflowWindow):
def __init__(self, iface, modDirectory, prefDirectory):
QDialog.__init__(self, iface.mainWindow())
self.iface = iface
self.setupUi(self)

#... some irrelevent Ui stuff ...

self.workerThread = buildMODFLOWdata()
self.workerThread.progress.connect(self.updateProg ressBar)
self.RunButton.clicked.connect(self.startWriting)

def startWriting(self):
Tolerance = self.TolerVal.text()
MaxOutIterVal = self.MaxOutIterVal.text()
self.workerThread.writeMODFLOWInputs(self.modDir,s elf.prefDir,MaxOutIterVal,Tolerance)

class buildMODFLOWdata(QThread):
progress = pyqtSignal(int)

def writeMODFLOWInputs(self,modDirectory,prefDirectory ,MaxOutIterVal,Tolerance):

#... some other time consuming processes here ...
#... amounting to signals to increase self.progress...

NAMfile = r'%s'%(self.modDirectory+'mfnam.txt')
MODFLOW = r'%s'%(sys.path[4] + "/GraphicGroundwater/mf2005.exe")
qMODFLOW = QProcess()
qMODFLOW.setWorkingDirectory(self.modDirectory)
qMODFLOW.finished.connect(self.returnResultStats)
qMODFLOW.start(MODFLOW,[NAMfile])
qMODFLOW.waitForFinished(40000)

I suppose I will receive the negativity as motivation to find another means.
Thanks.

Added after 33 minutes:

Also,
@jthomps:
I am not 'hell bent' on any actions which deviate from an ideal functioning of this program.

After rereading the Doc's into more depth - it seems that any of the wait* commands go against
my needs (agreed with jthomps), and thus resolve that removing this QProcess from the main
thread appears to be the action to take (?).

[jefftee]

Well, I understand that you hold superior experience with Qt - but are claims like this necessary?
If I offended you, I apologize. I thought that you had not read the docs for QProcess::waitForFinished() based on your statement that "Having to hard set this wait time seems non-sensical, so there is obviously something I am missing."

Back to your "hang" issue, I do not believe that you are properly using QThread and as a result, your line 16 is running in your GUI thread and not the new QThread you're attempting to use.

Starting the QThread causes the run method to be executed. If you have not overridden the run method (and you have not), then the default implementation simply calls QThread::exec(). You are attempting to run the following on line 16:



self.workerThread.writeMODFLOWInputs(self.modDir,s elf.prefDir,MaxOutIterVal,Tolerance)


The code above will execute in the GUI thread, which is why your app is hanging regardless of whether you use popen or QProcess. There are two ways that I know of to have your code execute in a separate thread:



Implement the QThread::run method that will perform the processing you want when you start the thread
Use the worker object -> moveToThread method (google it) where you create an object that has signals/slots and then move the object to the new thread. Once that is done, you can run code in the new thread by using queued signal/slot mechanism


I would recommend the latter since this seems to be the current recommendation for using QThread in Qt. Until you fix your attempted usage of QThread, I am afraid that you will not be able to achieve the desired results.

Using the waitFor* methods of QProcess would be acceptable once you are actually executing that code in a separate thread. Same for popen, your poll() would be OK if actually running in a separate thread, but unfortunately from what I can tell, all of your attempts are still executing in the GUI thread.

I am not a Python person, but here's a link from stackoverflow that shows 3 different methods of using QThread from PyQt:

http://stackoverflow.com/questions/6783194/background-thread-with-qthread-in-pyqt

Good luck.

[jkrienert]

@jthomps
I appreciate the notion of apology; sometimes the seeming confinement of text-only communications occasionally leaves me dissonant, and no offence was taken.
Thank you for your valuable clues.


(Remaining 7% battery spawns delay in repairing my misinterpretation of QThread until tomorrow.)

[jefftee]

After rereading the Doc's into more depth - it seems that any of the wait* commands go against
my needs (agreed with jthomps), and thus resolve that removing this QProcess from the main
thread appears to be the action to take (?).
To be clear, the QProcess (or popen for that matter) will create a new process at the OS level, so you can readily create the process from your main GUI thread if you want. If you decide to do that, you will need to use the signals emitted by QProcess and avoid the waitFor* functions altogether.

I recommend you use the signals because you stated that the process you're launching writes stdout or stderr that can be used by your program to monitor progress and it's trivial to use QProcess::readyReadStandardOutput and QProcess::readyReadStandardError to monitor your progress, etc. If you connect a slot to QProcess::finished, then there's really no reason this all can't be done in the GUI thread and you can remove the complexity added by trying to use QThread.

[wysota]

Instead of calling waitForFinished() you should be returning to the event loop and let signals and slots do their magic. Just make sure qMODFLOW doesn't go out of scope when the function returns.

[anda_skoa]

Ideally, the goal is to:
-open external program
-send program singular argument
-listen for the (varied amount of) lines written in the program (as displayed in console if not running as shell)
wherein each on of these lines would help to gauge update of the progress bar
-catch signal when the external program has concluded
! All while maintaining a non-freezing Gui

This is exactly what QProcess is designed to do.

You create the instance, connect to its signals (for when its finished and when output becomes available on the child process' stdout or stderr), then start the process.

Cheers,
_

[jkrienert]

Are signal slot communications the soul actions to prevent QProcess from going out of scope?

[wysota]

Are signal slot communications the soul actions to prevent QProcess from going out of scope?

Regular Python rules apply here -- if a last reference to an object is released, the object becomes eligible for garbage collecting. So the easiest thing to do is to assign the object to a property in the encapsulating class (e.g. self.qMODFLOW).

[jkrienert]

Wow. If I could only buy all of you a couple beverages of your ideal I truly would. Things are improving and working much more nominally.

I have a tangential question:
If a process duration is somewhat unknown in length - what are some conceptual actions to take with regards to updating a progress bar?
(It is supposed that the progress bar would be updated on QProcess outputs (for each byte block - update 'such'%)
-I'm guessing some simple algorithm must be built to limit reaching 100% early (ect.).

I will post an update to the code here shortly - conclusively showing the solution chosen.

Best,

Added after 1 21 minutes:

Significant improvements :
- Enables accurate updating of progress bar relative to external program
- Allows continued use of Ui without freeze
- Allows user to kill the external process via cancel button without OS hangups
- (personal opinion) much more user / programmer friendly!


class runMODFLOW(QDialog, Ui_ModflowWindow):
def __init__(self, iface, modDirectory, prefDirectory):
QDialog.__init__(self, iface.mainWindow())
self.iface = iface
self.setupUi(self)
self.RunButton.clicked.connect(self.startWriting)

def startWriting(self):
self.Tolerance = self.TolerVal.text()
self.MaxOutIterVal = self.MaxOutIterVal.text()
self.writeMODFLOWInputs()

def preMODProgress(self):
self.ModflowProgress.setValue(self.currentValue)
self.currentValue = self.currentValue + 2

def MODProgRatio(self):
totalProgressBlocks = 0
for idx,period in enumerate(range(self.numStressPeriods)):
totalProgressBlocks = int(totalProgressBlocks)+int(self.tempStressOption sCSV[idx][2])
numerator = 100 - int(self.ModflowProgress.value())
self.QProcValStep = numerator/totalProgressBlocks

def MODProgress(self):
self.ModflowProgress.setValue(self.ModflowProgress .value()+self.QProcValStep)

def writeMODFLOWInputs(self):
#... processes sending signals to preMODProgress() ...
self.preMODProgress()

self.MODProgRatio()
self.MODFLOWstart()

def MODFLOWstart(self):
NAMfile = r'%s'%(self.modDir+'mfnam.txt')
MODFLOW = r'%s'%(sys.path[4] + "/GraphicGroundwater/mf2005.exe")
self.qMODFLOW = QProcess()
self.qMODFLOW.setWorkingDirectory(self.modDir)
self.qMODFLOW.readyRead.connect(self.MODProgress)
self.qMODFLOW.finished.connect(self.finishedProgre ss)
self.qMODFLOW.start(MODFLOW,[NAMfile])

def finishedProgress(self):
#... some other housekeeping here ...
self.ModflowProgress.setValue(100)

The 'MODProgRatio()' function assesses the inputs for the ext-program, and makes a rough blocks(s) calculation of the #of QProcess output strings.
This allows a proper increment of the progress bar synchronous with the actual state of the ext-program.

Thousand thanks to all whom helped clarify my understanding of QProcess.

[jefftee]

I have a tangential question:
If a process duration is somewhat unknown in length - what are some conceptual actions to take with regards to updating a progress bar?
(It is supposed that the progress bar would be updated on QProcess outputs (for each byte block - update 'such'%)
-I'm guessing some simple algorithm must be built to limit reaching 100% early (ect.).

If the duration is truly unknown, then you can set the minimum and maximum of the QProgressBar to zero, which will show a "busy" progress bar. This is intended to show that your program is working, but the completion percentage is not known.

[jkrienert]

@jthomps:
That is an interesting alternative.
Although I wondered about a user who might be frustrated or impatient by a long process.
This concern motivated the build of a simple algorithm loosely based on the exe's inputs/outputs [in previous post 'MODProgRatio'].
Works surprisingly well.

Being new to this forum, I wondered if there is a means to mark a thread as [solved] at least w/respect to the original question?

[jefftee]

If your current progress methodology works for you, then you're all set. Too bad your external process doesn't write progress output as it would be trivial to read that output and accurately report progress to your GUI.

[jkrienert]

Its unfortunate that the external program doesn't net any preprocess calculation of expected runtime, but considering what the program is doing - I'm happy to oblige and be patient!
Its an extremely rapid finite difference calculation suite that was originally FORTRAN compiled in the 1980's by members of the USGS.

The algorithm I mentioned works by looking at the users input data. It counts the number of blocks going in, which fortunately also represent each Output line while the process is running.
Thus, before running the program; (total% / number of blocks) - and with each signal from QProcess after start(), the progress bar gets a block.
Rough functionality - but relatively close to the realities of the programs calculations.

Thanks again jthomps for your help.