Need concurrency advice with callback function

[BreakBad]

(PyQt 4.8, Python 2.7)
After reading through the following I'm still left a bit puzzled as to which approach to attempt to utilize for my situation. Any advice would be appreciated.
Keeping_the_GUI_Responsive (http://www.qtcentre.org/wiki/index.php?title=Keeping_the_GUI_Responsive)

I have a module, flp.so library to which I have no control over. It takes a function as a constructor argument, and performs a complex/long calculation when invoking the run() method. Here I subclassed QThread, and invoke flp.run inside the thread's run method. In turn the flp module will invoke the callback method, which emits a signal. I catch the signal back in my main thread, and update a model for a QTreeView and update a QProgressBar.


def my_runner(QThread):
def __init__(self,callback=None):
self.callback = callback

def callback(self):
# f_runner.f_output is array that is appended every iteration of f_runner.run()
emit(SIGNAL('update_main'),self.f_runner.f_output. pop())

def run(self):
import flp
self.f_runner = flp.f_runner(callback=self.callback)
self.f_runner.run()


However, the GUI is not very responsive, as the signals are probably stacking up and the events can't be processed fast enough. I try to avoid invoking QApplication.processEvents(), but tried it here and very bad things happen.

I have been rethinking this design but have gotten stuck. Eventually I would like the main thread GUI to be responsive while this thread is working and updating the progress bar and table view in the main.

TYVM for reading.

[wysota]

I'd have to have some more info to give you any precise hints. Currently I can only advise to batch updates to the model so that the view doesn't have to redraw itself each time you add a row to the model.

[BreakBad]

Thanks for the response!

The only thing thats happening on the main thread side is connecting the update signal to this slot, where I've just been tinkering with repaint, etc....:



self.connect(my_runner,SIGNAL('update_main'),self. update_main)
....
def update_main(self,data):
self.run_progress_bar.increment()
#self.run_progress_bar.repaint()
row = self.output_model.rowCount()
for col in range(len(data)):
item = QtGui.QStandardItem(data[col])
#self.output_model.beginInsertRows(QtCore.QModelIn dex(),row,row)
self.output_model.setItem(row,col,item)
#self.output_model.endInsertRows()


The runs could be anywhere from a few rows to 1,000 rows, each row taking anywhere from a fraction of a second to half a second to calculate. Eventually my 'customers' would like to also have multiple runs going concurrently. We have 16 processor/core workstations and would like to utilize them. So perhaps using Runnables would be better?

I agree, that batching the rows of data would be much better. I'll look into creating a consumer thread with a timer *edit: instead of a producer with a signal

[wysota]

If your GUI gets frozen then provided that worker threads are running correctly, the problem is with refreshing views thus batching is something that can give you the biggest speedup. I suggest you try it and see for yourself.