Cirno
26th December 2011, 01:57
I have a script I'm writing to batch download images from an imageboard via JSON/XML api. Previously, it had been purely CLI, but recently I've been trying to build a UI in PyQt, with great success but for one problem: thread blocking issues, nonresponsive GUI when actually calling the worker threads within my script. So, I'm trying to switch from threading.Thread to QThread, to make it easier to manage (by emitting a threadFinished SIGNAL to update my GUI), but I can't seem to get it set up properly. Whenever I run the script, The threads die prematurely. I am running on windows, with PyQt4 on Python 2.7.2.
Exception KeyError: KeyError(1188,) in <module 'threading' from 'C:\Python27\lib
\threading.pyc'> ignored
QObject::killTimers: timers cannot be stopped from another thread
this is the output I receive.
Direct code in question:
in crawler.py:
## md5_queue is queue of an empty dict of md5sum/filename to be filled by Url_Download()
## queue is a queue tuple of filenames/urls
num_conn = int(max_threads)
threads = []
for download in range(num_conn):
t = Url_Download(queue, md5_queue)
t.start()
threads.append(t)
from functions.py (poorly named, I know) the Url_Download() class:
class Url_Download(QThread):
file_finished = pyqtSignal(QString, int, name="fileFinished")
def __init__(self, dl_queue, md5_queue):
self.dl_queue = dl_queue
self.md5_queue = md5_queue
QThread.__init__(self)
def run(self):
while 1:
try:
count = 0
file_url, file_path, md5 = self.dl_queue.get_nowait()
file_extension = str(file_url)[-4:]
file_name = md5 + file_extension
while count < 3:
count +=1
fetch_url(file_url, file_path, md5)
if md5 == hash_sum(file_path):
self.md5_queue.put_nowait((md5, file_name))
self.file_finished.emit("Test!", 10)
break
if count > 3:
print 'File failed to download, {} might be corrupt.'.format(file_name)
qsize = self.dl_queue.qsize()
if qsize > 0:
print 'Count Remaining: ', qsize
except Queue.Empty:
raise SystemExit
except:
traceback.print_exc(file=sys.stderr)
sys.stderr.flush()
and from GUI.py, the slot connect:
self.connect(self, SIGNAL("fileFinished(QString, int)"), self.handle_test, Qt.QueuedConnection)
Git (testing branch) for the code: https://github.com/CirnoIsTheStrongest/BiriCrawler/tree/testing
Please note, this is my first ever attempt at coding anything.
Exception KeyError: KeyError(1188,) in <module 'threading' from 'C:\Python27\lib
\threading.pyc'> ignored
QObject::killTimers: timers cannot be stopped from another thread
this is the output I receive.
Direct code in question:
in crawler.py:
## md5_queue is queue of an empty dict of md5sum/filename to be filled by Url_Download()
## queue is a queue tuple of filenames/urls
num_conn = int(max_threads)
threads = []
for download in range(num_conn):
t = Url_Download(queue, md5_queue)
t.start()
threads.append(t)
from functions.py (poorly named, I know) the Url_Download() class:
class Url_Download(QThread):
file_finished = pyqtSignal(QString, int, name="fileFinished")
def __init__(self, dl_queue, md5_queue):
self.dl_queue = dl_queue
self.md5_queue = md5_queue
QThread.__init__(self)
def run(self):
while 1:
try:
count = 0
file_url, file_path, md5 = self.dl_queue.get_nowait()
file_extension = str(file_url)[-4:]
file_name = md5 + file_extension
while count < 3:
count +=1
fetch_url(file_url, file_path, md5)
if md5 == hash_sum(file_path):
self.md5_queue.put_nowait((md5, file_name))
self.file_finished.emit("Test!", 10)
break
if count > 3:
print 'File failed to download, {} might be corrupt.'.format(file_name)
qsize = self.dl_queue.qsize()
if qsize > 0:
print 'Count Remaining: ', qsize
except Queue.Empty:
raise SystemExit
except:
traceback.print_exc(file=sys.stderr)
sys.stderr.flush()
and from GUI.py, the slot connect:
self.connect(self, SIGNAL("fileFinished(QString, int)"), self.handle_test, Qt.QueuedConnection)
Git (testing branch) for the code: https://github.com/CirnoIsTheStrongest/BiriCrawler/tree/testing
Please note, this is my first ever attempt at coding anything.