I have a script I'm writing to batch download images from an imageboard via JSON/XML api. Previously, it had been purely CLI, but recently I've been trying to build a UI in PyQt, with great success but for one problem: thread blocking issues, nonresponsive GUI when actually calling the worker threads within my script. So, I'm trying to switch from threading.Thread to QThread, to make it easier to manage (by emitting a threadFinished SIGNAL to update my GUI), but I can't seem to get it set up properly. Whenever I run the script, The threads die prematurely. I am running on windows, with PyQt4 on Python 2.7.2.
Exception KeyError: KeyError(1188,) in <module 'threading' from 'C:\Python27\lib
\threading.pyc'> ignored
QObject::killTimers: timers cannot be stopped from another thread
Exception KeyError: KeyError(1188,) in <module 'threading' from 'C:\Python27\lib
\threading.pyc'> ignored
QObject::killTimers: timers cannot be stopped from another thread
To copy to clipboard, switch view to plain text mode
this is the output I receive.
Direct code in question:
in crawler.py:
## md5_queue is queue of an empty dict of md5sum/filename to be filled by Url_Download()
## queue is a queue tuple of filenames/urls
num_conn = int(max_threads)
threads = []
for download in range(num_conn):
t = Url_Download(queue, md5_queue)
t.start()
threads.append(t)
## md5_queue is queue of an empty dict of md5sum/filename to be filled by Url_Download()
## queue is a queue tuple of filenames/urls
num_conn = int(max_threads)
threads = []
for download in range(num_conn):
t = Url_Download(queue, md5_queue)
t.start()
threads.append(t)
To copy to clipboard, switch view to plain text mode
from functions.py (poorly named, I know) the Url_Download() class:
file_finished
= pyqtSignal
(QString,
int, name
="fileFinished")
def __init__(self, dl_queue, md5_queue):
self.dl_queue = dl_queue
self.md5_queue = md5_queue
def run(self):
while 1:
try:
count = 0
file_url, file_path, md5 = self.dl_queue.get_nowait()
file_extension = str(file_url)[-4:]
file_name = md5 + file_extension
while count < 3:
count +=1
fetch_url(file_url, file_path, md5)
if md5 == hash_sum(file_path):
self.md5_queue.put_nowait((md5, file_name))
self.file_finished.emit("Test!", 10)
break
if count > 3:
print 'File failed to download, {} might be corrupt.'.format(file_name)
qsize = self.dl_queue.qsize()
if qsize > 0:
print 'Count Remaining: ', qsize
except Queue.Empty:
raise SystemExit
except:
traceback.print_exc(file=sys.stderr)
sys.stderr.flush()
class Url_Download(QThread):
file_finished = pyqtSignal(QString, int, name="fileFinished")
def __init__(self, dl_queue, md5_queue):
self.dl_queue = dl_queue
self.md5_queue = md5_queue
QThread.__init__(self)
def run(self):
while 1:
try:
count = 0
file_url, file_path, md5 = self.dl_queue.get_nowait()
file_extension = str(file_url)[-4:]
file_name = md5 + file_extension
while count < 3:
count +=1
fetch_url(file_url, file_path, md5)
if md5 == hash_sum(file_path):
self.md5_queue.put_nowait((md5, file_name))
self.file_finished.emit("Test!", 10)
break
if count > 3:
print 'File failed to download, {} might be corrupt.'.format(file_name)
qsize = self.dl_queue.qsize()
if qsize > 0:
print 'Count Remaining: ', qsize
except Queue.Empty:
raise SystemExit
except:
traceback.print_exc(file=sys.stderr)
sys.stderr.flush()
To copy to clipboard, switch view to plain text mode
and from GUI.py, the slot connect:
self.connect(self, SIGNAL("fileFinished(QString, int)"), self.handle_test, Qt.QueuedConnection)
self.connect(self, SIGNAL("fileFinished(QString, int)"), self.handle_test, Qt.QueuedConnection)
To copy to clipboard, switch view to plain text mode
Git (testing branch) for the code: https://github.com/CirnoIsTheStronge...r/tree/testing
Please note, this is my first ever attempt at coding anything.
Bookmarks