Thread Design Question

[craigdillabaugh]

Hi,
I am just learning how to work with threads and I wanted to know if someone could help me with suggestions as to how I might go about solving the following problem. I don't need code (although it would be great), just some ideas on how to design my application.

I have an application that needs to download a large number of large files from an FTP server and then perform some processing on those files. Since downloading will take quite a while I want to start processing files as their downloads complete. To further complicate things the 'processing' is being done by an external program that I will likely use by calling the C system() function.

In reading the documentation for QSemaphore I thought I could modify the example program at:
http://doc.trolltech.com/4.5/threads-semaphores.html
for my needs.

I would have a Producer class which downloads the file and a Conumser class which processes the file. My 'buffer' will simply be a list of file names (I use QFtp to get a listing beforehand) I want the Producer to acquire a filename and release it once completed. My Consumer would then acquire the filenames as they were released.

My problems are:
1. I use QFtp to download my files. If I start a download (or series of downloads) from Producer::run() how can I pause that thread until I get the Finished signal back from my QFtp notifying me that a download has been completed, so that I can then release a resource to the Consumer thread and start my next QFtp instance.

2. In my Consumer thread I would like to know when the process spawned by my 'system()' calls finishes :crying:. My idea was to modify the program being called so that it outputs a temporary file when it finishes (I have access to the code), and occassionally check for this file as notification that the system call is done. Is there a better way to do this?

Thank you for any suggestions.
Craig

2.

[fullmetalcoder]

Have you considered using a QThreadPool (http://doc.trolltech.com/latest/qthreadpool.html) instead of manually managing the threading? It could dramatically simplify your design while making it slightly more scalable.

Also, if the external app used for processing accepts input from stdin you could use QProcess (http://doc.trolltech.com/latest/qprocess.html) to stream the data from QFtp (http://doc.trolltech.com/latest/qftp.html) instead of waiting for the download to complete. Another advantage of using QProcess (http://doc.trolltech.com/latest/qprocess.html) instead of system() is that it answers your second question.

[craigdillabaugh]

Have you considered using a QThreadPool (http://doc.trolltech.com/latest/qthreadpool.html) instead of manually managing the threading? It could dramatically simplify your design while making it slightly more scalable.

Also, if the external app used for processing accepts input from stdin you could use QProcess (http://doc.trolltech.com/latest/qprocess.html) to stream the data from QFtp (http://doc.trolltech.com/latest/qftp.html) instead of waiting for the download to complete. Another advantage of using QProcess (http://doc.trolltech.com/latest/qprocess.html) instead of system() is that it answers your second question.

Thanks for the suggestions. I am looking into both. I am still confused. The following is a modified version of what is included in the QThreadPool docs that shows my specific problem:



class DownLoader : public QRunnable
{
void run()
{
QFtp *ftp = new QFtp("ftp.site.url");
QFile *file = new QFile();
//Setup connection etc.
//connect ftp's commandFinished to my commandFinished slot.
ftp->get("filetodownload", cur_file);
}
public slots:
void commandFinished(int commandId, bool error) { }
}

DownLoader *d = new DownLoader();
// QThreadPool takes ownership and deletes 'hello' automatically
QThreadPool::globalInstance()->start(d);


Since ftp->get() returns immediately won't my Runnable terminate prior to the download completing and me receiving the commandFinished from my QFtp object. Also, I would close the file in commandFinished, so how can I make my run() wait and then close my file.

This is of course the fundamental problem I have using regular threads too.

Craig

[fullmetalcoder]

The impossibility to block the ftp transfer is indeed an issue I overlooked which renders QThreadPool useless. However it implies that you don't need to create a thread per connection as QFtp does everything asynchronously anyway. Instead a single thread sending the get commands and waiting for them to finish (storing (command id, file name) pairs for further processing) would do.

Also I am quite sure the above code will fail as the QFile to which you want to write the data pumped from QFtp does not point anywhere and is not opened in write mode.

[craigdillabaugh]

The impossibility to block the ftp transfer is indeed an issue I overlooked which renders QThreadPool useless. However it implies that you don't need to create a thread per connection as QFtp does everything asynchronously anyway. Instead a single thread sending the get commands and waiting for them to finish (storing (command id, file name) pairs for further processing) would do.

Also I am quite sure the above code will fail as the QFile to which you want to write the data pumped from QFtp does not point anywhere and is not opened in write mode.

Thanks again. I have a solution due to your suggestion. One thing I did find out is that it does not work to call QFtp::get() from within the run() method of a QThread as while the loop in the run() method is executing the signals from QFtp are apparently not received by my class derived from QThread (as far as I can see anyways).

I've posted my (possibly not the best) solution below, in case anyone runs into the same problem. I simply run my downloader as a regular class and when a get() command finishes I launch the next ftp request. As files finish I can notify my data processing class by emitting a signal from my Downloader class (not shown).



void FileDownloader::getNext() {
//m_files is a QString vector holding all the filenames to download.
//m_cur_file is a QFile* declared as a class variable.
//m_counter is an int declared as a class variable.

if(m_counter == m_files.size()) {
emit isDone();
return;
}
else {
m_cur_file = new QFile(m_files.at(m_counter++));
if( !m_cur_file->open(QIODevice::WriteOnly)) {
qDebug() << "Unable to open " << m_cur_file->fileName();
}
else {
m_ftp->get(filename, m_cur_file);
}
}
}

void FileDownloader::commandFinished(int commandId, bool error) {
....
if (m_ftp->currentCommand() == QFtp::Get) {
if (error ) {
qDebug() << "Error downloading " << m_cur_file->fileName() << "\n";
m_cur_file->close();
m_cur_file->remove();
}
else {
m_cur_file->close();
}
delete m_cur_file;
m_cur_file = 0;
getNext(); //Now fetch the next file - getNext() stops the process
//once all files are received.
}
}