QFtp::Get, yet one more zero-length files thread

[i92guboj]

Hello.

I am trying to get a bunch of files from an ftp repo.



///
/// FTP stuff
///

ftp = new QFtp;
ftp->connectToHost("moo.cow");
ftp->setTransferMode(QFtp::Passive);
ftp->login("foo", "bar");
ftp->cd(QString("/"));
lista_archivos_ftp = new QStringList;
ftp->list("/*.jpg");

//ftp.cd("img_art");

connect(ftp, SIGNAL(dataTransferProgress(qint64, qint64)),
this, SLOT(my_gestor_progreso(qint64, qint64)));

connect(ftp, SIGNAL(commandStarted(int)),
this, SLOT(my_empieza_comando_ftp(int)));

connect(ftp, SIGNAL(commandFinished(int,bool)),
this, SLOT(my_acaba_comando_ftp(int,bool)));

connect(ftp, SIGNAL(done(bool)),
this, SLOT(my_hecho_ftp(bool)));

connect(ftp, SIGNAL(stateChanged(int)),
this, SLOT(my_estado_cambiado_ftp(int)));

connect(ftp, SIGNAL(listInfo(QUrlInfo)),
this, SLOT(my_list_list_ftp(QUrlInfo)));


Besides the usual stuff, I have the following relevant methods:



void kooker::my_acaba_comando_ftp(int comando, bool error)
{
if(error)
{
// booo
}
else
{
if(ftp->currentCommand() == QFtp::List)
on_btn_descargar_clicked();
}
}

void kooker::my_list_list_ftp(QUrlInfo url)
{
my_print_log(QString(Q_FUNC_INFO).append(url.name( )));
lista_archivos_ftp->append(url.name());
}

void kooker::on_btn_descargar_clicked()
{
qDebug() << Q_FUNC_INFO;
QEventLoop loop;
connect(ftp, SIGNAL(commandFinished(int,bool)),
&loop, SLOT(quit()));

QString item;
foreach(item, *lista_archivos_ftp)
{
QString img_local_path = QString(photoDir->path().append("/%1").arg(item));
QFile *file = new QFile(img_local_path);

if(!file->open(QIODevice::WriteOnly))
{
delete file;
return;
}

QString ftp_file = QString("/%1").arg(item);
ftp->get(ftp_file, file);
loop.exec();

file->close();
}
}


Note that, despite its name, on_btn_descargar_clicked() is called when the ftp list command completes, and not via a button. The funniest part is that, sometimes, I get a single zero bytes jpg file, it seems to hang in QFtp::get forever since the loop doesn't quit(). But some other times, I get all the files, also with zero length.

So, please, can you tell me if you see anything wrong with the code I wrote? I am having a really difficult time with this one. I've been struggling with this for days. Oh, and before you ask, I have also tried the loop-less (AKA standard) version, like this, with the same results:



void kooker::on_btn_descargar_clicked()
{
qDebug() << Q_FUNC_INFO;
// QEventLoop loop;
// connect(ftp, SIGNAL(commandFinished(int,bool)),
// &loop, SLOT(quit()));

QString item;
foreach(item, *lista_archivos_ftp)
{
QString img_local_path = QString(photoDir->path().append("/%1").arg(item));
ftp_local_file = new QFile(img_local_path);

if(!ftp_local_file->open(QIODevice::WriteOnly))
{
delete ftp_local_file;
return;
}

QString ftp_file = QString("/%1").arg(item);
ftp->get(ftp_file, ftp_local_file);
// loop.exec();

// file->close();
}
}

void kooker::my_acaba_comando_ftp(int comando, bool error)
{
if(error)
{
// booooooo
}
else
{
if(ftp->currentCommand() == QFtp::List)
on_btn_descargar_clicked();

if(ftp->currentCommand() == QFtp::Get || ftp->currentCommand() == QFtp::Put)
ftp_local_file->close();
}
}


In this case, I seem to always get the whole bunch of files, never a single one, but also with zero length.


Any tip is welcome, I am a bit stuck with this one :(

[stampede]

Can you download the files with Qt FTP example application (http://qt-project.org/doc/qt-4.8/network-qftp.html) ?

[i92guboj]

I can download them even with my QEventLoop based code in another tool I made the past week. But now it doesn't work in this new program. I have no idea why. Either I am missing something that's really simple or there's some obscure bug in QFtp.

In the case of the QEventLoop version, the commandFinished is never emitted, why?

In the case of the regular version, I do what any other example in the net does (included the one you posted, which I have reviewer a dozen times by now): open the file and get it when currentCommand() == QFtp::List, and close() it when currentCommand() == QFtp::Get. Right?

The slot tied fo transferProgress() is telling me that the data is being downloaded, and a new one I added to the readyRead() signal also shows the data is being downloaded... I really don't know what to think of this.




void kooker::my_readyRead()
{
QByteArray array = ftp->readAll();
qDebug() << Q_FUNC_INFO << QString("GOT: '%1'").arg(QString(array));
}


// produces qDebug() output below....
=========================================

void kooker::my_commandStarted(int) "Launched command 19: '8'"
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 0 bytes out of 4029."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 1448 bytes out of 4029."
void kooker::my_readyRead() "GOT: 'ÿØÿà '"
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 2896 bytes out of 4029."
void kooker::my_readyRead() "GOT: '‚¤r÷{ã?•vv*ÄQÏ‚9G ÝqØú{ŠR„fIÄöŠât� �¶ÂÂ¦Â“;W
2I'õ5…áÍ{Ã*ñ}žà …¸� �Â‚3þx®…‘dR®¡”õd Ã¥qÃ¥vféÝhd¿ˆ¬c”DDÆBÃ� �=¡:1 cëÍ=µ¸A¨2Î͵U¾Pzó ŸN=t°·vRÂ¥v›u㜏ӊb é9–ÖH›o˜¢'"
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 2920 bytes out of 4029."
void kooker::my_readyRead() "GOT: ')®2¤S¨ë@˜žD=f±Õ͎¢$?w'"
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 4029 bytes out of 4029."
void kooker::my_readyRead() "GOT: '8oÂ¥o^!Gá5›©[_1yUÁô"K©©Ô³õ*bÁÃ*Â¥'"
void kooker::my_commandStarted(int) "Launched command 20: '8'"


A funny thing is that 1 second after I run this code, +100 empty files are created or updated, and their modification time is set to the current time. But, ftp goes on (at least that's what qDebug() is telling me) for almost one minute, fetching files with are thrown to a QFile that's been close a year ago, hence, ending nowhere. I know what the problem is. I have no idea *where* it lies, or how to fix it by any means.

[stampede]

Of course you are aware that ftp->get() is asynchronous, right ? So if you have one "global" pointer to QFile and you assign it to each get() command in a loop, without waiting for previous get() to finish, things are gonna go wrong. In your "ftp_command_finished()" slot you call close() only on the last QFile object, so yes, you can see bunch of files with 0 length created, and never really closed.
Better, at least more clear, approach would be to create a queue of pending downloads, and start next pending job only when previous is done.

I can download them even with my QEventLoop based code in another tool I made the past week
So refactor that code and reuse the needed components ;)

[i92guboj]

Yes, I am aware...

I am also sure I missing something obvious, but can't imagine what. First thing first, the slot tied to commandFinished() is run *each time* a command finishes, right? And not only on the last Get(). Either that or I am completely wrong about the whole QFtp thing, which wouldn't surprise me either :(

So, I am closing each file, and I am doing it in the same way that example (and the rest of examples on the net) does (or do). Is there something else I need to do to "wait" for Get() to finish? I have no idea how to do that, if there's something to do at all. No example does that. They simple call get, and wait for ftp->currentCommand() == QFtp::Get in the slot tied to the signal commandFinished(). Is there anything else that needs to be done in the while?

Secondly, if your assumption was right, the last file would not be empty, which it is.

Thirdly, I am using a QStringList to "queue" the items. When QFtp::List completes, a slot is called, and that slop is supposed to loop through the list, call Get() on each element with a given file name based on the ftp element name. That file which is global is then closed (supposedly) when QFtp::Get() emits commandFinished(), and so some more qDebug() speech shows:



void kooker::my_commandStarted(int) "Launched command 8: '3'"
void kooker::my_stateChanged(int) "FTP STATUS changed: 'QFtp::HostLookup'"
void kooker::my_stateChanged(int) "FTP STATUS changed: 'QFtp::Connecting'"
void kooker::my_stateChanged(int) "FTP STATUS changed: 'QFtp::Connected'"
void kooker::my_commandStarted(int) "Launched command 9: '1'"
void kooker::my_commandStarted(int) "Launched command 10: '4'"
void kooker::my_stateChanged(int) "FTP STATUS changed: 'QFtp::LoggingIn'"
void kooker::my_commandStarted(int) "Launched command 11: '7'"
void kooker::my_commandStarted(int) "Launched command 12: '6'"
void kooker::fetchFtpList() Entering...
void kooker::my_commandStarted(int) "Launched command 13: '8'"
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 0 bytes out of 4266."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 1448 bytes out of 4266."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 2896 bytes out of 4266."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 2920 bytes out of 4266."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 4266 bytes out of 4266."
void kooker::my_commandFinished(int, bool) "closing file associated with command 13, 0 bytes"
void kooker::my_commandStarted(int) "Launched command 14: '8'"
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 0 bytes out of 2342."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 1448 bytes out of 2342."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 2342 bytes out of 2342."
void kooker::my_commandFinished(int, bool) "closing file associated with command 14, 0 bytes"
void kooker::my_commandStarted(int) "Launched command 15: '8'"
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 0 bytes out of 3099."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 1448 bytes out of 3099."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 2896 bytes out of 3099."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 2920 bytes out of 3099."
void kooker::my_dataTransferProgress(qint64, qint64) "Downloaded 3099 bytes out of 3099."
void kooker::my_commandFinished(int, bool) "closing file associated with command 15, 0 bytes"
void kooker::my_commandStarted(int) "Launched command 16: '8'"


Thanks for your suggestions, I will review this a bit more and see if I can find what's wrong.

[stampede]

So, I am closing each file
No, you don't:) You have a "global" file pointer which you overwrite on each loop iteration, so example of control flow could look like this:

1.
[ftp_local_file = 0, loop iteration = 0]
ftp_local_file = new QFile(); // lets call it file_1
issue get command with ftp_local_file = file_1
2.
[ftp_local_file = file_1, loop i = 1]
ftp_local_file = new QFile(); // file_2
issue get command with ftp_local_file = file_2
3. a file has been downloaded, now:
ftp_local_file = file_2, you call close() on it (could be fine ONLY IF ftp_local_file is still file_2 ! so if there is more loop iterations, pointer to file_2 is lost and ftp_local_file points to another file)
4. another file has been downloaded, ftp_local_file is still file_2 (or maybe another value), you call close() on it again
file_1 may be downloaded, but never will be closed

memory leaks from all QFile objects file_i, for i= 1, ..., N-1, where N is your download count.
So it should work ok if you have only 1 download, but i dont know, maybe there are other bugs.

Create a download queue and start next download in the get_finished() slot, will be far more easier to implement than searching here for bugs.

[i92guboj]

I thank you for your effort, but I've been trying to deal with this for a week almost 24/7 and I am giving up. This really doesn't make any sense, more and more taking into account QFtp is going into oblivion.

I've wasted one more hour trying to follow your suggestion and obviously QFtp is too complex for me. Crazy, considering I use libcurl without problems all the time. I'll be using libcurl once more since it just works. Obviously some defect in my mental model prevents me from being able to use QFtp.

A last question. I am too tired to deal with this right now, but, in the case I want to try this in the future I will be using the newer QNetworkAccessManager stuff, which is supposedly the replacement for all this stuff in QT5. Do you know if there's a way to list remote files using that?

Again, thanks for the dedication and the advice. I am just too tired to deal with this right now.

[stampede]

I can perfectly understand you :D
Maybe starting from scratch could be refreshing, try to define a data structure representing a single download, when looping through a list of all files to download, dont issue get() requests yet, but first put them all into a list. Next, call a doJob() function :


// pseudocode..

void nextDownload(){
if (!this->_listOfDownloads.isEmpty()){
this->_currentDownload = _listOfDownloads.takeFirst();
this->initializeDownload(_currentDownload); // this should start ftp->get() , connect signals etc
...
}
}

void downloadFinished(){
if (this->_currentDownload) {
_currentDownload->file->close();
delete _currentDownload;
}
this->nextDownload();
}

//...

just an idea of how you can organize this in a different way, sometimes it really helps to start with a fresh perspective.


Do you know if there's a way to list remote files using that?
Unfortunately, I don't know. But downloading files from ftp works ok with QNAM :) I remember searching for a way to issue a "list" command via QNAM in Qt4 ~1 year ago and I couldn't find a solution. I don't know, maybe something changed in Qt5.

[i92guboj]

Thank you, I might try, but not today.

As for QNAM, well, if it can't even list files, it's useless for me. You know, fetching a file without knowing its name is only slightly less difficult than rewriting gcc using qBasic.

I have another question though, in case you or anyone else can answer. I've use QEventLoop in another tool to force sync here, and it worked ok. However, this time it doesn't work. The snippet would be something in the lines of:



QEventLoop loop;
connect(ftp, SIGNAL(commandFinished(int,bool)), &loop, SLOT(quit()));

QString item;
QFile *file;
foreach(item, *ftp_file_list)
{
QString img_local_path = QString(photoDir->path().append("/%1").arg(item));
file = new QFile(img_local_path);

if(!file->open(QIODevice::WriteOnly))
{
delete file;
return;
}

QString ftp_file = QString("/%1").arg(item);
ftp->get(ftp_file, file);

loop.exec();
file->close();
}


But this never goes beyond the first iteration, because it gets stuck in the loop. The qtcreator debugging tools show that I enter the loop, and I can proceed until the loop instruction. After that, nothing else (QFTp-related) happens. It just sits there forever.

But, anyway...

Added after 1 30 minutes:

This seems to work.



void kooker::fetchFtpList()
{
qDebug() << Q_FUNC_INFO << "Entering...";
if(!ftp_file_list->isEmpty())
{
QString item = ftp_file_list->takeFirst();
QString img_local_path = QString(photoDir->path().append("/%1").arg(item));
ftp_local_file = new QFile(img_local_path);

if(!ftp_local_file->open(QIODevice::WriteOnly))
{
delete ftp_local_file;
return;
}

QString ftp_file = QString("/%1").arg(item);
ftp->get(ftp_file, ftp_local_file);
}
//}
}

void kooker::my_commandFinished(int command, bool error)
{
if(error)
{
qDebug() << QString("FTP command %1 returned: '%2'")
.arg(command)
.arg(ftp->errorString());
}
else
{
if(ftp->currentCommand() == QFtp::List)
fetchFtpList();

if(ftp->currentCommand() == QFtp::Get)
{
qDebug() << Q_FUNC_INFO << QString("closing file associated with command %1, %2 bytes")
.arg(command)
.arg(ftp_local_file->size());
ftp_local_file->close();
fetchFtpList();
}
}
}