How to check the file to download from ftp site exists ?

[nikhilqt]

Hi,

I need to check whether the file to download from the given ftp site exists. Is there any method where I can use to check whether the file exists before initiating the download using QFTP:get(...) (http://developer.qt.nokia.com/doc/qt-4.8/qftp.html#get).

Thanks in advance.

[wysota]

Sure, look here: How to check if FTP file exists (lmgtfy.com?q=ftp check if file exists)

[nikhilqt]

Thanks for the immediate response. It means I need to use non-Qt APIs to check for the file existence? Are theren't any Qt APIs to check for the file existence.

Thanks for your help.

[AlexSudnik]

int QFtp::list ( const QString & dir = QString() )

Lists the contents of directory dir on the FTP server. If dir is empty, it lists the contents of the current directory.

The listInfo() signal is emitted for each directory entry found.


void QFtp::listInfo ( const QUrlInfo & i ) [signal]

This signal is emitted for each directory entry the list() command finds. The details of the entry are stored in i.

and the last one ^^:

QString QUrlInfo::name () const

Returns the file name of the URL.

[nikhilqt]

I am actually using this method for retrieving the list. But this is kind of work around to check whether file exists. Because every time I need to do QFtp::cd(...) and get the list and then check whether the file exists. I was actually looking for any straight forward APIs where I need to login with my credentials to ftp site and check whether the file I need exists or not. As I was searching some of the other languages support it. So I was curious whether Qt has it. Thanks for your help.

Please let me know if there is any direct APIs whether in Qt or C++. Thanks.

[ChrisW67]

It means I need to use non-Qt APIs to check for the file existence?
No, it doesn't mean that. It means that the question is asked routinely all across the 'net, using various libraries and languages, and the approach to the problem is generally the same. Either:

Try to download the file, if it works the file exists...
Issue an FTP "ls filename" command and see if the file name is present in the output.


The QFtp class has a function to do both of those things.
Only the first option is available using QNetworkAccessManager AFAICT, but you could abort() the download as soon as you see the metadataChanged() signal.

[wysota]

Only the first option is available using QNetworkAccessManager AFAICT, but you could abort() the download as soon as you see the metadataChanged() signal.

There is QFtp::list() available (since OP is using QFtp anyway).

[ChrisW67]

There you go pointing the OP straight at it ;) ... I was trying to be more subtle by saying that QFtp had functions for both and that some RTFM action might be needed.

[wysota]

Ouch, sorry.. didn't read your post carefully enough... Still one has to read the docs to learn how to process the response QFtp receives.

[nikhilqt]

There you go pointing the OP straight at it ;) ... I was trying to be more subtle by saying that QFtp had functions for both and that some RTFM action might be needed.

Could you please expand OP and RTFM and please let me know how to interpret it. Thanks.

Added after 5 minutes:

While fetching the data I am using QFtp::get and if the operation returned 'error' true for QFtp::CommandFinished(int, bool error) then the rest of QFtp commands are not executing. Since all the ftp commands are asynchronous, if one of the command fails does the remaining rest following them aborts ?, because my program does not do anything if any one of the ftp command returns 'error'.

[wysota]

Could you please expand OP and RTFM and please let me know how to interpret it.
OP is "original poster" (meaning you). RTFM is... well... STFW for it :)

The Jargon File (http://catb.org/jargon/html/index.html)

[ChrisW67]

While fetching the data I am using QFtp::get and if the operation returned 'error' true for QFtp::CommandFinished(int, bool error) then the rest of QFtp commands are not executing. Since all the ftp commands are asynchronous, if one of the command fails does the remaining rest following them aborts ?, because my program does not do anything if any one of the ftp command returns 'error'.

From the friendly manual for QFtp (you have read it, haven't you? ;) )...


If an error occurs during the execution of one of the commands in a sequence of commands, all the pending commands (i.e. scheduled, but not yet executed commands) are cleared and no signals are emitted for them.

[nikhilqt]

From the friendly manual for QFtp (you have read it, haven't you? ;) )...

Oh! it is there in the documentation :D. I didn't see it hard enough. So now it creates a issue for me for even 'ls filename' command, because the remaining QFtp::get(...) (http://developer.qt.nokia.com/doc/qt-4.8/qftp.html#get) requests would be lost since the former command fails if the file is not present.

[ChrisW67]

Why? If the list command fails there is no file to download. If you a queuing several file list/get commands at once, don't.

I expect the list command will not fail if the file does not exist: it should just return no directory entries. (I have not verified that)

[nikhilqt]

I expect the list command will not fail if the file does not exist: it should just return no directory entries. (I have not verified that)

But my requirement is to check whether that specific file exists or not in that folder. I cannot achieve it using list command. So as you said I used 'ls filename'.


Why? If the list command fails there is no file to download. If you a queuing several file list/get commands at once, don't.
There are some 40-50 files I need to download and this number might increase as well. So I queue up the request of all the raw command('ls filename') for all the files.

My approach was to check the error status of all the raw commands and then selectively call the QFtp::get method only for those files which exists. But here the checking queue is itself failing for me. :confused:

[wysota]

But my requirement is to check whether that specific file exists or not in that folder. I cannot achieve it using list command.
Why not?



There are some 40-50 files I need to download and this number might increase as well. So I queue up the request of all the raw command('ls filename') for all the files.
Don't do that then. Store the file data elsewhere and push commands to the ftp client as you begin processing a particular entry.

[nikhilqt]

Thanks. I made it work. In the process, I observed one more situation, where the QFtp::Get command will be running and there won't be write to the file in the local file system. This happens quite a sometime after the download of some 20-30 files. Is this because ftp has timeout and the state is been disconnected ?

If I close the application and start it again, then the downloading starts normally. Have anybody experienced this behavior ?

[nikhilqt]

Guys, Any help regarding this. Is there any timeout for the QFtp::Get command. Or do i need to close the ftp connection after sometime and need to re-initiate it ? Please share me your thoughts on it. Thanks

[wysota]

I think you are adressing the question in the wrong place. If you want to learn how FTP servers work, then learn about FTP. Qt has nothing to do with this. Your "issue" is stricly FTP related from the very beginning.

[nikhilqt]

I think you are adressing the question in the wrong place. If you want to learn how FTP servers work, then learn about FTP. Qt has nothing to do with this. Your "issue" is stricly FTP related from the very beginning.

I checked FTP is working fine. The connection is up and running. I figured out the situation when it stops downloading. I have QTimer setup, which is running in a different thread, when it fires a 'timeout' signal(While QFtp::Get is in progress) the download stops.

Is this because of the thread affinity. Do I need to call moveToThread(..) to initiate the main thread event loop ? Please comment.

Thanks.

[wysota]

I checked FTP is working fine.
What do you consider "fine" when speaking about a protocol? If you know how ftp dictates the servers to manage the connection state, why are you asking those questions? The protocol clearly states how to abort a running command and what the state of the connection is afterwards.

[davethomaspilot]

Read this thread and didn't see anything useful. The problem seems non-trivial, yet the responses suggest it's only difficult if you don't understand ftp. I disagree!

My situation is similar, except I want to do an ftp of a file and create a directory structure on the remote host based on a part of the local file's path. Which means I need to ftp->mkdir to create directories as needed before doing the ftp->put().

For example, I have a file named /la/lb/d/e/f/file.jpg. I want it to end up on the ftp server in something like /ftptop/d/e/f/file.jpg

So, there needs to be a sequence of ftp->mkdir, ftp->cd commands to create the directory structure on the remote host before doing the final ftp->put.

Something like this doesn't work:




QStringList dirs2Make = ftpDir.split("/");
qDebug()<< dirs2Make;
foreach (QString d,dirs2Make)
{
qDebug() << "making directory " << d;
ftp->mkdir(d);
ftp->cd(d);
}
ftp->put ftpFile;


Because if directories already exist in the remote path, the sequence will terminate. So, if a directory exists, cd into it. If not mkdir first, then cd into. Sounds simple, but it's not that easy.

The QFtp commands are asynchronous. Which means you have to schedule a sequence and wait for it to complete or for a command in the sequence to fail. Then create another sequence based on if and how the prior scheduled sequence completed.

Figuring out which command in a sequence failed requires keeping an association of a sequence number to the each of the commands in the sequence.

So, unless I'm missing something, there's more too it than just understanding how ftp works. I'm thinking it would be easier to just shell out to a script that does the ftp commands--that way the ftp commands can more easily be dynamically created based on the pass/fail result of a prior command.

Basically, what I'd need is a way to do everything but the actual put synchronously. Commands like login, cd, mkdir typically complete quickly and often alter what should happen next based on whether they pass or fail.

[wysota]

So don't schedule a sequence of commands but rather schedule the next one only after the previous one completes. What exactly is the problem?

[davethomaspilot]

The problem is the relative complexity of the code.

What would you suggest the code that must be in the commandFinished slot? There's no userdata * that can be passed to the slot. All you get is a sequence number and an error code. So, I guess the slot has to have code to code that depends on external class variables to figure out what to do on fail.

Just an unnecessarily complex implementation, IMHO. Versus a simple return code from a synchronous command. Even a simple flag to NOT abort the sequence on command failure would simplify things for me signficiantly. Then, the equivalent of mkdir -p in ftp would just become ftp->mkdir mydir; ftp->cd mydir.

I really don't care if the mkdir fails if it's because it already exists. And if the ftp->mdkir fails for some other unknown reason, I don't how I'd handle the error versus handling the error in the subsequent ftp->cd.

Thanks for the reply,

Dave Thomas

Added after 4 minutes:

Maybe I don't have the best practice for making an asynchronous command synchronous.

Say my application can do nothing useful except keep the GUI responsive until a command complets (or set of commands in this specific example). Should I enter a spin loop with a QApplication::processEvents looking for a flag that gets changed in a slot when a particular signal gets emitted? What controls how much cpu resource gets spent spinning in the event loop versus getting the commands done (when they are on the same thread).

[anda_skoa]

First, a common pattern to wrap asychronous operations together with their context is the job or command pattern.
Each command object has its context, can be started and signals success/failure.
A sequence command can then get a list of smaller commands and start one after the previous one has succeeded.
Things like "check if dir exists, create if not" can also be combined in a composite command which can then be used in the sequence.

See also this blog http://cmollekopf.wordpress.com/2014/10/03/putting-the-code-where-it-belongs/

Second, any event loop based asynchronous operation in Qt can, with some caution, behave synchronous by using a local event loop instance, e.g. using a helper function


void runCommand(QFtp *ftp, int commandId)
{
QEventLoop loop;
connect(ftp, SIGNAL(commandFinished(int, bool)), &loop, SLOT(quit()));
loop.exec();
}



runCommand(ftp, ftp->mkdir(d));


Alternatively use something like a QProgressDialog to provide both the local event loop and keeping the user from interacting with the rest of the application.

Cheers,
_

[davethomaspilot]

From the URL you provided (and thanks for that):


What you’ll notice immediately that this involves a lot of boilerplate code. It also introduces a lot of complexity in a seemingly trivial task.....So while KJob gives us a tool to wrap asynchronous operations in a way that they become reusable, it comes at the cost of quite a bit of boilerplate code. It also means that what can be written synchronously in a simple function, requires a class when writing the same code asynchronously

That's really my point--complexity versus a simple synchronous call. So, that's why I'm thinking use of the QFtp class for my purposes isn't a good idea versus doing it externally in a script where synchronous ftp commands ARE available.

The idiom you posted for making a synchronous command is better than what I'm doing--at least the spin loop isn't visible in my code ;-). Thanks!

But, I'm assuming the overhead to loop and process events in the exec is small enough not to significantly impact execution of the commands themselves? That's the objection I have about using an asynchronous command when a synchronous operation is desired.

And, there's considerably more to the implementation than understanding ftp commands.

Thanks,

Dave Thomas

[wysota]

You don't need KJob and you don't need any boilerplate code. KJob aims to be a general purpose mechanism, your case is very specific so you only need code tailored to your exact situation. You can have a queue of commands to be executed and each command can be in form of if-then, e.g. if directory X doesn't exist, create directory X. Then you schedule a command from the tip of the queue to ftp and when the command is finished, you check the condition, pop the command off the queue and repeat the whole process with the new tip. You only need the data structure for the queue and a slot connected to a signal emitted when ftp command finishes execution.

[davethomaspilot]

this thread has had LOTS of reads--I think it is a very common problem.


situation. You can have a queue of commands to be executed and each command can be in form of if-then, e.g. if directory X doesn't exist, create directory X.

Well, yes at at high enough abstraction level.

I know this is obvious to you but there's a bit of code behind the "if-then stuff". Declare signals and slots. Connect signals and slots , Schedule the command. Go into an event loop. Execute code in slot that gets called when command finishes. Set something in the slot that let's the code that scheduled the ftp command know if it's passed or failed.

Versus checking a return code of a syncrhonous command in an your favorite scripting language and calling with something like QProcess.

I come out thinking the script is much simpler.

[wysota]

this thread has had LOTS of reads--I think it is a very common problem.
Or someone has set a periodic download of this exact page :)


I know this is obvious to you but there's a bit of code behind the "if-then stuff". Declare signals and slots. Connect signals and slots , Schedule the command. Go into an event loop. Execute code in slot that gets called when command finishes. Set something in the slot that let's the code that scheduled the ftp command know if it's passed or failed.
No, not really, you are overcomplicating things. You need one slot connected to QFtp::commandFinished() and in that slot pass the result of the ftp command to the job on the front of the queue, let it decide whether to execute the "then" command or not, pop it from the queue, execute the command from the current tip of the queue, rince and repeat. No need for any signal declarations, signal connecting or disconnecting, no need for any extra event loops.

[anda_skoa]

That's really my point--complexity versus a simple synchronous call.

Sure, these are just the side effects of different use cases.
Qt applications are usually faced with the tasked of doing more than one thing at the same time, e.g. having a UI and doing network communication, or doing timeout guarded network communication, etc.

So Qt's classes for potentially long running operations are designed to be asynchronous to make that possible.

So it seems your program is a non-GUI/console tool that does not need any of that and you might indeed be better of by using a script instead.

Cheers,
_