Question about ftp using Qt

[lni]

Hi,

I need to ftp 30000 files from a particular website with same username and password, I am using the following code.

My question: will the ftp login 30000 times, or will it keep the connection alive during the entire ftp session until all files are ftped? Also how to I set timeout?

Thanks

QUrl url;
  url.setUserName( "ftp" );
  url.setPassword( "xxx" );
 
  QNetworkAccessManager manager;
 
  QEventLoop eventLoop;
 
  connect( &manager, SIGNAL( finished( QNetworkReply* ) ),
               &eventLoop, SLOT( quit() ) );
 
 
  foreach ( const QString& file, fileList ) {
 
    QUrl.setUurl( "ftp://ftp.xxx.com/" + file );
    QNetworkReply *reply = manager.get( QNetworkRequest( url ) );
 
    /// block until each file is finished, how do I set timeout in the next line?
    eventLoop.exec();
 
    /// get the reply content
    reply->readAll();
    ....
 
    reply->deleteLater();
 
  }

QUrl url;
  url.setUserName( "ftp" );
  url.setPassword( "xxx" );

  QNetworkAccessManager manager;

  QEventLoop eventLoop;

  connect( &manager, SIGNAL( finished( QNetworkReply* ) ),
               &eventLoop, SLOT( quit() ) );


  foreach ( const QString& file, fileList ) {

    QUrl.setUurl( "ftp://ftp.xxx.com/" + file );
    QNetworkReply *reply = manager.get( QNetworkRequest( url ) );

    /// block until each file is finished, how do I set timeout in the next line?
    eventLoop.exec();
    
    /// get the reply content
    reply->readAll();
    ....

    reply->deleteLater();

  }

To copy to clipboard, switch view to plain text mode 

[ChrisW67]

QNetworkAccessManager will reuse connections for HTTP but I have never investigated FTP. You can check with Wireshark fairly quickly. In your example (clearly not your real code) you are not sending credentials so there might only be an anonymous FTP login.

If you were not explicitly serialising your requests QNetworkAccessManager will also process several at the same time. It would be a good idea to check if the transfer was successful before you rely on the data.

Set the timeout for what?

[lni]

Thanks!

Set timeout for connection or maximum duration of the ftp job. I have managed to use QTimer for that.

However, I now have another problem.

I connect to QNetworkAccessManager::finished signal, and then read data using reply->readAll(). Problem is reply sometimes appends new line "\n" in between data chunk, causing the xml file to break.

For example, tag <conversion> becomes:

<convers
ion>


This cause xml parser to fail. The xml file itself has many new lines, so I don't to use QString::replace( "\n", "" ), because this will cause entire xml file to be a single line.

How do I fix this problem?

Thanks

[ChrisW67]

I connect to QNetworkAccessManager::finished signal, and then read data using reply->readAll(). Problem is reply sometimes appends new line "\n" in between data chunk, causing the xml file to break.

No, Qt doesn't. That raises the question, what are you actually doing to the QByteArray after readAll() gives it to you?

[lni]

No, Qt doesn't. That raises the question, what are you actually doing to the QByteArray after readAll() gives it to you?

I did:

QString content = reply->readAll();

Then parse the content using xml parser, and found that "\n" is inserted in between data in some of the files.

[ChrisW67]

Is the FTP server public? Example file?

[anda_skoa]

QString content = reply->readAll();

That does an uncontrolled 8-bit encoding to 16-bit unicode conversion, usually not something you want to do when processing XML. The XML document usually has a processing instruction for the correct encoding and the XML parsers react to that.

Which XML parser are you using?

Cheers,
_

[lni]

I am now pretty sure QNetworkReply append "\n" in between chunk of data (not all files though).

I switch to QByteArray, the same thing happens.


It complains:

ERROR: code = 3 , what = "Expected '>', but got '[a-zA-Z]'."

Here is part of the code:

QByteArray content =  reply->readAll();
 
QFile file( filename );
file.open( QIODevice::WriteOnly );
file.write( content );
 
// parse the file using QXmlStreamReader
QXmlStreamReader reader( content );
...

QByteArray content =  reply->readAll();

QFile file( filename );
file.open( QIODevice::WriteOnly );
file.write( content );

// parse the file using QXmlStreamReader
QXmlStreamReader reader( content );
...

To copy to clipboard, switch view to plain text mode 

I look at the file, some of the xml tag is broken into 2 lines.

There are only English characters in the content. No other language.


And I hit another problem: I submit a job to ftp 300 files, one of the files doesn't exist, it emits error code 203 (ContentNotFoundError). This is fine. But immediately, all remain files got 201 error code (ContentAccessDenied).

If I remove the bad files, the ftp runs without problem. It get the files. As soon as I put back the non-existing file, 203 is emitted, and immediately the rest files got 201, and job stops.

I am using Qt 4.8.1 under CentOS.

PS. I Just google search, and found someone is having the same problem: http://www.qtforum.org/article/33629...essdenied.html

[anda_skoa]

Very strange.

You could alternatively try to use QFtp.

Cheers,
_

[wysota]

And I hit another problem: I submit a job to ftp 300 files, one of the files doesn't exist, it emits error code 203 (ContentNotFoundError). This is fine. But immediately, all remain files got 201 error code (ContentAccessDenied).

If I remove the bad files, the ftp runs without problem. It get the files. As soon as I put back the non-existing file, 203 is emitted, and immediately the rest files got 201, and job stops.

That may be the proper behavior of the ftp server you are connecting to.

[ChrisW67]

I am now pretty sure QNetworkReply append "\n" in between chunk of data (not all files though).

Until you demonstrate that the server is not sending these newline characters I am going to assume you are mistaken. If this was a normal behaviour of QNetworkReply on FTP links this would not bode well for lots of Qt FTP usage and the bug tracker would be running hot.

I just tested fetching a few dozen XML files without a byte out of place (compared byte for byte with version gathered by rsync):

#include <QtCore>
#include <QtNetwork>
 
int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);
 
    const QStringList fileList = QStringList()
        << "assistant/metadata.xml"
        << "qtsvg/metadata.xml"
        << "qtopenvg/metadata.xml"
        << "qtphonon/metadata.xml"
        << "qt3support/metadata.xml";  // ... plenty more like these
 
    QNetworkAccessManager manager;
 
    QEventLoop eventLoop;
 
    QObject::connect(&manager, SIGNAL(finished(QNetworkReply*)), &eventLoop, SLOT(quit()));
 
    foreach (const QString& file, fileList ) {
 
        QUrl url("ftp://mirror.internode.on.net/pub/gentoo-portage/dev-qt/" + file);
        url.setUserName( "anonymous" );
        url.setPassword( "anon@example.com" );
        QNetworkReply *reply = manager.get( QNetworkRequest( url ) );
 
        /// block until each file is finished, how do I set timeout in the next line?
        eventLoop.exec();
 
        /// get the reply content
        qDebug() << file << "Error" << reply->error();
        QByteArray data = reply->readAll();
        QString outFileName(file);
        QFile out(outFileName.replace("/", "_"));
        if (out.open(QIODevice::WriteOnly)) {
            out.write(data);
            out.close();
        }
 
        reply->deleteLater();
    }
    return 0;
}

#include <QtCore>
#include <QtNetwork>

int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);

    const QStringList fileList = QStringList()
        << "assistant/metadata.xml"
        << "qtsvg/metadata.xml"
        << "qtopenvg/metadata.xml"
        << "qtphonon/metadata.xml"
        << "qt3support/metadata.xml";  // ... plenty more like these

    QNetworkAccessManager manager;

    QEventLoop eventLoop;

    QObject::connect(&manager, SIGNAL(finished(QNetworkReply*)), &eventLoop, SLOT(quit()));

    foreach (const QString& file, fileList ) {

        QUrl url("ftp://mirror.internode.on.net/pub/gentoo-portage/dev-qt/" + file);
        url.setUserName( "anonymous" );
        url.setPassword( "anon@example.com" );
        QNetworkReply *reply = manager.get( QNetworkRequest( url ) );

        /// block until each file is finished, how do I set timeout in the next line?
        eventLoop.exec();

        /// get the reply content
        qDebug() << file << "Error" << reply->error();
        QByteArray data = reply->readAll();
        QString outFileName(file);
        QFile out(outFileName.replace("/", "_"));
        if (out.open(QIODevice::WriteOnly)) {
            out.write(data);
            out.close();
        }

        reply->deleteLater();
    }
    return 0;
}

To copy to clipboard, switch view to plain text mode 

For the record:

• QNetworkAccessManager only executed a single FTP login (USER/PASS).
• If a file did not exist I got the same behaviour as you did on subsequent files. It seems that QNetworkAccessManager closed the TCP connection on the FTP 550 response and did not try to reopen it for the later ones. This might be a bug.

[lni]

The file size I try to ftp is around 10,000 byte counts, if this information is helpful. Your sample xml file size perhaps is too small (my random thought).

I didn't block each file, instead I submit all ftp jobs, and then wait until all finish or timeout.

However, it is more serious for 201 error caused by 203 error. I switch to Qt 5.2, same problem exists.

#include <QtCore>
#include <QtNetwork>
#include <QDebug>
 
int main(int argc, char **argv)
{
  QCoreApplication app(argc, argv);
 
  QNetworkAccessManager manager;
 
  QEventLoop eventLoop;
 
  QObject::connect(&manager, SIGNAL(finished(QNetworkReply*)), &eventLoop, SLOT(quit()));
 
  QList<QNetworkReply*> replies;
 
  int loops = 200;
  for ( int idx = 0; idx < loops; idx++ ) {
 
    /// let's ftp same file 200 times and compare them
    /// (But I just got first file, why?)
    QUrl url( "ftp://mirror.internode.on.net/pub/gentoo-portage/dev-qt/assistant/metadata.xml" );
    url.setUserName( "anonymous" );
    url.setPassword( "anon@example.com" );
    replies << manager.get( QNetworkRequest( url ) );
 
  }
 
  /// you need to set time out before exec is call
  /// allow 60 seconds timeout for each ftp
  QTimer::singleShot( loops * 60 * 1000, &eventLoop, SLOT( quit() ) );
 
  /// block until all files area finished
  eventLoop.exec();
 
  qDebug() << "done";
 
  QByteArray ba0 = replies.at( 0 )->readAll();
  qDebug() << ba0;
 
  for ( int idx = 1; idx < loops; idx++ ) {
    QByteArray ba = replies.at( idx )->readAll();
    if ( ba0.size() != ba.size() ) {
      qDebug() << idx << ": Size not equal";
      qDebug() << ba;
      break;
    }
  }
 
 
  return 0;
}

#include <QtCore>
#include <QtNetwork>
#include <QDebug>

int main(int argc, char **argv)
{
  QCoreApplication app(argc, argv);

  QNetworkAccessManager manager;

  QEventLoop eventLoop;

  QObject::connect(&manager, SIGNAL(finished(QNetworkReply*)), &eventLoop, SLOT(quit()));

  QList<QNetworkReply*> replies;

  int loops = 200;
  for ( int idx = 0; idx < loops; idx++ ) {

    /// let's ftp same file 200 times and compare them
    /// (But I just got first file, why?)
    QUrl url( "ftp://mirror.internode.on.net/pub/gentoo-portage/dev-qt/assistant/metadata.xml" );
    url.setUserName( "anonymous" );
    url.setPassword( "anon@example.com" );
    replies << manager.get( QNetworkRequest( url ) );

  }

  /// you need to set time out before exec is call
  /// allow 60 seconds timeout for each ftp
  QTimer::singleShot( loops * 60 * 1000, &eventLoop, SLOT( quit() ) );

  /// block until all files area finished
  eventLoop.exec();

  qDebug() << "done";

  QByteArray ba0 = replies.at( 0 )->readAll();
  qDebug() << ba0;

  for ( int idx = 1; idx < loops; idx++ ) {
    QByteArray ba = replies.at( idx )->readAll();
    if ( ba0.size() != ba.size() ) {
      qDebug() << idx << ": Size not equal";
      qDebug() << ba;
      break;
    }
  }


  return 0;
}

To copy to clipboard, switch view to plain text mode