"Too many open files" error

[jiveaxe]

Hi, I got the error in the title running a class derived from QNetworkAccessManager to download and save files. It works fine except when requesting a large number of file (600~). Here the implementation (quite common):



FileDownloader::FileDownloader(QObject *parent) :
QNetworkAccessManager(parent)
{
}

FileDownloader::~FileDownloader()
{
deleteLater();
}

void FileDownloader::setPath(const QString &path)
{
mPath = path;
}

void FileDownloader::setFileName(const QString &fileName)
{
mFileName = fileName;
}

void FileDownloader::setUrl(const QString &url)
{
mUrl = url;
}

void FileDownloader::download()
{
file = new QFile;
file->setFileName(mPath + QDir::separator() + mFileName);

if(file->exists())
if(!file->remove()) {
qDebug() << file->errorString();
return;
}

file->open(QIODevice::WriteOnly);

QNetworkRequest request;
request.setUrl(QUrl(mUrl));
reply = get(request);

connect(reply, SIGNAL(downloadProgress(qint64,qint64)), this, SLOT(onDownloadProgress(qint64,qint64)));
connect(this, SIGNAL(finished(QNetworkReply*)), this, SLOT(onFinished(QNetworkReply*)));
connect(reply, SIGNAL(readyRead()), this, SLOT(onReadyRead()));
connect(reply, SIGNAL(finished()), this, SLOT(onReplyFinished()));
}


void FileDownloader::onFinished(QNetworkReply *reply)
{
switch(reply->error()) {
case QNetworkReply::NoError:
qDebug() << "Downloaded:" << file->fileName();
break;

default:
qDebug(reply->errorString().toLatin1());
}

if(file->isOpen()) {
file->close();
file->deleteLater();
}
}

void FileDownloader::onReadyRead()
{
file->write(reply->readAll());
}

void FileDownloader::onReplyFinished()
{
if(file->isOpen()) {
file->close();
file->deleteLater();
}
}

void FileDownloader::onDownloadProgress(qint64, qint64)
{
qDebug(QString::number(bytesRead).toLatin1() + " - " + QString::number(bytesTotal).toLatin1());
}


I'm calling it this way:



foreach(QString file, files) {
FileDownloader * downloader = new FileDownloader(this);
downloader->setUrl(/*...*/);
downloader->setPath(/*...*/);
downloader->setFileName(/*...*/);
downloader->download();
}


Searching the web I read that a solution should be QRunnable, but I'm using it in the wrong way:
* had subclassed QRunnable:

class FileDownloader : public QNetworkAccessManager, public QRunnable
* renamed FileDownloader::download() in FileDownloader::run()
* modified calling routine:


foreach(QString file, files) {
FileDownloader * downloader = new FileDownloader(this);
downloader->setUrl(/*...*/);
downloader->setPath(/*...*/);
downloader->setFileName(/*...*/);
QThreadPool::globalInstance()->start(downloader); // <- new version


In console I got repetition of the following error:


QObject: Cannot create children for a parent that is in a different thread.
(Parent is FileDownloader(0x13297e0), parent's thread is QThread(0x118c420), current thread is QThread(0x13525d0)

Any help will be very appreciated. Thanks.

[ChrisW67]

The "Too many open files" error, which you completely fail to mention in the , is because all operating system have a practical limit to the number of simultaneously open files/sockets that you are exceeding. This is partly a result of resource limits in internal tables etc. and partly a defence mechanism against DOS attacks. You are attempting 600+ simultaneous downloads, which strikes me as a very odd thing. You should serialise the downloads into a few streams.

[jiveaxe]

Solved queuing all downloads in a QQueue and cotrolling the number of simultaneous downloads.