Issue with download from url

[ejoshva]

nreply->readAll()

nreply->readAll()

To copy to clipboard, switch view to plain text mode 

This line is throwing an error as below only for windows machine. The same code works fine for mac.
Invalid parameter passed to C runtime function.
Invalid parameter passed to C runtime function.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc


The download code is as below

zoushoFtpTransfer :: zoushoFtpTransfer()
{
	m_manager = new QNetworkAccessManager();
	m_request = new QNetworkRequest;
	m_reply = NULL;
 
	m_isDownloadSccess = false;
    connect(m_manager, SIGNAL(sslErrors(QNetworkReply*,QList<QSslError>)),
            this, SLOT(handleSslErrors(QNetworkReply*,QList<QSslError>)));
    connect(m_manager,SIGNAL(finished(QNetworkReply*)),this,SLOT(replyfinished(QNetworkReply*)));
    m_htmlDownloadCount = 0 ;
}
void zoushoFtpTransfer :: downloadFromServer()
{
	QNetworkRequest request;
	QUrl url;
	url.setUrl(m_url);
	url.setUserName(m_username);
	url.setPassword(m_password);
	request.setUrl(url);
	zoushoConstructFilePath();
	zoushoConstructDownloadDir();
	zoushoConstructDownloadPath();
	m_reply =  m_manager->get(request);
}
 
void zoushoFtpTransfer :: replyfinished(QNetworkReply *nreply)
{
 
	if(nreply->error() )
	{
		qDebug() << "download failed...." <<nreply->errorString();
        m_isDownloadFailure = true;
	}
	else
     {
        //m_url = "http://www.ssparkl.com/download.php?filename=fullbook/LifeSciences_Grade10_1947.zip";
 
        if(m_url.endsWith(".zip"))
        {
           QFile file(m_localPath);
           file.open(QIODevice::WriteOnly);
           file.write(nreply->readAll());
           file.close();
           m_isDownloadSccess = true;
           return;
        }
}

zoushoFtpTransfer :: zoushoFtpTransfer()
{
	m_manager = new QNetworkAccessManager();
	m_request = new QNetworkRequest;
	m_reply = NULL;

	m_isDownloadSccess = false;
    connect(m_manager, SIGNAL(sslErrors(QNetworkReply*,QList<QSslError>)),
            this, SLOT(handleSslErrors(QNetworkReply*,QList<QSslError>)));
    connect(m_manager,SIGNAL(finished(QNetworkReply*)),this,SLOT(replyfinished(QNetworkReply*)));
    m_htmlDownloadCount = 0 ;
}
void zoushoFtpTransfer :: downloadFromServer()
{
	QNetworkRequest request;
	QUrl url;
	url.setUrl(m_url);
	url.setUserName(m_username);
	url.setPassword(m_password);
	request.setUrl(url);
	zoushoConstructFilePath();
	zoushoConstructDownloadDir();
	zoushoConstructDownloadPath();
	m_reply =  m_manager->get(request);
}

void zoushoFtpTransfer :: replyfinished(QNetworkReply *nreply)
{

	if(nreply->error() )
	{
		qDebug() << "download failed...." <<nreply->errorString();
        m_isDownloadFailure = true;
	}
	else
     {
        //m_url = "http://www.ssparkl.com/download.php?filename=fullbook/LifeSciences_Grade10_1947.zip";
   
        if(m_url.endsWith(".zip"))
        {
           QFile file(m_localPath);
           file.open(QIODevice::WriteOnly);
           file.write(nreply->readAll());
           file.close();
           m_isDownloadSccess = true;
           return;
        }
}

To copy to clipboard, switch view to plain text mode 

The download file size is 318459022.
I checked the received data using the below code and it final received size is 318459022.

connect(m_WebCtrl.get(request), SIGNAL(downloadProgress(qint64,qint64)), this, SLOT(downloadprogress(qint64,qint64)));

connect(m_WebCtrl.get(request), SIGNAL(downloadProgress(qint64,qint64)), this, SLOT(downloadprogress(qint64,qint64)));

To copy to clipboard, switch view to plain text mode 

Book with size of 20MB are getting downloaded without issues.

Kindly help me to know, where I am going wrong

[wysota]

You are trying to download a 3GB file and then make a copy of it using readAll(). How much RAM do you have in your machine? Is it at least 8GB? Does your Windows box allow you to allocate that much in a single process?

[jefftee]

My guess is you're running out of memory trying to read the entire response at once with QNetworkReply::readAll. Try using the QNetworkReply::readyRead() signal to read data from the request and write data to the file as it becomes available rather than waiting until the entire file has been received and trying to read it all at once.

[anda_skoa]

Have you tried writing the data when it arrives instead of waiting for all of it?

Cheers,
_

[ejoshva]

Tried with readyRead() but got the error as QIODevice not open

You are trying to download a 3GB file and then make a copy of it using readAll(). How much RAM do you have in your machine? Is it at least 8GB? Does your Windows box allow you to allocate that much in a single process?

I had a doubt on this perspective. My machine RAM is 4GB.

In that case, I would have write into file as and when it is received

---

Added after 5 minutes:

Have you tried writing the data when it arrives instead of waiting for all of it?

Cheers,
_

How do I do that?

[jefftee]

How do I do that?

With the readyRead() signal...

Show the code when you tried to use readyRead() please.

[ejoshva]

connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));
 
m_reply =  m_manager->get(m_request);
 
void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));

m_reply =  m_manager->get(m_request);

void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

To copy to clipboard, switch view to plain text mode 

[jefftee]

Move your connect statement *after* the get request. Also, where do you open file for writing?

Edit: If you are going to write the file in the way you're attempting, you need to make sure you open the file to append data to the end of the file, or else each write will overlay prior data you've written.

[ejoshva]

Thanks pin pointing the file:pen().
That was the issue, now it's getting written to file.
But only the last received alone is written to the file.

m_reply =  m_manager->get(m_request);
 
connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));
 
 
void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.open(QIODevice::writeOnly);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

m_reply =  m_manager->get(m_request);

connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));
 
 
void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.open(QIODevice::writeOnly);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

To copy to clipboard, switch view to plain text mode 

[wysota]

See the rest of the previous comment.

[ejoshva]

by making use of QNetworkReply::readyRead() the issue is resolved.

Thank for you timely replies and suggestion for the fix

[ejoshva]

Now I am able to download and write all the content using the readyRead()

m_reply =  m_manager->get(m_request);
 
connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));
 
 
void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.open(QIODevice::writeOnly);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

m_reply =  m_manager->get(m_request);
 
connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));
 
 
void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.open(QIODevice::writeOnly);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

To copy to clipboard, switch view to plain text mode 

Again there is one more issue, after writing onto file, I am storing all the data for that file in QByteArray.
Its storing in QBytearray till 134459022 of data, after which bad_alloc error is thrown.

Let me know what data type can be used to store all the 318459022 size of data

[anda_skoa]

Let me know what data type can be used to store all the 318459022 size of data

It is not a matter of data type. QByteArray can easily handle that, your system can't.

Cheers,
_

[yeye_olive]

This last snippet of code you posted is exactly the same as the previous one, and suffers from exactly the same problems that were pointed to you. Since you claim to have now solved these problems, I suppose that this code is irrelevant to your new question.

If you have understood that you could not keep the whole file in memory before writing it to disk in one go, it should come as no surprise that attempting to completely load it in memory fails for the same reason.

[wysota]

Now I am able to download and write all the content using the readyRead()

m_reply =  m_manager->get(m_request);
 
connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));
 
 
void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.open(QIODevice::writeOnly);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

m_reply =  m_manager->get(m_request);
 
connect(m_reply,SIGNAL(readyRead()),this,SLOT(readingReadyBytes()));
 
 
void zoushoFtpTransfer::readingReadyBytes()
{
    QFile file(m_localPath);
    file.open(QIODevice::writeOnly);
    file.write(m_reply->read(m_reply->bytesAvailable()));
    file.close();
    qDebug()<<"size : "<<file.size();
    m_isDownloadSccess = true;
}

To copy to clipboard, switch view to plain text mode 

Again there is one more issue, after writing onto file, I am storing all the data for that file in QByteArray.
Its storing in QBytearray till 134459022 of data, after which bad_alloc error is thrown.

Let me know what data type can be used to store all the 318459022 size of data

I'll give you a hint:

7 = 4+3 // to read 7MB, read 4MB and then 3
10 = 4+4+2 // to read 10MB read 4MB, then again 4MB and then 2 MB
445 = 4+...+4+1 // to read 445MB read 4MB as long as there is more than 4MB to read and then read whatever remains

And another hint:

If you tell read() to read 4MB but there is only 1MB available, it will read 1MB

Question:

How much memory is needed to read 445MB of data this way?

Now go and carve an algorithm out of it.

[ejoshva]

Thanks for your replies.

@wysota - will try out an algorithm based on your hints

I'll give you a hint:

7 = 4+3 // to read 7MB, read 4MB and then 3
10 = 4+4+2 // to read 10MB read 4MB, then again 4MB and then 2 MB
445 = 4+...+4+1 // to read 445MB read 4MB as long as there is more than 4MB to read and then read whatever remains

And another hint:

If you tell read() to read 4MB but there is only 1MB available, it will read 1MB

Question:

How much memory is needed to read 445MB of data this way?

Now go and carve an algorithm out of it.

ans : 4 MB