Using QTcpSocket to download image from the web

[Huk]

Hello everybody.

I know there were many topics with similar subjects, but I can't seem to find an answer for my problems so... here we are.

I want to create application, that would be able to download images from the internet. I know I could use QNetworkManager, but I would like to display image as soon as some part of it gets downloaded (like most browsers do), and QNetworkManager seems to display image after download is complete, so I want to try QTcpSocket. As a test, I am trying to download this image:

http://www.google.pl/images/srpr/nav_logo73.png

Here is some code:

QMessageBox MB;
QTcpSocket socket;
connect(&socket,SIGNAL(error(QAbstractSocket::SocketError)),this,SLOT(error(QAbstractSocket::SocketError)));
socket.connectToHost("www.google.pl",80,QIODevice::ReadWrite);
socket.waitForConnected();
 
socket.write("GET /images/srpr/nav_logo73.png");
 
socket.waitForReadyRead();
QByteArray ba;
ba=socket.readAll();
MB.setText(QString(ba));
MB.exec();
 
.
void MainWindow::error(QAbstractSocket::SocketError socketError)
{
    MB.setText("Error! "+QString::number(socketError));
    MB.exec();
}

QMessageBox MB;
QTcpSocket socket;
connect(&socket,SIGNAL(error(QAbstractSocket::SocketError)),this,SLOT(error(QAbstractSocket::SocketError)));
socket.connectToHost("www.google.pl",80,QIODevice::ReadWrite);
socket.waitForConnected();

socket.write("GET /images/srpr/nav_logo73.png");

socket.waitForReadyRead();
QByteArray ba;
ba=socket.readAll();
MB.setText(QString(ba));
MB.exec();

.
void MainWindow::error(QAbstractSocket::SocketError socketError)
{
    MB.setText("Error! "+QString::number(socketError));
    MB.exec();
}

To copy to clipboard, switch view to plain text mode 

Of course the above is meant to display any data that gets to the byteArray, but even this doesn't work, I got "SocketTimeoutError". What am I doing wrong?

Thanks in advance.

[squidge]

Your example is assuming a blocking socket rather than reading data whenever it is available.

Also, QNetworkAccessManager also emits the readyRead() signal whenever new data arrives.

[Huk]

@squidge:

Thanks for the reply, I have modified my application, and now I use signals instead of blocking code:

socket.connectToHost("google.pl",80);
connect(&socket,SIGNAL(error(QAbstractSocket::SocketError)),this,SLOT(error(QAbstractSocket::SocketError)));
connect(&socket,SIGNAL(readyRead()),this,SLOT(read()));
connect(&socket,SIGNAL(connected()),this,SLOT(connectX()));
 
void MainWindow::connectX()
{
    socket.setSocketOption(QAbstractSocket::KeepAliveOption,1);
    QUrl url("http://www.google.pl/images/srpr/nav_logo73.png");
 
    ui->statusBar->showMessage("CONNECTED");
    socket.write("GET "+ url.toEncoded(QUrl::RemoveScheme | QUrl::RemoveAuthority) +"\r\n");
}
 
void MainWindow::read()
{
    ba+=socket.readAll();
}
 
void MainWindow::error(QAbstractSocket::SocketError socketError)
{
    ui->statusBar->showMessage("Error! "+QString::number(socketError));
}

socket.connectToHost("google.pl",80);
connect(&socket,SIGNAL(error(QAbstractSocket::SocketError)),this,SLOT(error(QAbstractSocket::SocketError)));
connect(&socket,SIGNAL(readyRead()),this,SLOT(read()));
connect(&socket,SIGNAL(connected()),this,SLOT(connectX()));

void MainWindow::connectX()
{
    socket.setSocketOption(QAbstractSocket::KeepAliveOption,1);
    QUrl url("http://www.google.pl/images/srpr/nav_logo73.png");

    ui->statusBar->showMessage("CONNECTED");
    socket.write("GET "+ url.toEncoded(QUrl::RemoveScheme | QUrl::RemoveAuthority) +"\r\n");
}

void MainWindow::read()
{
    ba+=socket.readAll();
}

void MainWindow::error(QAbstractSocket::SocketError socketError)
{
    ui->statusBar->showMessage("Error! "+QString::number(socketError));
}

To copy to clipboard, switch view to plain text mode 

however weird thing is that with some links it works OK (like the google link mentioned in the first post), but with others I get "Bad request", also I can do:

socket.write("GET /images/srpr/nav_logo73.png\r\n");

socket.write("GET /images/srpr/nav_logo73.png\r\n");

To copy to clipboard, switch view to plain text mode 

but not HEAD:

socket.write("HEAD /images/srpr/nav_logo73.png\r\n");

socket.write("HEAD /images/srpr/nav_logo73.png\r\n");

To copy to clipboard, switch view to plain text mode 

which gives me "Bad request".

I am not a networking expert (obviously ;] ) but I think the problem is that while browsers and QNetworkAccesManager sends GET, HEAD and other HTTP requests that they are somehow identified (by wireshark for example) as HTTP, and my requests are identified as TCP (wireshark informs me that my requests are "TCP segment of a reassembled PDU", when I look inside I can see my request), please correct me if I am wrong.

How can I fix that?

Also, QNetworkAccessManager also emits the readyRead() signal whenever new data arrives.

But how can I read that data? I don't see any read function for QNetworkAccessManager that could be used here to read partial data, unless cache() can be used in that manner?

Thanks in advance.

EDIT:

OK now I see that I can use something like:

QNetworkAccessManager *manager = new QNetworkAccessManager(this);
QNetworkReply *reply;
 
reply=manager->get(QNetworkRequest(QUrl("http://www.google.pl/images/srpr/nav_logo73.png")));
connect(reply,SIGNAL(readyRead()),this,SLOT(read2()));
 
void MainWindow::read2()
{
    ba+=reply->readAll();
    QPixmap temp;
    temp.loadFromData(ba);
    ui->display->setPixmap(temp);
}

QNetworkAccessManager *manager = new QNetworkAccessManager(this);
QNetworkReply *reply;

reply=manager->get(QNetworkRequest(QUrl("http://www.google.pl/images/srpr/nav_logo73.png")));
connect(reply,SIGNAL(readyRead()),this,SLOT(read2()));

void MainWindow::read2()
{
    ba+=reply->readAll();
    QPixmap temp;
    temp.loadFromData(ba);
    ui->display->setPixmap(temp);
}

To copy to clipboard, switch view to plain text mode 

and it works however is something like the above safe to use?

[squidge]

I don't see any problems with using code like that. The only thing I can think of is if the 'get' function finishes before your connect is executed, but that is unlikely (impossible?) to happen as the event driven architecture will not run until your function completes anyway.

[ChrisW67]

however weird thing is that with some links it works OK (like the google link mentioned in the first post), but with others I get "Bad request", also I can do:

socket.write("GET /images/srpr/nav_logo73.png\r\n");

socket.write("GET /images/srpr/nav_logo73.png\r\n");

To copy to clipboard, switch view to plain text mode 

but not HEAD:

socket.write("HEAD /images/srpr/nav_logo73.png\r\n");

socket.write("HEAD /images/srpr/nav_logo73.png\r\n");

To copy to clipboard, switch view to plain text mode 

which gives me "Bad request".

I am not a networking expert (obviously ;] ) but I think the problem is that while browsers and QNetworkAccesManager sends GET, HEAD and other HTTP requests that they are somehow identified (by wireshark for example) as HTTP, and my requests are identified as TCP (wireshark informs me that my requests are "TCP segment of a reassembled PDU", when I look inside I can see my request), please correct me if I am wrong.

How can I fix that?

A correct GET or HEAD request looks like:

GET /images/srpr/nav_logo73.png HTTP/1.0\r\n
HEAD /images/srpr/nav_logo73.png HTTP/1.1\r\n

GET /images/srpr/nav_logo73.png HTTP/1.0\r\n
HEAD /images/srpr/nav_logo73.png HTTP/1.1\r\n

To copy to clipboard, switch view to plain text mode 

(See http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html)
Some servers will accept malformed requests, some won't. If you are going to reinvent the wheel, try to make it round

[Huk]

@ChrisW67:

I realize that ;] however if I add "HTTP/1.1 or HTTP/1.0 or HTTP/1." suffix I get no data at all, not even an error, so there still must be something wrong, and I have no idea what... however I am still opened to suggestions, although I will use QNetworkAccessManager, I would still like to know why QTcpSocket GET and HEAD doesn't work.

Thanks for help so far.

[squidge]

Maybe you are not terminating the lines properly. The best way to find out is to use something like Wireshark or similar which will show you the exact data being sent and received. You can then compare this to an application which gets the proper response.

[Huk]

@squidge:

Yes, that was the problem ;]

Instead of:

GET /images/srpr/nav_logo73.png HTTP/1.0\r\n

GET /images/srpr/nav_logo73.png HTTP/1.0\r\n

To copy to clipboard, switch view to plain text mode 

we have to write:

GET /images/srpr/nav_logo73.png HTTP/1.0\r\n\r\n

GET /images/srpr/nav_logo73.png HTTP/1.0\r\n\r\n

To copy to clipboard, switch view to plain text mode 

now it works, and Wireshark finally see this as HTTP request

Also some servers seem to require you to specify the host like this:

GET /images/srpr/nav_logo73.png HTTP/1.1\r\nHost: www.google.pl\r\n\r\n

GET /images/srpr/nav_logo73.png HTTP/1.1\r\nHost: www.google.pl\r\n\r\n

To copy to clipboard, switch view to plain text mode 

otherwise you will get "Bad request".

Thanks for the help everyone.

Problem seems solved

[squidge]

Yes, sometimes the host is required because more than one website is hosted on one IP address.

Best thing to do is to always provide it.

But now you know how it works, you'll be using QNetworkManager anyway, right?

(and don't forget some hosts will return status codes such as 'redirection', you must handle these as well)

[Huk]

Yes, sometimes the host is required because more than one website is hosted on one IP address.

Best thing to do is to always provide it.

(and don't forget some hosts will return status codes such as 'redirection', you must handle these as well)

I see, thanks for the information.

But now you know how it works, you'll be using QNetworkManager anyway, right?

For the current application: yes, but if I ever need to create something more advanced, QTcpSocket will definitely be used there