How to see the final request from QNetworkRequest

[krippy2k]

Hi all,

I'm currently writing an application that must communicate with a secure web server via HTTPS. I am using the QNetworkAccessManager class.

I have run into a few issues along the way and it would be so much easier to solve them if I could see the text of the final HTTP request that is sent. So I'm wondering is there an easy way to see the exact HTTP request that is being generated by QNetworkRequest in plain text and/or the exact HTTP response as it is received?

Thanks

[wysota]

Use a network sniffer.

[krippy2k]

A network sniffer doesn't help me with HTTPS requests.

[wysota]

Sure it does. A good network sniffer can be set up in a way to perform a "man in the middle" attack on your own SSL traffic. And you can also disable SSL and use HTTP, the contents of the request will be the same.

[krippy2k]

I have yet to see a network sniffer that will do that. Can you tell me which one will?

Anyway, even if there is, that is kind of beside the point of this thread. Disabling SSL and using an HTTP analyzer is what I have been doing, but it's a pain in the ass, as would be just using some network sniffer that can attack my own SSL traffic, which is why I am here asking if there is a way to get the request string from Qt and save me the hassle. But I guess there isn't. I'll just have to dig into the source code and figure out how to do it myself.

I'd still be interested in knowing of a network sniffer that will do that, as that might be useful at some point. But as far as I can tell none of the sniffers that we have will do it.

[wysota]

I have yet to see a network sniffer that will do that. Can you tell me which one will?
AFAIR I've been using ethereal but in general you can even use netcat or sslserver for this.


which is why I am here asking if there is a way to get the request string from Qt and save me the hassle. But I guess there isn't.
If there was an easy way to do it, would I suggest using a network sniffer?


I'd still be interested in knowing of a network sniffer that will do that, as that might be useful at some point. But as far as I can tell none of the sniffers that we have will do it.

Man in the middle attack is trivial to implement, you can even write your own application using QSslServer and QSslSocket that does it. It would take only a few minutes to implement such a thing.

[krippy2k]

If there was an easy way to do it, would I suggest using a network sniffer?

I don't know if you would or not. People on forums have a way of coming up with gratuitous responses. Being new here I didn't realize you were the ultimate authority on Qt and was irrationally holding out hope that somebody else might know :P

I've always marveled that other HTTP/S libraries didn't provide this type of functionality, and since Qt has done so many other things better than other libraries I was hoping they might have done this. Now I know better.

That being said, I'm in the process of digging into the Qt source to add this functionality, as it would be very useful to myself and probably to others. I'll publish the results when it's finished.

[wysota]

You can use QHttp directly then you'll have access to the actual request content. But QNetworkAccessManager is much better.

Anyway, why do you need to access the contents of the request? If it's about headers then those are accessible.

[krippy2k]

Well it's just nice to be able to visualize the entire request. It helps to spot problems quicker. Being able to see all of the headers, including cookies, user-agent, Content-Type, etc can be useful. It's also nice to be able to see the actual operation request.

Here is an example of where it probably would have saved me time. I had to parse out a URL from a response and redirect to that URL. This URL was already URL encoded. Now when I made a request for that URL, Qt encoded it again. This is understandable, and ultimately my fault, but I couldn't easily see what was happening so i didn't know why the request was broken. If I would have been able to just visualize the entire HTTP request, I probably would have noticed the double encoding in the GET / portion of the HTTP request immediately instead of wasting 20 minutes fartin' around.

I found QHttpNetworkRequestPrivate in the Qt source where it seems to ultimately resolve the request to a QByteArray, so I should be able to work something in to make that accessible to user code.

[wysota]

Well it's just nice to be able to visualize the entire request. It helps to spot problems quicker. Being able to see all of the headers, including cookies, user-agent, Content-Type, etc can be useful. It's also nice to be able to see the actual operation request.
For the headers there is QNetworkRequest::rawHeaderList(). Just retrieve it from the object contained in QNetworkReply, not from the request you fill yourself.


Here is an example of where it probably would have saved me time. I had to parse out a URL from a response and redirect to that URL. This URL was already URL encoded. Now when I made a request for that URL, Qt encoded it again. This is understandable, and ultimately my fault, but I couldn't easily see what was happening so i didn't know why the request was broken. If I would have been able to just visualize the entire HTTP request, I probably would have noticed the double encoding in the GET / portion of the HTTP request immediately instead of wasting 20 minutes fartin' around.
This you could have accessed through QNetworkReply::request() and QNetworkRequest::url(). You can dump that info to debug to see the contents (along the raw header list).

[krippy2k]

Thanks, that is good to know, I did not notice that the request was tacked on to the reply with all of the headers and final url. That could be useful.

I just did some savage hackery in the Qt source to obtain the final request data though so I can now see the exact data that is being sent. The glory of open source!

Cheers

[krippy2k]

For the headers there is QNetworkRequest::rawHeaderList(). Just retrieve it from the object contained in QNetworkReply, not from the request you fill yourself.



BTW, I tried this but it doesn't seem to work. Maybe I misunderstood what you meant.


This is the code that I tried:



QNetworkRequest request = reply->request();

qDebug() << "Request headers: ";

QList<QByteArray> reqHeaders = request.rawHeaderList();
foreach( QByteArray reqName, reqHeaders )
{
QByteArray reqValue = request.rawHeader( reqName );
qDebug() << reqName << ": " << reqValue;
}


This will only print any headers that I specifically set with setRawHeader on the original request. If I don't set any headers myself then it doesn't print anything, even though I know it's sending at least Host and User-Agent headers.

Am I doing something wrong here?

Thanks

[krippy2k]

Actually I'm mistaken, I see now that it will also print some Cookie headers after cookies are set. I guess I see why the other headers aren't seen there; they're never a part of the request but are just added directly to the output byte array in the private classes.

[wysota]

they're never a part of the request but are just added directly to the output byte array in the private classes.

If that's so then I'd file a bug report on that.