PDA

View Full Version : CRLF in HTTP response



giusepped
31st August 2011, 15:54
I am writing a class to parse the HTTP headers.
I need to pick up one particulr line .
The problem is that some servers encode the CRLF with "\r\n" others with "\n".
How to detect or handle different encoding?

high_flyer
31st August 2011, 16:08
From what you posted, all serves will use '\n', sometimes with or without '\r'
So look for '\n' and you are on the safe side.

yeye_olive
31st August 2011, 17:33
Read the headers through a QTextStream and use QTextStream::readLine().

wysota
31st August 2011, 19:23
I am writing a class to parse the HTTP headers.
I need to pick up one particulr line .
The problem is that some servers encode the CRLF with "\r\n" others with "\n".
How to detect or handle different encoding?

What's wrong with QHttpRequestHeader?

yeye_olive
1st September 2011, 00:37
What's wrong with QHttpRequestHeader?
It is obsolete.

Besides, the OP apparently needs to parse the header sent by a server, in which case the equally obsolete QHttpResponseHeader seems better suited.

I suppose the modern approach would be to use QNetworkReply::header(), or QNetworkReply::rawHeader(), etc.

wysota
1st September 2011, 01:07
It is obsolete.
So? It's obsolete because of a reason that doesn't apply here.


I suppose the modern approach would be to use QNetworkReply::header(), or QNetworkReply::rawHeader(), etc.
QNetworkReply may not be obsolete but it is perfectly useless if one doesn't use QNetworkAccessManager. And if he does, there is no need to parse http headers, is there?

yeye_olive
1st September 2011, 13:32
So? It's obsolete because of a reason that doesn't apply here.
I admit I do not know that reason, but I would be interested in learning it. In any case the docs state:

This class is obsolete. It is provided to keep old source code working. We strongly advise against using it in new code.
Since I can see no clear indication of whether giusepped is developing new code or building on existing code already using QHttp* classes, I find that state of affairs worth mentioning.

QNetworkReply may not be obsolete but it is perfectly useless if one doesn't use QNetworkAccessManager. And if he does, there is no need to parse http headers, is there?
I cannot think of any reason to parse the headers manually either, at least to the point that the end-of-line encoding matters, regardless of whether the QNetworkAccessManager or QHttp* approach is used, but I have no idea what giusepped is trying to achieve.

wysota
1st September 2011, 18:19
I admit I do not know that reason, but I would be interested in learning it.
It's obsolete because it was meant to be used with QHttp and since QHttp is obsolete now, this class is also marked as such. But it doesn't mean that it's obsolete for every other usecase (like when you're developing your own http server or proxy).


Since I can see no clear indication of whether giusepped is developing new code or building on existing code already using QHttp* classes, I find that state of affairs worth mentioning.
Do you see any indication that the OP wants to download anything via http?


I cannot think of any reason to parse the headers manually either
I see lots of reasons to do that but only if your goal is not to issue a simple get or post request to a remote http server.

giusepped
2nd September 2011, 14:26
I cannot use QNetworkAccesManager because I am building an http proxy. Maybe it can be done also by using it, but I think it is more difficult.
Using simple QTcpSocket is faster, but at that point I have to parse the headers.

wysota
2nd September 2011, 16:25
Have a look here: http://www.qtcentre.org/threads/34082-NetworqDebugger. It's not a HTTP proxy but maybe it will give you some insights. For parsing HTTP headers I suggest to use QHttpRequestHeader and QHttpResponseHeader.

giusepped
5th September 2011, 12:39
It seems that the file you attached in the post cited is not valid. Could you post it her?
However the problem is that everything from remote host can come in chunk, also headers.
G

giusepped
6th September 2011, 12:00
I solved myself the problem, which is non trivial.
Web server can use different scheme for separate header lines.
You can have
'\n' or '\r\n' or '\n\r'.
Believe me beacause I tested a lot of embedded web server (not the bigger ones).
The (most) important part of the header is the Content-Length.
If you (as me) are planning to change the body, you must carefull re-write the Content-Length value.
Follow the code.
The "line" variable is a line you want to analyze, for example (Content-Length). It is simple to extract it from the header


int Util::getCR(const QByteArray &line)
{
int c = 0;
char ch;
QChar sc;
for(int i=0;i<line.size();i++)
{
sc = line.at(i);
if (sc.isDigit())
ch = sc.unicode() - '0';
//qDebug()<<"UTIL getCR byte"<<line.at(i)<<sc.unicode()*1;;


if (c == 13)

{
// We found CR ASCCI '0D'
// Check if there is a LF

if (i == line.size()-1)
{
sc = line.at(i+1);
c = sc.unicode() * 1;
if (c == 10)
return i+1;
}
else
return i;

}
else
// Do the same but first is '0A'
if ( c == 10)
{
// We found CR ASCCI '0D'
// Check if there is a LF

if (i == line.size()-1)
{
sc = line.at(i+1);
c = sc.unicode() * 1;
if (c == 13)
return i+1;
}
else
return i;
}
}



}

wysota
6th September 2011, 12:25
It seems that the file you attached in the post cited is not valid.
It is valid.

giusepped
6th September 2011, 18:43
I tried to unzip it many times. Alway the zipper complains that the archive is invalid.
Regards

squidge
6th September 2011, 22:52
I can confirm the file is valid as I've just downloaded it.

Confirm your software is working OK.

ChrisW67
7th September 2011, 01:46
Web server can use different scheme for separate header lines.
You can have
'\n' or '\r\n' or '\n\r'.
Believe me beacause I tested a lot of embedded web server (not the bigger ones).

You should report any embedded web server that uses something other "\r\n" to the creator of the server software. The HTTP RFC is clear:
"HTTP/1.1 defines the sequence CR LF as the end-of-line marker for all protocol elements except the entity-body"
http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.2 (Was the same in HTTP/1).

The RFC advises tolerance (http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.3) but even it does not mention LFCR as a possibility. You still have to accept the rubbish data from the server, but if you don't tell them they cannot fix it.

giusepped
7th September 2011, 09:55
You should report any embedded web server that uses something other "\r\n" to the creator of the server software. The HTTP RFC is clear:

I agree with you. But also TCP has an RFC and there out there a lot of TCP flavors totally out of the standard (think of download accelerators).
Anyway, what you suggest is very difficult when the producer of the embedded video server is an unknown China based maker and you have to provide your client with a solution. On the other hand, we are engineer after all, aren't we?

wysota
7th September 2011, 14:44
I agree with you. But also TCP has an RFC and there out there a lot of TCP flavors totally out of the standard (think of download accelerators).
Not really. There are a lot of extensions to TCP defined and they have separate RFC documents. It is true that some implementations (mostly Windows) do not conform in 100% to TCP specs but this is mostly related to unused or reserved fields.

giusepped
7th September 2011, 15:49
However, your class is of little help and I ma stucked with this problem.
I am looking for someone can help me.
Could you take a look at my code?
It works for most web servers except one.
Regards

wysota
7th September 2011, 17:04
Did you try using QHttpRequestHeader and QHttpResponseHeader as I suggested?

giusepped
7th September 2011, 19:48
Yes, it does not work.
Regards

wysota
7th September 2011, 19:53
Could you provide a sample of http headers for which the class fails? You can provide it as an attachment to your post.

giusepped
22nd September 2011, 08:24
I deeply analyzed my problems, and it seems that it is not a problem of header parsing.
Basically, my application makes use of the proxy I implemented. Well, if I open a particular website outisde
the application, e.g. in IE and proxy set to mine, the website loads correctly.
If I run an activex inside my app, which behaves like IE, the app freezes.
To load the activex I use this very simple code (the app is based on MDI mainwindow):



MdiBrowser *child = createMdiBrowser();
child->setMode(0);
child->setWindowTitle(id.name);
child->setAddress(ip);
child->show();


and the class MdiBrowser has



MdiBrowser::MdiBrowser(QWidget *parent)
: QWidget(parent)
{
...
wIE->setControl("{8856F961-340A-11D0-A96B-00C04FD705A2}");
}
void MdiBrowser::setAddress(const QHostAddress &a)
{
wIE->dynamicCall("Navigate(const QString&)",a);
}


Added after 43 minutes:

Maybe the problem is the mingw not compatible with QAxWidget.
Read this for example
https://bugreports.qt.nokia.com/browse/QTBUG-16565