CRLF in HTTP response

[giusepped]

I am writing a class to parse the HTTP headers.
I need to pick up one particulr line .
The problem is that some servers encode the CRLF with "\r\n" others with "\n".
How to detect or handle different encoding?

[high_flyer]

From what you posted, all serves will use '\n', sometimes with or without '\r'
So look for '\n' and you are on the safe side.

[yeye_olive]

Read the headers through a QTextStream and use QTextStream::readLine().

[wysota]

I am writing a class to parse the HTTP headers.
I need to pick up one particulr line .
The problem is that some servers encode the CRLF with "\r\n" others with "\n".
How to detect or handle different encoding?

What's wrong with QHttpRequestHeader?

[yeye_olive]

What's wrong with QHttpRequestHeader?
It is obsolete.

Besides, the OP apparently needs to parse the header sent by a server, in which case the equally obsolete QHttpResponseHeader seems better suited.

I suppose the modern approach would be to use QNetworkReply::header(), or QNetworkReply::rawHeader(), etc.

[wysota]

It is obsolete.
So? It's obsolete because of a reason that doesn't apply here.


I suppose the modern approach would be to use QNetworkReply::header(), or QNetworkReply::rawHeader(), etc.
QNetworkReply may not be obsolete but it is perfectly useless if one doesn't use QNetworkAccessManager. And if he does, there is no need to parse http headers, is there?

[yeye_olive]

So? It's obsolete because of a reason that doesn't apply here.
I admit I do not know that reason, but I would be interested in learning it. In any case the docs state:

This class is obsolete. It is provided to keep old source code working. We strongly advise against using it in new code.
Since I can see no clear indication of whether giusepped is developing new code or building on existing code already using QHttp* classes, I find that state of affairs worth mentioning.

QNetworkReply may not be obsolete but it is perfectly useless if one doesn't use QNetworkAccessManager. And if he does, there is no need to parse http headers, is there?
I cannot think of any reason to parse the headers manually either, at least to the point that the end-of-line encoding matters, regardless of whether the QNetworkAccessManager or QHttp* approach is used, but I have no idea what giusepped is trying to achieve.

[wysota]

I admit I do not know that reason, but I would be interested in learning it.
It's obsolete because it was meant to be used with QHttp and since QHttp is obsolete now, this class is also marked as such. But it doesn't mean that it's obsolete for every other usecase (like when you're developing your own http server or proxy).


Since I can see no clear indication of whether giusepped is developing new code or building on existing code already using QHttp* classes, I find that state of affairs worth mentioning.
Do you see any indication that the OP wants to download anything via http?


I cannot think of any reason to parse the headers manually either
I see lots of reasons to do that but only if your goal is not to issue a simple get or post request to a remote http server.

[giusepped]

I cannot use QNetworkAccesManager because I am building an http proxy. Maybe it can be done also by using it, but I think it is more difficult.
Using simple QTcpSocket is faster, but at that point I have to parse the headers.

[wysota]

Have a look here: http://www.qtcentre.org/threads/34082-NetworqDebugger. It's not a HTTP proxy but maybe it will give you some insights. For parsing HTTP headers I suggest to use QHttpRequestHeader and QHttpResponseHeader.

[giusepped]

It seems that the file you attached in the post cited is not valid. Could you post it her?
However the problem is that everything from remote host can come in chunk, also headers.
G

[giusepped]

I solved myself the problem, which is non trivial.
Web server can use different scheme for separate header lines.
You can have
'\n' or '\r\n' or '\n\r'.
Believe me beacause I tested a lot of embedded web server (not the bigger ones).
The (most) important part of the header is the Content-Length.
If you (as me) are planning to change the body, you must carefull re-write the Content-Length value.
Follow the code.
The "line" variable is a line you want to analyze, for example (Content-Length). It is simple to extract it from the header


int Util::getCR(const QByteArray &line)
{
int c = 0;
char ch;
QChar sc;
for(int i=0;i<line.size();i++)
{
sc = line.at(i);
if (sc.isDigit())
ch = sc.unicode() - '0';
//qDebug()<<"UTIL getCR byte"<<line.at(i)<<sc.unicode()*1;;


if (c == 13)

{
// We found CR ASCCI '0D'
// Check if there is a LF

if (i == line.size()-1)
{
sc = line.at(i+1);
c = sc.unicode() * 1;
if (c == 10)
return i+1;
}
else
return i;

}
else
// Do the same but first is '0A'
if ( c == 10)
{
// We found CR ASCCI '0D'
// Check if there is a LF

if (i == line.size()-1)
{
sc = line.at(i+1);
c = sc.unicode() * 1;
if (c == 13)
return i+1;
}
else
return i;
}
}



}

[wysota]

It seems that the file you attached in the post cited is not valid.
It is valid.

[giusepped]

I tried to unzip it many times. Alway the zipper complains that the archive is invalid.
Regards

[squidge]

I can confirm the file is valid as I've just downloaded it.

Confirm your software is working OK.

[ChrisW67]

Web server can use different scheme for separate header lines.
You can have
'\n' or '\r\n' or '\n\r'.
Believe me beacause I tested a lot of embedded web server (not the bigger ones).

You should report any embedded web server that uses something other "\r\n" to the creator of the server software. The HTTP RFC is clear:
"HTTP/1.1 defines the sequence CR LF as the end-of-line marker for all protocol elements except the entity-body"
http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.2 (Was the same in HTTP/1).

The RFC advises tolerance (http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.3) but even it does not mention LFCR as a possibility. You still have to accept the rubbish data from the server, but if you don't tell them they cannot fix it.

[giusepped]

You should report any embedded web server that uses something other "\r\n" to the creator of the server software. The HTTP RFC is clear:

I agree with you. But also TCP has an RFC and there out there a lot of TCP flavors totally out of the standard (think of download accelerators).
Anyway, what you suggest is very difficult when the producer of the embedded video server is an unknown China based maker and you have to provide your client with a solution. On the other hand, we are engineer after all, aren't we?

[wysota]

I agree with you. But also TCP has an RFC and there out there a lot of TCP flavors totally out of the standard (think of download accelerators).
Not really. There are a lot of extensions to TCP defined and they have separate RFC documents. It is true that some implementations (mostly Windows) do not conform in 100% to TCP specs but this is mostly related to unused or reserved fields.

[giusepped]

However, your class is of little help and I ma stucked with this problem.
I am looking for someone can help me.
Could you take a look at my code?
It works for most web servers except one.
Regards

[wysota]

Did you try using QHttpRequestHeader and QHttpResponseHeader as I suggested?

[giusepped]

Yes, it does not work.
Regards

[wysota]

Could you provide a sample of http headers for which the class fails? You can provide it as an attachment to your post.

[giusepped]

I deeply analyzed my problems, and it seems that it is not a problem of header parsing.
Basically, my application makes use of the proxy I implemented. Well, if I open a particular website outisde
the application, e.g. in IE and proxy set to mine, the website loads correctly.
If I run an activex inside my app, which behaves like IE, the app freezes.
To load the activex I use this very simple code (the app is based on MDI mainwindow):



MdiBrowser *child = createMdiBrowser();
child->setMode(0);
child->setWindowTitle(id.name);
child->setAddress(ip);
child->show();


and the class MdiBrowser has



MdiBrowser::MdiBrowser(QWidget *parent)
: QWidget(parent)
{
...
wIE->setControl("{8856F961-340A-11D0-A96B-00C04FD705A2}");
}
void MdiBrowser::setAddress(const QHostAddress &a)
{
wIE->dynamicCall("Navigate(const QString&)",a);
}


Added after 43 minutes:

Maybe the problem is the mingw not compatible with QAxWidget.
Read this for example
https://bugreports.qt.nokia.com/browse/QTBUG-16565