QT cannot read continuation or non-http traffic

[SailingDreams]

Hi

I am trying to communicate to an AXIS product and it sends me serial data over HTTP. The sequence is:

HTTP GET ->
<- HTTP 204 - No Content
<- HTTP Continuation or non-Http traffic
<- HTTP Continuation or non-Http traffic
<- HTTP Continuation or non-Http traffic
etc

HTTP GET ->
<- HTTP 204 - No Content
<- HTTP Continuation or non-Http traffic
<- HTTP Continuation or non-Http traffic
<- HTTP Continuation or non-Http traffic
etc

To copy to clipboard, switch view to plain text mode 

I have traced through the QT code to the function QHttpPrivate::_q_slotReadyRead(). It appears that QT thinks it must find a valid header in the " HTTP Continuation or non-Http traffic" packets. Since the code can't find the header, it returns and does not extract the data.

void QHttpPrivate::_q_slotReadyRead()
{
    Q_Q(QHttp);
    QHttp::State oldState = state;
    if (state != QHttp::Reading) {
        setState(QHttp::Reading);
        readHeader = true;
        headerStr = QLatin1String("");
        bytesDone = 0;
        chunkedSize = -1;
        repost = false;
    }
 
    while (readHeader) {
        bool end = false;
        QString tmp;
        while (!end && socket->canReadLine()) {
            tmp = QString::fromAscii(socket->readLine());
            if (tmp == QLatin1String("\r\n") || tmp == QLatin1String("\n") || tmp.isEmpty())
                end = true;
            else
                headerStr += tmp;
        }
 
        if (!end)
            return;  //****This is where it returns, ARG*********

void QHttpPrivate::_q_slotReadyRead()
{
    Q_Q(QHttp);
    QHttp::State oldState = state;
    if (state != QHttp::Reading) {
        setState(QHttp::Reading);
        readHeader = true;
        headerStr = QLatin1String("");
        bytesDone = 0;
        chunkedSize = -1;
        repost = false;
    }

    while (readHeader) {
        bool end = false;
        QString tmp;
        while (!end && socket->canReadLine()) {
            tmp = QString::fromAscii(socket->readLine());
            if (tmp == QLatin1String("\r\n") || tmp == QLatin1String("\n") || tmp.isEmpty())
                end = true;
            else
                headerStr += tmp;
        }

        if (!end)
            return;  //****This is where it returns, ARG*********

To copy to clipboard, switch view to plain text mode 

These packet types are apparently common (see http://ayis.javaeye.com/blog/351217), so why would QT drop them? Is there a bug in QT?

Any help would be greatly appreciated!

[wysota]

I think you are overinterpreting what you see. If what you say was true, you wouldn't be able to download anything larger than one TCP segment using QHttp which is obviously not the case. QHttp works in the application layer of the TCP/IP stack so it doesn't see segments or packets, it just sees a stream of bytes. The response has to begin with a header (that's what the HTTP protocol requires) but it doesn't mean every IP packet has to contain an HTTP header.

Are you reading the data that comes in through QHttp? Are you sure the read buffer is not full?

[SailingDreams]

Hi

I agree that QHttp should be able to read the data from a buffer, but what I think is happening is that the response 204 causes QHttp to move from the reading state to the connected state. I'm sure that if the AXIS sent a response 200, I would be able to read the data from the buffer.

I have set breakpoints in the code (specifically QHttpPrivate::_q_slotReadyRead()) and see it reading HTTP code 204, then transitioning to the connected state. Once there, it does not want to read.

I also tried using QNetworkAccessManager and it has the same problem. The 204 code is preventing QTs state machine from remaining in the read state.

I'm trying to do this now with QSslSocket. This is going to be painful. I wish I could find an example where someone DID NOT use QHttp to setup an HTTP GET.

Thanks for your help!

[SailingDreams]

How about this idea (I'd sure appreciate a OK or NAY from a QT guru)

1) Setup my own socket using QSslSocket
2) Use QHttp:setSocket to get QHttp to use that socket
3) use QSslSocket.bytesAvailable to see if I get data
4) and if I do, use QSslSocket.read()

I assume that read() will read the buffer and not the individual packets?

Thanks in advance.

[wysota]

I have set breakpoints in the code (specifically QHttpPrivate::_q_slotReadyRead()) and see it reading HTTP code 204, then transitioning to the connected state. Once there, it does not want to read.

According to the specification of HTTP:

The 204 response MUST NOT include a message-body, and thus is always terminated by the first empty line after the header fields.

Thus I think what QHttp does is correct... It reads headers and returns to the connected state.

How about this idea (I'd sure appreciate a OK or NAY from a QT guru)

1) Setup my own socket using QSslSocket
2) Use QHttp:setSocket to get QHttp to use that socket
3) use QSslSocket.bytesAvailable to see if I get data
4) and if I do, use QSslSocket.read()

I assume that read() will read the buffer and not the individual packets?

Thanks in advance.

You can either read from the socket yourself or let it be managed by QHttp, not both at once.

[SailingDreams]

Thanks. Thats a great point: can't get read TCP if HTTP manages the socket.

I will try to shut down QHttp with abort(). Hopefully, since the socket was created external to QHttp, the socket stays open and then I can use QSslSocket.bytesAvailable() and read().

I think what AXIS is doing must be non-standard for sending serial data. They do video streams differently. For video, you do a GET and then AXIS starts streaming data in "TCP segment of a reassembled packet". Then its easy, you just use bytesAvailable() and read() to get the data. (the first TCP packet has HTTP 200 OK in it).

I wish there were a way to make QHttp go back to the reading state once it sees the 204.

If I am successful at shutting down QHttp with abort (it leaves the socket open), I worry that my next challenge will be TCP not reading the "HTTP Continuation or non-Http traffic
". I say this because these packets do not have a header termination \n\r and hence I've observed (by stepping through the code) that QHttp thinks the whole packet is a header and that the header is continued in the next packet. Hopefully QTcpSocket does not reject the data as being HTTP header info.

[wysota]

The socket will be closed.

[SailingDreams]

You are right. Arg. QHTTP::abort also kills the TCP connection. So much for their documentation:

QHttp does not take ownership of the socket, and will not delete socket when destroyed.

The good news is I was able to use QTcpSocket::read() and bytesAvailable() without deleting the QHttp object. I'm getting my data now!!

Thank-you very much for your help!

[SailingDreams]

Alas, I thought I had this problem solved. Using the QTcpSocket::read() worked when I sent data that was the character pattern 0 to 9,a to z, but when I connected the AXIS serial port to the device that feeds it serial data, something in the serial data makes QT think there is a valid HTTP header and it created to new request (very strange). The output below is from slots connnected to the various QHttp signals.

It is very strange that QT creates a Id=0 request. Why does it do that?

Opened file serialCapture.txt
requestStarted: request type setHost rx : id = 1 (id = 1)
requestFinished: request setHost rx : id = 1, ID = 1, error = No error
requestStarted: request type setSocket rx : id = 3 (id = 3)
requestFinished: request setSocket rx : id = 3, ID = 3, error = No error
requestStarted: request type GET rx connect : id = 4 (id = 4)
stateChanged: request type GET rx connect : id = 4 Id=4 is changed to Sending
stateChanged: ---reading state---
stateChanged: request type GET rx connect : id = 4 Id=4 is changed to Reading
readResponseHeader: code = 204 for Id = 4
stateChanged: request type GET rx connect : id = 4 Id=4 is changed to Connected
requestFinished: request GET rx connect : id = 4, ID = 4, error = No error
done: No Error.
******I start sending my canned serial data here (which comes as HTTP continuation of data packets)**************
stateChanged: ---reading state---
stateChanged: request type Id=0 is changed to Reading
readData: has read 1 packets of 51 bytes
readData: has read 5 packets of 3807 bytes
Closing stream capture file
readData: has read 9 packets of 7524 bytes
readData: has read 13 packets of 11256 bytes
readData: has read 17 packets of 14990 bytes
readData: has read 21 packets of 16868 bytes
******The device starts sending serial data here(which comes as HTTP continuation of data packets)**************

stateChanged: request type Id=0 is changed to Closing
stateChanged: request type Id=0 is changed to Unconnected
*****At this point the TCP connection is closed by QHttp. Arg!********

It looks like I have to write my own HTTP handler. Darn.

[wysota]

You're doing something very non-standard so I wouldn't rely on QHttp behaving in a sane way in such situation.