Results 1 to 12 of 12

Thread: Reading web content with QNetworkAccessManager and QNetworkReply

  1. #1
    Join Date
    Jul 2010
    Location
    Poland
    Posts
    184
    Thanks
    70
    Thanked 7 Times in 6 Posts
    Qt products
    Qt4 Qt/Embedded
    Platforms
    Unix/X11 Windows

    Default Reading web content with QNetworkAccessManager and QNetworkReply

    Hello!

    I've got problem with reading content of web pages using QNetworkAccessManager and QNetworkReply. I've got code:

    Qt Code:
    1. void MainWindow::getXML()
    2. {
    3. qDebug() << "Getting content..." << endl;
    4.  
    5. QNetworkRequest request(QUrl("http://www.google.pl"));
    6. qDebug() << "Network request..." << endl;
    7.  
    8. NetRepl = NetAccMan.get(request);
    9. qDebug() << "Network reply..." << endl;
    10.  
    11. connect(NetRepl, SIGNAL(readyRead()), this, SLOT(parseXML()));
    12. qDebug() << "Connect..." << endl;
    13. }
    14.  
    15. void MainWindow::parseXML()
    16. {
    17. qDebug() << "Ready to parse";
    18.  
    19. QByteArray newData = NetRepl->read(2048);
    20.  
    21. qDebug() << newData << endl;
    22. [...]
    23. }
    To copy to clipboard, switch view to plain text mode 

    It reads almost every content, but there are some pages that I can't get content (it doesn't go to parseXML()). For example my WWW server (it's simple http server based on small AVR controller) generates simple pages with data that I want to parse:

    Qt Code:
    1. <RESP>
    2. <FUNC>some_func</FUNC>
    3. <VAL>true</VAL>
    4. <DATA>
    5. <D1>1</D1>
    6. <D2>2</D2>
    7. </DATA>
    8. </RESP>
    To copy to clipboard, switch view to plain text mode 

    Only this and nothing more. Do I need some header? How does it work because I'm a little bit confused.

    thanks in advance
    best regards
    Tomasz

  2. #2
    Join Date
    Jul 2010
    Location
    Poland
    Posts
    184
    Thanks
    70
    Thanked 7 Times in 6 Posts
    Qt products
    Qt4 Qt/Embedded
    Platforms
    Unix/X11 Windows

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    When I want to read data using my web browser, everything works great. Maybe, there is other way, than using QNetworkAccessManager and QNetworkReply to read somethig from http server?

    thanks in advance
    best regards
    Tomasz

  3. #3
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    The QIODevice::readyRead() signal indicates that some data is available, not necessarily a lot of data or all the data. Unless you are incrementally parsing the XML you probably want the QNetworkReply::finished() signal.

  4. #4
    Join Date
    Jul 2010
    Location
    Poland
    Posts
    184
    Thanks
    70
    Thanked 7 Times in 6 Posts
    Qt products
    Qt4 Qt/Embedded
    Platforms
    Unix/X11 Windows

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    I've tried to use it as You said, and now It goes to parseXML() but when I'm showing the content of a 'newData' buffer - It's empty. Maybe I'm doing something wrong? Any idea?

    I can add, that I've got some CGI functions on my AVR, that I use to control microcontroller, and when I trigger them using QNetworkAccessManager and QNetworkReply all of them works, but I can't still read results.

    thanks in advance
    best regards
    Tomasz
    Last edited by Tomasz; 7th February 2011 at 13:12.

  5. #5
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    If you are getting nothing at all in the QByteArray then look at QNetworkReply::error() for a clue. You could also check that QNetworkReply::isFinished() returns true.

    I assume that NetRepl is a class member variable and that only one request can be outstanding at any time. If you allow more than one request the second and subsequently reply pointers will overwrite earlier ones. This situation may result in an unfinished request being accessed in response to another request completing. Use the QNetworkAccessManager::finished() signal, and don't store the QNetworkReply pointer, if you need multiple requests active.

    If any of the responses can possibly be larger than 2048 bytes then you would be safer using readAll().

  6. #6
    Join Date
    Jul 2010
    Location
    Poland
    Posts
    184
    Thanks
    70
    Thanked 7 Times in 6 Posts
    Qt products
    Qt4 Qt/Embedded
    Platforms
    Unix/X11 Windows

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    Quote Originally Posted by ChrisW67 View Post
    If you are getting nothing at all in the QByteArray then look at QNetworkReply::error() for a clue. You could also check that QNetworkReply::isFinished() returns true. [...]
    I've done as You said:

    Qt Code:
    1. void MainWindow::getXML()
    2. {
    3. ui->textEdit->append("Getting content...");
    4.  
    5. QNetworkRequest request(QUrl("[my_address]"));
    6. ui->textEdit->append("Network request...");
    7.  
    8. NetRepl = NetAccMan.get(request);
    9. ui->textEdit->append("Network reply...");
    10.  
    11. connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
    12.  
    13. ui->textEdit->append("Connect...");
    14. }
    15.  
    16. void MainWindow::parseXML()
    17. {
    18. ui->textEdit->append("Ready to download and parse...");
    19. ui->textEdit->append(QString("isFinished:").append(QString::number(NetRepl->isFinished())));
    20. ui->textEdit->append(QString("error:").append(QString::number(NetRepl->error())));
    21.  
    22. QByteArray newData = NetRepl->read(2048);
    23.  
    24. ui->textEdit->append(QString(newData));
    25.  
    26. NetRepl->deleteLater();
    27. }
    To copy to clipboard, switch view to plain text mode 

    In my header file I've got:

    Qt Code:
    1. [...]
    2. private:
    3. Ui::MainWindow *ui;
    4.  
    5. QNetworkReply *NetRepl;
    6. QNetworkAccessManager NetAccMan;
    To copy to clipboard, switch view to plain text mode 

    I've used finished() signal of QNetworkReply, as You said earlier (connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));). Now It goes to parseXML() and QNetworkReply::error() returns 2 (QNetworkReply::RemoteHostClosedError) and QNetworkReply::isFinished() returns 1. Buffer (newData) is empty. I thik there is no more than one reqest at time, and any of response isn't bigger than 2048 bytes. Any ideas how can I make it work (in browser it works)?

    If it's important I can add that those pages that I want to read are CGI functions, which return some values. Address looks like this: http://ip_address/cgi-bin/cgi_functi...?parameter=xxx.

    thanks in advance
    best regards
    Tomasz
    Last edited by Tomasz; 8th February 2011 at 14:19.

  7. #7
    Join Date
    Jul 2010
    Location
    Poland
    Posts
    184
    Thanks
    70
    Thanked 7 Times in 6 Posts
    Qt products
    Qt4 Qt/Embedded
    Platforms
    Unix/X11 Windows

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    Maybe I should use some other methods to achieve what I want? Maybe some other classes? My code works fine when I set address that don't exists (correct IP with page that don't exists) then my server responses with "404 Not Found" and I can read It.

    thanks in advance
    best regards
    Tomasz

  8. #8
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    Is your AVR generating realistic HTTP headers? A Length: atribute? Can you post the result of this:
    Qt Code:
    1. wget -S http://your.target.url
    To copy to clipboard, switch view to plain text mode 
    (I assume you are using a Linux box)

  9. The following user says thank you to ChrisW67 for this useful post:

    Tomasz (9th February 2011)

  10. #9
    Join Date
    Jul 2010
    Location
    Poland
    Posts
    184
    Thanks
    70
    Thanked 7 Times in 6 Posts
    Qt products
    Qt4 Qt/Embedded
    Platforms
    Unix/X11 Windows

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    Quote Originally Posted by ChrisW67 View Post
    Is your AVR generating realistic HTTP headers?
    You're right. Actually there where no HTTP headers. I haven't even thought about It. I've modified server source code, and now It works just fine! But I've got one more question - can I use the same "QNetworkReply *NetRepl;" and "QNetworkAccessManager NetAccMan;" a couple times? I mean for example in two different functions:

    Qt Code:
    1. void MainWindow::getXML1()
    2. {
    3. QNetworkRequest request(QUrl("[my_address_1]"));
    4. NetRepl = NetAccMan.get(request);
    5. connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
    6. }
    7.  
    8. void MainWindow::getXML2()
    9. {
    10. QNetworkRequest request(QUrl("[my_address_2]"));
    11. NetRepl = NetAccMan.get(request);
    12. connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
    13. }
    To copy to clipboard, switch view to plain text mode 

    And ofcourse in parseXML() I will have:
    Qt Code:
    1. void MainWindow::parseXML()
    2. {
    3. [...]
    4.  
    5. NetRepl->disconnect();
    6. NetRepl->deleteLater();
    7. }
    To copy to clipboard, switch view to plain text mode 

    Would It (putting deleteLater() only once) be enough to get rid of all unused stuff after downloading XML content? Event If one function is triggered after other? I'm asking because class reference says this about deleteLater():

    Note that entering and leaving a new event loop (e.g., by opening a modal dialog) will not perform the deferred deletion; for the object to be deleted, the control must return to the event loop from which deleteLater() was called.
    thanks in advance
    best regards
    Tomasz

  11. #10
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    Shows just how tolerant web browsers are.

    You can use the same QNetworkAccessManager for the life of the program. You get a new QNetworkReply with each request.

    Starting a request is a non-blocking operation, so several requests can be issued before any reply is received. If you store the reply pointer in a member variable (in NetRepl) and the second request is issued before the first is finished completely then you lose track of the first request. In that case you have a memory leak. Use the finished signal from QNetworkAccessManager then you need not keep a copy of the request pointer. The slot receives the reply pointer for the reply that just finished (they may not arrive in the order you issued them). You still need to arrange deletion.

  12. The following user says thank you to ChrisW67 for this useful post:

    Tomasz (10th February 2011)

  13. #11
    Join Date
    Jul 2010
    Location
    Poland
    Posts
    184
    Thanks
    70
    Thanked 7 Times in 6 Posts
    Qt products
    Qt4 Qt/Embedded
    Platforms
    Unix/X11 Windows

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    So I should do something like this:

    Qt Code:
    1. void MainWindow::getXML()
    2. {
    3. QNetworkRequest request(QUrl("[my_address]"));
    4. NetAccMan.get(request);
    5. connect(&NetAccMan, SIGNAL(finished(QNetworkReply*)), this, SLOT(parseXML(QNetworkReply*)));
    6. }
    7.  
    8. void MainWindow::parseXML(QNetworkReply *networkReply)
    9. {
    10. [...]
    11.  
    12. NetAccMan->disconnect();
    13. networkReply->deleteLater();
    14. }
    To copy to clipboard, switch view to plain text mode 

    Would it be correct? Or deletion in this case should be arranged in different way?

    thanks in advance
    best regards
    Tomasz

  14. #12
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Reading web content with QNetworkAccessManager and QNetworkReply

    Delete looks fine to me. You don't want to connect/disconnect your QNetworkAccessManager with every request though, just make the connection once in the MainWindow constructor and wait until the MainWindow is destroyed for automatic disconnection.
    Last edited by ChrisW67; 10th February 2011 at 01:45.

  15. The following user says thank you to ChrisW67 for this useful post:

    Tomasz (10th February 2011)

Similar Threads

  1. Replies: 2
    Last Post: 18th January 2012, 16:01
  2. Replies: 1
    Last Post: 10th October 2010, 00:54
  3. Replies: 2
    Last Post: 18th June 2010, 12:04
  4. Problems reading content of QFile
    By martinn in forum Newbie
    Replies: 12
    Last Post: 6th April 2010, 19:42
  5. Replies: 2
    Last Post: 9th January 2010, 03:03

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.