Results 1 to 15 of 15

Thread: Concurrent file downloading

  1. #1
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Question Concurrent file downloading

    Download one file at the time with source code below is easy, and also download multiply file also isn't problem when using http://doc.qt.io/qt-4.8/QQueue, but downloading files in parallel i can't get it!
    Assume we have:
    I make code shorter and remove other codes for demonstration purpose ;)
    downloader.h
    Qt Code:
    1. #ifndef DOWNLOADER_H
    2. #define DOWNLOADER_H
    3.  
    4. #include <QObject>
    5. #include <QFile>
    6. #include <QFileInfo>
    7. #include <QNetworkAccessManager>
    8. #include <QNetworkRequest>
    9. #include <QNetworkReply>
    10. #include <QUrl>
    11.  
    12. class Dowloader : public QObject
    13. {
    14. Q_OBJECT
    15. public:
    16. explicit Dowloader(QObject *parent = 0);
    17.  
    18. public slots:
    19. void theDownload(const QString &urlLink);
    20.  
    21. private slots:
    22. void startRequest(const QUrl &url);
    23. void httpReadyRead();
    24. void httpFinished();
    25.  
    26. private:
    27. QNetworkAccessManager manager;
    28. QNetworkReply *reply;
    29. QFile *file;
    30. };
    31.  
    32. #endif // DOWNLOADER_H
    To copy to clipboard, switch view to plain text mode 
    downloader.cpp
    Qt Code:
    1. #include "dowloader.h"
    2.  
    3. Dowloader::Dowloader(QObject *parent) :
    4. QObject(parent)
    5. {}
    6.  
    7. void Dowloader::theDownload(const QString &urlLink)
    8. {
    9. QUrl url(urlLink);
    10. QFileInfo fInfo(url.path());
    11. QString fileName = fInfo.fileName();
    12. if(fileName.isEmpty())
    13. fileName = "download";
    14.  
    15. file = new QFile(fileName);
    16.  
    17. if(!file->open(QIODevice::WriteOnly)){
    18. delete file;
    19. file = 0;
    20. return;
    21. }
    22. startRequest(url);
    23. }
    24.  
    25. void Dowloader::replyMetaDataChanged()
    26. {}
    27.  
    28. void Dowloader::startRequest(const QUrl &url)
    29. {
    30. QNetworkRequest request(url);
    31. reply = manager.get(request);
    32. // conncet signals
    33. connect(reply, SIGNAL(metaDataChanged()), this, SLOT(replyMetaDataChanged()));
    34. connect(reply, SIGNAL(readyRead()), this, SLOT(httpReadyRead()));
    35. connect(reply, SIGNAL(finished()), this, SLOT(httpFinished()));
    36. }
    37.  
    38. void Dowloader::httpReadyRead()
    39. {
    40. if(file) file->write(reply->readAll());
    41. }
    42.  
    43. void Dowloader::httpFinished()
    44. {
    45. file->flush();
    46. file->close();
    47. reply->deleteLater();
    48. reply = 0;
    49. delete file;
    50. file = 0;
    51. }
    To copy to clipboard, switch view to plain text mode 
    The main problem here is, in the httpReadyRead() ans httpFinished() how can operate on the correct file, reply? as you see in the *.h file they are *pointer, it means when i add new file current QNetworkReply *reply, QFile *file, gonna re-initialize with new values!
    What changes need in this code to make Concurrent file downloading possible? // Doesn't mean provide some code or something, just give me a scenario about it!

    Thanks
    Last edited by Alir3z4; 20th February 2012 at 16:49.
    ...یه مرد هیچوقت زمین نمیخوره

  2. #2
    Join Date
    Jan 2006
    Location
    Germany
    Posts
    4,380
    Thanks
    19
    Thanked 1,005 Times in 913 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows Symbian S60
    Wiki edits
    5

    Default Re: Concurrent file downloading

    QSignalMapper and QHash can help.

  3. The following user says thank you to Lykurg for this useful post:

    Alir3z4 (20th February 2012)

  4. #3
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Default Re: Concurrent file downloading

    Quote Originally Posted by Lykurg View Post
    QSignalMapper and QHash can help.
    Yes, first thing came to my head was using QHash, QSignalMapper, but what should i keep in the QHash?
    pointer or reference?
    pointer like this?
    Qt Code:
    1. QHash<QUrl, QHash<QNetworkReply*, QFile*> > *downloads;
    To copy to clipboard, switch view to plain text mode 
    if pointers, what happen when i add new download and re-initialize *reply*, *file, or something like that? // or maybe i shouldn't use *reply*, *file and just make pointers add them to hash on the fly?
    or i have to keep ref in it?
    Last edited by Alir3z4; 20th February 2012 at 17:26.
    ...یه مرد هیچوقت زمین نمیخوره

  5. #4
    Join Date
    Jan 2006
    Location
    Germany
    Posts
    4,380
    Thanks
    19
    Thanked 1,005 Times in 913 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows Symbian S60
    Wiki edits
    5

    Default Re: Concurrent file downloading

    Pointers and file/replay aren't getting overwritten since you store it in the hash. Use QHash<QNetworkReply*, QFile*>. Just make sure you delete the pointers after the downloads are finished.

  6. The following user says thank you to Lykurg for this useful post:

    Alir3z4 (20th February 2012)

  7. #5
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Concurrent file downloading

    You can also not store the reply pointer at all and use the QNetworkAccessManager::finished() signal, which passes the QNetworkReply object that finished. This is really only suitable for smaller files where the entire content is easily stored in memory and progress indication is not important.

  8. #6
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Default Re: Concurrent file downloading

    Quote Originally Posted by ChrisW67 View Post
    You can also not store the reply pointer at all and use the QNetworkAccessManager::finished() signal, which passes the QNetworkReply object that finished. This is really only suitable for smaller files where the entire content is easily stored in memory and progress indication is not important.
    As you said, for small file file.
    //
    By the way, i'm doing the QSignalMapper approach, but the new problem [weird] is just at the first time readyRead() signal emitted httpReadyRead() gonna work properly
    startRequest edited:
    Qt Code:
    1. void Dowloader::startRequest(const QUrl &url)
    2. {
    3. QNetworkRequest request(url);
    4. reply = manager.get(request);
    5. // conncet signals
    6. downloads->insert(reply, file);
    7. qDebug() << "downloads->insert(" << reply << "," << file << ");";
    8.  
    9. connect(downloads->key(file), SIGNAL(readyRead()), signalMapper, SLOT(map()));
    10. signalMapper->setMapping(downloads->key(file), downloads->value(reply));
    11. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpReadyRead(QObject*)));
    12. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpFinished(QObject*)));
    13. }
    To copy to clipboard, switch view to plain text mode 

    i changed the httpReadyRead to
    Qt Code:
    1. void Dowloader::httpReadyRead(QObject *_file)
    2. {
    3. QFile *theFile = qobject_cast<QFile*>(_file);
    4. if(file) theFile->write(downloads->key(theFile)->readAll());
    5. }
    To copy to clipboard, switch view to plain text mode 

    and downloads also is:
    Qt Code:
    1. QHash<QNetworkReply*, QFile*> *downloads
    To copy to clipboard, switch view to plain text mode 

    i change the code as i learn from docs, i don't know why writing to file in httpReadyRead works just at th first time, but next times when readyRead emitted from reply, theFile->openMode() print-out "NotOpen"
    any idea?


    Added after 30 minutes:


    EDITED:
    Before reading the reply data [reply->readAll();] if i force the *theFile to reopen it again [ theFile->open(QIODevice::WriteOnly); ] then i can writing the file
    But as soon readyRead emitted the openMode of theFile changed to NotOpen!
    What is the problem ?
    Last edited by Alir3z4; 21st February 2012 at 01:48.
    ...یه مرد هیچوقت زمین نمیخوره

  9. #7
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Default Re: Concurrent file downloading

    Actually i found the problem why *file openMode() changed to "NotOpen", this is QSignalMapper issue and get me confuse!
    this is the guilty line:
    Qt Code:
    1. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpFinished(QObject*)));
    To copy to clipboard, switch view to plain text mode 
    The problem is httpFinished(QObject*) gonna emitted right after httpReadyRead(QObject*) emits !
    when i // comment the httpFinished(QObject*), everything is fine! because in the httpFinished(QObject*) i shoulda flush()/close()/0/delete the file pointer, and of course do deleteLater()/remove form hash the reply pointer!

    So why httpFinished(QObject*) emitted immediately even when reply actually is not finished! ?


    Added after 8 minutes:


    Or should i use different QSignlaMapper for httpfinished() ?

    What? actually yes, i used different QSignalMapper and it works!
    did i get something wrong about QSignalMapper?
    Last edited by Alir3z4; 21st February 2012 at 15:55.
    ...یه مرد هیچوقت زمین نمیخوره

  10. #8
    Join Date
    Feb 2011
    Posts
    354
    Thanks
    17
    Thanked 27 Times in 24 Posts
    Qt products
    Qt4 Qt5
    Platforms
    MacOS X Windows

    Default Re: Concurrent file downloading

    look how you established your connections: you connected two slots to the same signal, that is why your slots are called together.

  11. #9
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Default Re: Concurrent file downloading

    Quote Originally Posted by mentalmushroom View Post
    look how you established your connections: you connected two slots to the same signal, that is why your slots are called together.
    ignore initialed connecting signals code
    i connected them before in this way:
    Qt Code:
    1. void Dowloader::startRequest(const QUrl &url)
    2. {
    3. QNetworkRequest request(url);
    4. reply = manager.get(request);
    5. QHash<QNetworkReply*, QFile*>::iterator i = downloads->insert( reply, file);
    6. downloads->insert(reply, file);
    7. signalMapper->setMapping(i.key(), i.value());
    8. connect(i.key(), SIGNAL(readyRead()), signalMapper, SLOT(map()));
    9. connect(i.key(), SIGNAL(finished()), signalMapper, SLOT(map()));
    10.  
    11. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpReadyRead(QObject*)));
    12. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpFinished(QObject*)));
    13. }
    To copy to clipboard, switch view to plain text mode 

    is this wrong ?
    ...یه مرد هیچوقت زمین نمیخوره

  12. #10
    Join Date
    Feb 2011
    Posts
    354
    Thanks
    17
    Thanked 27 Times in 24 Posts
    Qt products
    Qt4 Qt5
    Platforms
    MacOS X Windows

    Default Re: Concurrent file downloading

    it is wrong unless you want httpFinished(QObject*) to be called when readyRead is emitted

  13. #11
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Default Re: Concurrent file downloading

    Quote Originally Posted by mentalmushroom View Post
    it is wrong unless you want httpFinished(QObject*) to be called when readyRead is emitted
    Ow, i really lost in it now, how i want to called when readyRead is emitted ?
    I connect them as usual :| ?, i did't found any way to do it in different way!
    ...یه مرد هیچوقت زمین نمیخوره

  14. #12
    Join Date
    Feb 2011
    Posts
    354
    Thanks
    17
    Thanked 27 Times in 24 Posts
    Qt products
    Qt4 Qt5
    Platforms
    MacOS X Windows

    Default Re: Concurrent file downloading

    Qt Code:
    1. connect(i.key(), SIGNAL(readyRead()), signalMapper, SLOT(map()));
    2. connect(i.key(), SIGNAL(finished()), signalMapper, SLOT(map()));
    3.  
    4. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpReadyRead(QObject*)));
    5. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpFinished(QObject*)));
    To copy to clipboard, switch view to plain text mode 

    think a little bit how it works. when readyRead is emitted it calls map(), the same is done when finished is emitted. i guess, map slot emits mapped signal. you have two slots connected to the same signal (mapped), therefore they both will be invoked when any of the signals is emitted.

  15. The following user says thank you to mentalmushroom for this useful post:

    Alir3z4 (21st February 2012)

  16. #13
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Default Re: Concurrent file downloading

    Quote Originally Posted by mentalmushroom View Post
    Qt Code:
    1. connect(i.key(), SIGNAL(readyRead()), signalMapper, SLOT(map()));
    2. connect(i.key(), SIGNAL(finished()), signalMapper, SLOT(map()));
    3.  
    4. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpReadyRead(QObject*)));
    5. connect(signalMapper, SIGNAL(mapped(QObject*)), this, SLOT(httpFinished(QObject*)));
    To copy to clipboard, switch view to plain text mode 

    think a little bit how it works. when readyRead is emitted it calls map(), the same is done when finished is emitted. i guess, map slot emits mapped signal. you have two slots connected to the same signal (mapped), therefore they both will be invoked when any of the signals is emitted.
    Aha you are right, but i got your point, but i think using one instance of QSignalMapper in this situation is wrong!
    If i'm wrong correct me:i can use QSignalMapper for 1 SLOT which is associated to many signals, right ? // as i see the doc example about it!
    Last edited by Alir3z4; 21st February 2012 at 16:52.
    ...یه مرد هیچوقت زمین نمیخوره

  17. #14
    Join Date
    Feb 2011
    Posts
    354
    Thanks
    17
    Thanked 27 Times in 24 Posts
    Qt products
    Qt4 Qt5
    Platforms
    MacOS X Windows

    Default Re: Concurrent file downloading

    Yes, I think, you should use a different SignalMapper for the second signal.

    Technically you can connect several slots to one signal, also the same slot can be connected to several different signals, finally, you can connect the same slot to the same signal several times. Only you think whether this is what you want, because, I think, in your case you don't need this.

  18. The following user says thank you to mentalmushroom for this useful post:

    Alir3z4 (21st February 2012)

  19. #15
    Join Date
    Jun 2011
    Posts
    69
    Thanks
    13
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11

    Default Re: Concurrent file downloading

    Quote Originally Posted by mentalmushroom View Post
    Yes, I think, you should use a different SignalMapper for the second signal.
    That's what i posted http://www.qtcentre.org/threads/4754...502#post214502
    And you really assure me about it
    Thanks
    ...یه مرد هیچوقت زمین نمیخوره

Similar Threads

  1. QWebView page loading fails after file downloading
    By mentalmushroom in forum Qt Programming
    Replies: 1
    Last Post: 18th August 2011, 15:57
  2. downloading the file problems
    By migel in forum Newbie
    Replies: 0
    Last Post: 7th June 2011, 18:30
  3. Playing file with Phonon while stil downloading it.
    By alexandernst in forum Qt Programming
    Replies: 3
    Last Post: 10th April 2011, 12:25
  4. Downloading file over https with proxy
    By szraf in forum Qt Programming
    Replies: 0
    Last Post: 5th April 2011, 17:56
  5. File size of a remote file without downloading it
    By dirkdepauw in forum Qt Programming
    Replies: 5
    Last Post: 4th November 2010, 10:48

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.