Results 1 to 13 of 13

Thread: QString static callback function from CURL

  1. #1
    Join Date
    Oct 2006
    Location
    Hawaii
    Posts
    130
    Thanks
    48
    Thanked 4 Times in 4 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default QString static callback function from CURL

    I am using LibCURL to pull in data from various web sites. I have made the callback function work in the following way (which works properly but has horrible performance):

    Qt Code:
    1. struct MemoryStruct {
    2. char *memory;
    3. size_t size;
    4. };
    5.  
    6. static size_t WriteMemoryCallback(void *ptr, size_t size, size_t nmemb, void *data) {
    7.  
    8. size_t realsize = size * nmemb;
    9. struct MemoryStruct *mem = (struct MemoryStruct *)data;
    10.  
    11. //increase the size of "memory" by the size of the bytes that we have read
    12. mem->memory = (char *)realloc(mem->memory, mem->size + realsize + 1);
    13.  
    14. if (mem->memory) {
    15.  
    16.  
    17. //copy to the end of the memory chunk, the piece of data that we read from curl (pointed to by ptr, size of realsize):
    18. memcpy(&(mem->memory[mem->size]), ptr, realsize);
    19.  
    20. //update the size of the memory chunk:
    21. mem->size += realsize;
    22.  
    23. //put a 0 at the end of the memory chunk
    24. mem->memory[mem->size] = 0;
    25. }
    26. return realsize;
    27. }
    28.  
    29.  
    30. void FabwareMain::curl_init() {
    31.  
    32. //init the memory to hold what we gather with CURL
    33. struct MemoryStruct chunk;
    34. chunk.memory=NULL; /* we expect realloc(NULL, size) to work */
    35. chunk.size = 0; /* no data at this point */
    36.  
    37. //init curl:
    38. curl_global_init( CURL_GLOBAL_ALL ) ;
    39.  
    40. //create a curl handle:
    41. CURL *curl_handle = curl_easy_init() ;
    42. //the login page:
    43.  
    44.  
    45. //set curl options:
    46.  
    47. curl_easy_setopt(curl_handle, CURLOPT_COOKIEFILE, "");
    48.  
    49. curl_easy_setopt(curl_handle, CURLOPT_FOLLOWLOCATION, 1); //when redirected, follow the redirections
    50. curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20060601 Firefox/2.0.0.3 (Ubuntu-edgy)");
    51. curl_easy_setopt(curl_handle, CURLOPT_POST, 1);
    52. curl_easy_setopt(curl_handle, CURLOPT_URL, "http://www.somewhere.com");
    53. curl_easy_setopt(curl_handle, CURLOPT_POSTFIELDS, "");
    54. curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);
    55. curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
    56. //exec curl
    57. curl_easy_perform(curl_handle);
    58.  
    59. //convert the memory chunk to a qstring
    60.  
    61. resData = QString::fromUtf8 (chunk.memory);
    62. ui.textEdit->append (resData);
    63.  
    64. //parse the qstring
    65.  
    66.  
    67.  
    68. //clean up after ourselves:
    69. curl_easy_cleanup(curl_handle);
    70. curl_global_cleanup();
    71.  
    72. //clean up our memory chunk
    73. if(chunk.memory) {
    74. free(chunk.memory);
    75. }
    76.  
    77. }
    To copy to clipboard, switch view to plain text mode 

    The problem with this method, is that although it works, it consumes a ton of memory and time due to copying the data many times. First CURL copies the data into a buffer, then the callback function copies it into the data chunk, then we convert it into a Qstring, then we copy that into the text edit box.

    I have found that I can modify the callback function to put the data from CURL into a QString as it comes in, and parse out what I need on the fly. I can not however figure out how to make the data that is in the callback function QString be available outside of the function. I can't return the data as CURL calls the function, so I will need to store it in a way that can be accessed from the callback function and the rest of the Qt program.

    Heres the current callback function that I am modifying:

    Qt Code:
    1. struct MemoryStruct {
    2. char *memory;
    3. size_t size;
    4. };
    5.  
    6. static size_t WriteMemoryCallback(void *ptr, size_t size, size_t nmemb, void *data) {
    7.  
    8. size_t realsize = size * nmemb;
    9. struct MemoryStruct *mem = (struct MemoryStruct *)data;
    10.  
    11. //increase the size of "memory" by the size of the bytes that we have read
    12. mem->memory = (char *)realloc(mem->memory, mem->size + realsize + 1);
    13.  
    14. if (mem->memory) {
    15.  
    16. QString qmemData = QString::fromUtf8 ((char *)ptr);
    17. //parse out the stuff that I want
    18. //How do I get this QString (qmemData) to be accessable outside of this function
    19.  
    20. }
    21. return realsize;
    22. }
    To copy to clipboard, switch view to plain text mode 

    How do I get the QString qmemData to be accessable outside of this function? Im sure its something simple I am missing somewhere.

  2. #2
    Join Date
    Feb 2006
    Location
    Romania
    Posts
    2,744
    Thanks
    8
    Thanked 541 Times in 521 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QString static callback function from CURL

    I don't know if it fits your need, but what do you think of a static member in your main GUI class?

    Actually it has to be a pointer QString *qmemData;

    You can use it anywhere, modify it's contents, and possibly set a flag in your class that the string has been modified and update the widgets.

    Regards

  3. #3
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QString static callback function from CURL

    Using a QByteArray for the buffer would be a good thing too

    BTW. Is the data writing here safe? You copy a pointer and not its contents. Who has the ownership of the pointer? The callback function or the caller? Because if the latter, then you may run into trouble.

  4. #4
    Join Date
    Oct 2006
    Location
    Hawaii
    Posts
    130
    Thanks
    48
    Thanked 4 Times in 4 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QString static callback function from CURL

    This piece of code tells the CURL library where to put what its gathering:
    Qt Code:
    1. curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);
    To copy to clipboard, switch view to plain text mode 
    This piece of code tells it the callback function to use:
    Qt Code:
    1. curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
    To copy to clipboard, switch view to plain text mode 
    Once you execute curl exec it does whatever the options tell it to do, and calls the callback function whenever the remote server sends back data. Im pretty sure its safe as copying the pointer is what the examples in the CURL install tarball show, and I haven't had any problems with it.

    The main problem I face is that I am pulling in several megabytes of data, which normally would append to the data already in memory from the request. Rather than waste performance on pulling all of that data into memory I wanted to be able to look at the chunks as they come in, pull out what I want out of them, and pretty much throw away the rest.

    Since the callback function is a static function, I am not sure how I can make the data that is processed in it available to my QT app outside of that function.

    I did try the static member approach as marcel suggested, which compiled fine, however the program crashed when it got to the point that it started using that variable. Do you have an example of how this would work?

  5. #5
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QString static callback function from CURL

    Quote Originally Posted by tpf80 View Post
    The main problem I face is that I am pulling in several megabytes of data, which normally would append to the data already in memory from the request. Rather than waste performance on pulling all of that data into memory I wanted to be able to look at the chunks as they come in, pull out what I want out of them, and pretty much throw away the rest.
    So why do you use realloc for that? And is using curl really necessary? Do you use some specific options that QHttp can't handle?

    Since the callback function is a static function, I am not sure how I can make the data that is processed in it available to my QT app outside of that function.
    Through global variables or static members (in a singleton class).

    I did try the static member approach as marcel suggested, which compiled fine, however the program crashed when it got to the point that it started using that variable. Do you have an example of how this would work?
    Something like this should work:
    Qt Code:
    1. class S {
    2. public:
    3. S *instance(){
    4. static S* inst = new S;
    5. return inst;
    6. }
    7. const char *getPtr() const { return S::staticPtr; }
    8. static void callBack(){
    9. //...
    10. staticPtr = ...;
    11. }
    12. private:
    13. static char *staticPtr;
    14. };
    15. char *S::staticPtr = 0;
    To copy to clipboard, switch view to plain text mode 

  6. The following user says thank you to wysota for this useful post:

    tpf80 (11th May 2007)

  7. #6
    Join Date
    May 2006
    Posts
    788
    Thanks
    49
    Thanked 48 Times in 46 Posts
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: QString static callback function from CURL

    You dont need callback if you set a time limit! on curl!


    If you need callback use http://doc.trolltech.com/4.0/qhttp.html

    inside ..
    http://sourceforge.net/projects/qtexcel-xslt/
    you find more sample from curl ...


    Qt Code:
    1. /* return file contenets as qstring from local or remote file ... like php*/
    2. QString curl_get_contents( QString fullFileName )
    3. {
    4. QString inside = "";
    5. qDebug() << "###1 curl_get_contents "<< fullFileName;
    6. QString xdcookiefile, xml_export_file, wwwnetfile;
    7. if (fullFileName.contains("https://", Qt::CaseInsensitive)) {
    8. return inside;
    9. }
    10. Init();
    11. qDebug() << "###2 curl_get_contents "<< fullFileName;
    12.  
    13. xdcookiefile = QString( "%1biscotti.html" ).arg( WORK_CACHEDIR );
    14. xml_export_file = QString( "%1_xml_export.xml" ).arg( WORK_CACHEDIR );
    15. wwwnetfile = QString( "%1wwwfull.html" ).arg( WORK_CACHEDIR );
    16.  
    17. if (IsNetFile( fullFileName ) )
    18. {
    19. qt_unlink(wwwnetfile);
    20. qDebug() << "###3 curl_get_contents "<< fullFileName;
    21. /*QString xgeturl = fullFileName; http://shop.ecoplanet.ch/info.php */
    22. QByteArray lop = wwwnetfile.toAscii();
    23. char *localfile = lop.data();
    24. QByteArray der = xdcookiefile.toAscii();
    25. char *xwcookiefile = der.data();
    26. QByteArray ba = fullFileName.toAscii();
    27. char *url = ba.data();
    28. qDebug() << "### entra save to "<< localfile;
    29. qDebug() << "### get url "<< url;
    30. CURL *curl_handle;
    31. FILE *outfile;
    32. curl_global_init(CURL_GLOBAL_ALL);
    33. curl_global_init(CURL_GLOBAL_ALL);
    34. curl_handle = curl_easy_init();
    35. outfile = fopen(localfile, "w");
    36.  
    37. if (outfile!=NULL) {
    38. /* CURLOPT_COOKIE and must have CURLOPT_COOKIEJAR char * (file to write same as php) */
    39. curl_easy_setopt(curl_handle, CURLOPT_URL, url);
    40. curl_easy_setopt(curl_handle, CURLOPT_VERBOSE, TRUE);
    41. curl_easy_setopt(curl_handle, CURLOPT_COOKIEJAR, xwcookiefile );
    42. curl_easy_setopt(curl_handle, CURLOPT_COOKIEFILE, xwcookiefile );
    43. curl_easy_setopt(curl_handle, CURLOPT_FOLLOWLOCATION , 1);
    44. #if defined Q_WS_MAC
    45. curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Mac QT4 / PPK_W @ciz.ch" );
    46. #endif
    47. #if defined(Q_WS_WIN)
    48. curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Windows QT4 / PPK_W @ciz.ch" );
    49. #endif
    50. curl_easy_setopt(curl_handle, CURLOPT_MAXREDIRS , 2);
    51. curl_easy_setopt(curl_handle, CURLOPT_TIMEOUT , 20 );
    52. curl_easy_setopt(curl_handle, CURLOPT_FILE, outfile);
    53.  
    54. if (curl_easy_perform(curl_handle)==CURLE_OK) {
    55. fclose(outfile);
    56.  
    57. if (is_file(wwwnetfile)) {
    58. inside = ReadFile( wwwnetfile );
    59. if (inside.size() > 1) {
    60. ERROR_MSG = "";
    61. #if defined Q_WORKS_PEND
    62. /* not delete file */
    63. #else
    64. qt_unlink(wwwnetfile);
    65. #endif
    66. }
    67. }
    68.  
    69. } else {
    70. ERROR_MSG = "Error time out to get remote file";
    71. }
    72. }
    73. /* return grab result from local file */
    74. return inside;
    75. }
    76.  
    77.  
    78.  
    79.  
    80. /* ok is a fake normal local file init ..... */
    81. inside = ReadFile( fullFileName );
    82. return inside;
    83. }
    To copy to clipboard, switch view to plain text mode 

  8. #7
    Join Date
    Oct 2006
    Location
    Hawaii
    Posts
    130
    Thanks
    48
    Thanked 4 Times in 4 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QString static callback function from CURL

    So why do you use realloc for that? And is using curl really necessary? Do you use some specific options that QHttp can't handle?
    I used that in the first place because thats what the example from CURL themselves used. For my purposes, since I don't need the whole data chunk that is pulled in, I've removed that part and use pretty much this for making the data within the callback usable:

    Qt Code:
    1. resData = QString::fromUtf8 ((char *)ptr);
    To copy to clipboard, switch view to plain text mode 

    As for why I was using CURL instead of QHttp, It was mainly because I didn't see QHttp supporting HTTPS (at least in the free edition that I have at the moment) and I was already pretty familiar with CURL's functionality as a standalone program and using it in PHP. This is my first attempt at using it with C++ though.

    My problem is probably related to my unfamiliarity to using a variable in the way I need to. The static member code that you posted looks very close to what I had, I'm taking a look at it and see what I was missing.

  9. #8
    Join Date
    Oct 2006
    Location
    Hawaii
    Posts
    130
    Thanks
    48
    Thanked 4 Times in 4 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QString static callback function from CURL

    Actually, I found out that I might not need https for what I am doing. I think I'll try out QHttp and see if that doesn't solve my problem, rather than making a mess of my program trying to force it to work with curl.

  10. #9
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QString static callback function from CURL

    Quote Originally Posted by tpf80 View Post
    As for why I was using CURL instead of QHttp, It was mainly because I didn't see QHttp supporting HTTPS (at least in the free edition that I have at the moment)
    Qt 4.3+ supports SSL sockets. The release candidate of Qt 4.3 is already available at TT site.

  11. #10
    Join Date
    Oct 2006
    Location
    Hawaii
    Posts
    130
    Thanks
    48
    Thanked 4 Times in 4 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QString static callback function from CURL

    I also notice that QHttp doesn't really have a good way to do cookies like curl does.
    It seems like In order to keep my session, I would need to extract the cookie after each request, and then add it to the next one in a semi-manual fashion.

    When I use curl, I just can issue a command which creates a "cookiejar" in memory when i first start it up and handles preserving the cookies itself.

    So I guess I kindof have to decide what I want to do:

    1) use CURL and figure out this static member variable stuff to get the data I want out (since I already have curl logging in, doing cookies for the session, etc perfectly already)

    or..

    2) rewrite code in QHTTP which would feel cleaner but would have to worry about a few more things like the cookies myself.

  12. #11
    Join Date
    Oct 2006
    Location
    Hawaii
    Posts
    130
    Thanks
    48
    Thanked 4 Times in 4 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QString static callback function from CURL

    Quote Originally Posted by patrik08 View Post
    You dont need callback if you set a time limit! on curl!


    If you need callback use http://doc.trolltech.com/4.0/qhttp.html

    inside ..
    http://sourceforge.net/projects/qtexcel-xslt/
    you find more sample from curl ...

    In the code that you wrote it looks like curl is writing to a file. How would I use it without a callback function if I wanted to pull the data into memory only and not write anything to a file?

  13. #12
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QString static callback function from CURL

    Quote Originally Posted by tpf80 View Post
    I also notice that QHttp doesn't really have a good way to do cookies like curl does.
    It seems like In order to keep my session, I would need to extract the cookie after each request, and then add it to the next one in a semi-manual fashion.
    Not exactly (I use a base request header object where I set the cookie, so that I don't have to do it on every request), but in general you're correct.

    When I use curl, I just can issue a command which creates a "cookiejar" in memory when i first start it up and handles preserving the cookies itself.
    Curl is definitely a higher level tool than QHttp so it has much much more capabilities (QHttp is really only an interface to the protocol), but on the other hand it'd be much easier to parse the result using QHttp.

  14. #13
    Join Date
    Oct 2006
    Location
    Hawaii
    Posts
    130
    Thanks
    48
    Thanked 4 Times in 4 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Talking Re: QString static callback function from CURL

    Thanks to all that replied, in the end I used libcurl for what I was doing with the HTTP stuff, and got it to work real well in fact.

    What I ended up doing was using the std string/char functions in the callback function, so that it would take out what it needed on the fly and write it to the memory chunk. Since in the callback its using those, it made things easier and have much better performance.

    In the main Qt program, I used the QString, by appending the returned data chunk via QString::fromUtf8. I could then do with it what I wanted to.


Similar Threads

  1. Replies: 16
    Last Post: 23rd May 2008, 11:12
  2. KDE/QWT doubt on debian sarge
    By hildebrand in forum KDE Forum
    Replies: 13
    Last Post: 25th April 2007, 07:13
  3. Convert from iso-8859-1 to... Something else :-)
    By Nyphel in forum Qt Programming
    Replies: 4
    Last Post: 7th March 2007, 18:59
  4. use qpsql
    By raphaelf in forum Installation and Deployment
    Replies: 34
    Last Post: 22nd August 2006, 13:52
  5. I got two problems when I used static compiled library of QT4
    By qintm in forum Installation and Deployment
    Replies: 8
    Last Post: 20th April 2006, 09:52

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.