Not getting all data from database through QNetworkAccessManager.

[adutzu89]

I am trying to download all rows from a mysql table into sqlite using QNetworkAccessManager but it will not retrieve full data, more worse it retrieves random number of rows(134,135,220, 1409) but never full rows which in that tables are aprox. 4000;

Url setting:

void sinc::startNetworkRequest(){
QNetworkAccessManager *manager=new QNetworkAccessManager(this);
QNetworkRequest cerere;
QNetworkReply *rasp;
 
cerere.setUrl(QUrl("http://webserverurl.php?param1=01&param2=01"));
rasp=manager->get(cerere);
connect(rasp,SIGNAL(readyRead()),this,SLOT(insertToTable()));
connect(rasp,SIGNAL(error(QNetworkReply::NetworkError)),this,SLOT(slotEroare(QNetworkReply::NetworkError)));
}

void sinc::startNetworkRequest(){
QNetworkAccessManager *manager=new QNetworkAccessManager(this);
QNetworkRequest cerere;
QNetworkReply *rasp;

cerere.setUrl(QUrl("http://webserverurl.php?param1=01&param2=01"));
rasp=manager->get(cerere);
connect(rasp,SIGNAL(readyRead()),this,SLOT(insertToTable()));
connect(rasp,SIGNAL(error(QNetworkReply::NetworkError)),this,SLOT(slotEroare(QNetworkReply::NetworkError)));
}

To copy to clipboard, switch view to plain text mode 

Data retrieving slot:

void sinc::insertToTable(){
    QByteArray baRasp=rasp->readAll();
    QString str(baRasp);
    QStringList listaDet,listaElemR;
    listaDet=str.split(";");
    QSqlQuery curatare,adaug;
 
    curatare.exec("delete from destTable");
    for(int i=0; i<listaDet.size()-1; i++){
        listaElemR=listaDet[i].split(",");
        adaug.exec("insert into destTable values ("
                   "'"+listaElemR[0]+"'"
                ",'"+listaElemR[1]+"'"
                ",'"+listaElemR[2]+"'"
                ",'"+listaElemR[3]+"'"
                ",'"+listaElemR[4]+"'"
                ",'"+listaElemR[5]+"'"
                ",'"+listaElemR[6]+"'"
                ",'"+listaElemR[7]+"'"
                ",'"+listaElemR[8]+"'"
                ",'"+listaElemR[9]+"'"
                ",'"+listaElemR[10]+"'"
                ",'"+listaElemR[11]+"'"
                ",'"+listaElemR[12]+"'"
                ",'"+listaElemR[13]+"'"
                ",'"+listaElemR[14]+"'"
                ")");
    }
}

void sinc::insertToTable(){
    QByteArray baRasp=rasp->readAll();
    QString str(baRasp);
    QStringList listaDet,listaElemR;
    listaDet=str.split(";");
    QSqlQuery curatare,adaug;

    curatare.exec("delete from destTable");
    for(int i=0; i<listaDet.size()-1; i++){
        listaElemR=listaDet[i].split(",");
        adaug.exec("insert into destTable values ("
                   "'"+listaElemR[0]+"'"
                ",'"+listaElemR[1]+"'"
                ",'"+listaElemR[2]+"'"
                ",'"+listaElemR[3]+"'"
                ",'"+listaElemR[4]+"'"
                ",'"+listaElemR[5]+"'"
                ",'"+listaElemR[6]+"'"
                ",'"+listaElemR[7]+"'"
                ",'"+listaElemR[8]+"'"
                ",'"+listaElemR[9]+"'"
                ",'"+listaElemR[10]+"'"
                ",'"+listaElemR[11]+"'"
                ",'"+listaElemR[12]+"'"
                ",'"+listaElemR[13]+"'"
                ",'"+listaElemR[14]+"'"
                ")");
    }
}

To copy to clipboard, switch view to plain text mode 

If I want to retrieve the number of rows from the webserver( through echo mysqli_num_rows(result)) I get the full number rows, so it isn't mysql's,php's or webserver's fault.

[Lesiok]

How many times You service readyRead signal ? You should read data until signal finished is emitted.

[adutzu89]

Well the only time I post it.
So I should have 2 slots, 1 for reading the data(connected to readyRead()) and 1 for inserting the data read(connected to finished())?

---

Added after 59 minutes:

I've connected the slot to the finished() signal, and gives all rows.

[adutzu89]

Strangely if I do a count() in sqlite table the number of rows shown is a bit smaller then the rows got, but If I put a

qDebug()<<QString::number(i)<<" "<<listaElemR[0]<<endl;

qDebug()<<QString::number(i)<<" "<<listaElemR[0]<<endl;

To copy to clipboard, switch view to plain text mode 

in the for(int i=0; i<listaDet.size()-1; i++) block it goes all the way to the last line.

Example:
listaDet.size()-1=4463
select count(*) from destTable;=4453

[Lesiok]

What is a value returned from readAll() after signal finished ?

[adutzu89]

I do not understand your question.
If your talking "what is the value returned from readAll()", in my question I've post how I get the value.

QByteArray baRasp=rasp->readAll();
QString str(baRasp);
QStringList listaDet;
listaDet=str.split(";");

QByteArray baRasp=rasp->readAll();
QString str(baRasp);
QStringList listaDet;
listaDet=str.split(";");

To copy to clipboard, switch view to plain text mode 

qDebug()<<QString::number(listaDet.size())<<endl;  //this output's in the console 4464(meaning there should be 4463 in sqlite db)

qDebug()<<QString::number(listaDet.size())<<endl;  //this output's in the console 4464(meaning there should be 4463 in sqlite db)

To copy to clipboard, switch view to plain text mode 

[Lesiok]

I know this. I want to see what is in variable str - before splitting. I think that there is something more than the data from the database.

[adutzu89]

Oh, ok
Well then I will post the webserver's php code(example on how I get the data from mysql database and sending it):

$queryStr="select * from dbTable";
$query=mysqli_query($link,$queryStr);
while($r=mysqli_fetch_assoc($query)){
        $v1=$r['column1'];
        $v2=$r['column2'];
        $v3=$r['column3'];
        $v4=$r['column4'];
 
        $r.=$v1,$v2,$v3,$v4;";
    }
 
echo $r;

$queryStr="select * from dbTable";
$query=mysqli_query($link,$queryStr);
while($r=mysqli_fetch_assoc($query)){
        $v1=$r['column1'];
        $v2=$r['column2'];
        $v3=$r['column3'];
        $v4=$r['column4'];

        $r.=$v1,$v2,$v3,$v4;";
    }

echo $r;

To copy to clipboard, switch view to plain text mode 

[Lesiok]

Dear buddy, I do not care how answer is prepared only what it contains.

[adutzu89]

This is a small portion of the str's output in the console.

qDebug()<<str;

qDebug()<<str;

To copy to clipboard, switch view to plain text mode 

1,rem impex srl engross-1629670,rem impex srl engross-1629670, ,0,21,romania,galati,galati,galati unic stand 12 j17/1151/91,,
, , ,0,,ro1629670, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;2,bartel coscs-gl-2480347,bartel coscs-gl-2480347, ,0,21,romania,galati,galati,galati centru langa modern,,
, , ,0,,r2480347, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;3,mini magic srl gl,mini magic srl gl, ,0,21,romania,galati,galati,galati centru linga patis cernat,,
, , ,0,,, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;4,farm exprofesso srl gl-1628810,farm exprofesso srl gl-1628810, ,0,21,romania,galati,galati,galati,,
, , ,0,,1628810, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;5,farm patis cernat srl gl-5768396,farm patis cernat srl gl-5768396, ,0,21,romania,galati,galati,galati,j17/1501/1994,str.otelarilor,bl.d11a,
, , ,0,,r5768396, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;6,negro srl gl,negro srl gl, ,0,21,romania,galati,galati,galati,,
, , ,0,,, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;7,vipem srl gl aprozar-1641801,vipem srl gl aprozar-1641801, ,0,21,romania,galati,galati,galati aprozar,,
, , ,0,,r1641801, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;8,farm tink-baivaron srl gl,farm tink-baivaron srl gl, ,0,21,romania,galati,galati,galati str mihai bravu nr 4,,
, , ,0,,ro9762523, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;9,rem impex srl gl-1629670,rem impex srl gl-1629670, ,0,21,romania,galati,galati,galati micro 39 magazin,,
, , ,0,,ro1629670, ,0,1, ,0000-00-00 00:00:00,2013-11-23 13:34:28;

[Lesiok]

Are You sure that semicolon appears only at the end of each record ? Maybe some data contains semicolon and then one record is divided into two parts.

[adutzu89]

I will try and verify directly in the database.

There is no record of when searched for ";" , but there are a lot which contain ",". I will try an change the delimiters to some which are almost never used.
I wil return to tell you my result.

Thank you

---

Added after 18 minutes:

Searched, changed delimiters and have the same end result.

I believe it's a sqlite bug, maybe as I get the same thing using delimiters which weren't use at all and got the same number of rows from webserver and same result when using count() on the sqlite table(meaning 4464 in stringlist , 4453 counted in sqlite after insertion), also the number of rows inserted in sqlite are in the following range 0-4463 and I see in console that the last row inserted into the table is counted as 4463.

[adutzu89]

I found why I had 4453 rows inserted in sqlite instead of 4464.........forgot to escape special characters and those rows that didn't insert cotained " ' " character.

[Lesiok]

Read about QSqlQuery::prepare and ? placeholders.

[adutzu89]

Yes, now I am inserting using prepare() and bindvalue() and all rows are inserted.