QNetworkAccessManager and multiple simultaneous requests

I need to connect to multiple sites (variable number of sites, may be over 100) and fetch some data in a loop for each site.

I create a thread and a QNAM for each site. (For each site QNAM must have different cookies).

All good for few sites, but when sites number is big I get

"Too many open pipes"

and crash.

Then I've tried using a single thread but is too slow because it won't fetch data from next site until current site reply is finished.

There is some way to limit the number of simultaneous connection to 15 - 20 while the others waiting, and when there is a free slot open connection, do stuff and close connection so that the other threads can do their job ?

Quote:

---

Originally Posted by Angelo Cruz

I create a thread and a QNAM for each site. (For each site QNAM must have different cookies).

---

Different cookies as in "the same non-site specific cookie but each site should get a different value"?

Quote:

---

Originally Posted by Angelo Cruz

Then I've tried using a single thread but is too slow because it won't fetch data from next site until current site reply is finished.

---

You mean one thread and still one QNAM per site?
Even one QNAM can handle multiple connections simultaniously, so maybe you are blocking the processing somehow?

Cheers,
_

Thank you for your reply!

First I login to those sites and save the cookies.

When I use one thread I use one QNAM : I set the received cookies from login (otherwise the site won't accept my requests) , I use a QEventLoop to wait until I get the reply, then I set the QNAM with the cookies from the next site and so on!

What I'm thinking is to use some kind of global lock with 20 items, each site in his own thread, each thread check if one of 20 slots is free if find a free slot open new QNAM do his job then close QNAM and make free that slot. If no slot is free sleep 200 ms until a slot is freed. (or something like that).

That sounds overly complicated.

Can't you set all cookies and then just make your requests and leave the distribution of the jobs to QNAM?

Cheers,
_

Did you try this first w/o threading and forcing synchronous behavior? What issues did you run into with the asynchronous approach?