Many Many (most of the time sleeping) threads. Best approach?

[ander.pijoan]

Hi all,

I've been looking for a similar question but I haven't found any (maybe because this is not possible).

I am building an application that would simulate different entities executing concurrently (vehicles, people, etc). I would like each of them to have its own thread and to execute paralelly. Although being quite a big amount of elements (imagine 10000) most of the time their threads would be sleeping. I don't know if tackling this problem with QThreads is not the right way but currently I have tried different implementations:



Creating all entities as QRunnable and running them all paralelly at the same time:
I get a "Cannot create pipe main loop wake-up: too many files open" error, for running too many threads.


Creating all entities as QRunnable and calling the QThreadPool to run them :
The threadpool executes the number of threads defined by QThreadPool::globalInstance()->setMaxThreadCount() (let's say 28) and queues the rest. But even if the QRunnable attached to one of this 28 threads is sleeping, the thread wont be released and used by another queued entity (so in the end only 28 entities would be executed).


I've been trying today QtConcurrent::run() but for what I've tried it seems to work in the same way QThreadPool does



Is there any way to implement this? If QThreadPool would detach sleeping threads and run the queued awaiting ones all my problems would be fixed. Or am I approaching the problem completely wrong?

Thanks in advance for your time.

[wysota]

Thread pool is not a good approach as the number of threads in the pool is limited. Depending what your agents do you should either subclass QThread and implement each agent as a thread if the agent's code is synchronous or subclass QObject, implement each agent in that class and make them event-driven. Then you can either run all of them in a single thread or spawn a number of extra threads and move agent objects to different threads.

[ander.pijoan]

Thread pool is not a good approach as the number of threads in the pool is limited. Depending what your agents do you should either subclass QThread and implement each agent as a thread if the agent's code is synchronous

Doing so wouldn't there be the same problem as I had when running all QRunnables together?

[wysota]

Doing so wouldn't there be the same problem as I had when running all QRunnables together?

Could be. Having 1000 threads in an application is rarely a good option :)

[yeye_olive]

If you go for the synchronous approach, make sure to adjust the stack size of each QThread to a reasonable value. Your agents probably need far less stack space than the comfortable default value.

[ander.pijoan]

Or subclass QObject, implement each agent in that class and make them event-driven. Then you can either run all of them in a single thread or spawn a number of extra threads and move agent objects to different threads.

So for implementing this approach, you mean for example:

- Implementing the agents just as QObject.
- Add a battery of signal and slots for it to use by itself.
- Once started instead of sleeping, use a timer to invoke later a signal of its own?

Would this 10000 timers waiting for certain time to pass be a problem?

[wysota]

- Once started instead of sleeping, use a timer to invoke later a signal of its own?
Either that or rely on events (if the agent should respond upon some other agent's actions).


Would this 10000 timers waiting for certain time to pass be a problem?
No but I wouldn't do that. It's better that the object do "nothing". You can have one timer and connect every agent to that timer to provide a kind of "heartbeat" impulse.

[ander.pijoan]

Worked really well your proposed solution, thanks again :)