I doubt this is related to Qt and your attempt with system() seems to confirm that.

Maybe there is some system setting that limits the amount of memory a process group can allocate or that somehow restricts child processes.

Might be helpful to check on a Windows forum.

Cheers,
_