I doubt this is related to Qt and your attempt with system() seems to confirm that.
Maybe there is some system setting that limits the amount of memory a process group can allocate or that somehow restricts child processes.
Might be helpful to check on a Windows forum.
Cheers,
_
Bookmarks