which transmit approximately 4 gigabytes of data per second.
These kinds of data rates are not uncommon in some areas of scientific data acquisition, too. In typical applications I am aware of, there is a heavy amount of signal averaging that occurs between receiving the data from the detector and displaying it or saving it to disk. GPUs are great at this, and the first rule of GPU processing is get the data onto the GPU and never move it back - data transfers between GPU and CPU are the slowest part of CPU/GPU applications. So this implies that you want to use GPU-based graphics (eg. OpenGL) if possible.

The unfortunate thing about Qt 3d is that it seems to be designed mostly for use for games or similar things requiring photo-realistic scene rendering. No one has yet written any kind of 2- or 3-D data plotting layer for it.

You might want to look at something like VTK. It also has a steep learning curve, but it is designed for scientific data visualization and does have a Qt wrapper so you can embed VTK graphics into a Qt GUI.