QGraphicsView::scale() is absurdly slow over a remote X display?

I am fairly new to Qt, so I may be missing something obvious here, but I've looked around for a while and haven't had any luck.

Anyway, in a nutshell, I'm working on making a scrolling spectrogram display. It receives audio data in realtime and plots it on a time vs. frequency graph, and old data scrolls off of the graph to the right. The incoming data is fairly high-resolution, and I would like to keep it that way so that the user can zoom in to a specific location; however, by default I want the panel to scale the image data to fit the boundaries of the view on the frequency axis.

To do this, as audio arrives I process it and turn it into high-resolution QImages and add them to my QGraphicsScene, then I figure out the ratio between the QGraphicsView and the QImages and call QGraphicsView::scale() to make the images fit the view. This works great on a local display.

The problem is that this application will need to run on a remote Linux server and display on a local client. In my testing, I'm using an X11 connection forwarded through an SSH tunnel. When I call QGraphicsView::scale(), the display is very, very slow. It takes several seconds between screen updates, even over a LAN connection. If I remove the call to QGraphicsView::scale() so that everything is drawn at a 1:1 ratio, the display is very responsive. My theory is that the scale operation must be reading data from the client's graphic buffer, sending it back to the server, then performing the transformation and sending it back to the client, but I really don't know for sure.

Is there any way to make this faster? I'd like for the scale operation to be done entirely sever-side; this isn't an intensive enough operation that OpenGL accelerated rendering is necessary. I can scale the QImages before I add them to the scene, but this is kind of a pain because I need to keep the original high-resolution images around in case the user later wants to zoom in, and if they zoomed in I'd have to dump the entire scene and generate new images scaled from the originals as opposed to just applying a new scale to the QGraphicsView. It would be much nicer if there was some flag I could set on the QGraphicsView or QGraphicsScene to make them do all the processing server-side, but I've read over the docs and nothing sticks out at me...