Ok so I finally found the solution to my problem.

Since I didn't want to use the ScrollHandDrag mode I always got the problem that my mouse events were a bit "off". Now I discovered why: the function scrollContentsBy in qgraphicsview always re-sends the last mouse move event. Since I always saved my current/last mouse position in the mouse move event and calculated the difference and used that as the amount I got wrong values.

My solution: before calling my zoom or translate function call setInteractive(false). This prevents the scrollContentsBy function of re-sending the event. Afterwards set it back to true.