I'm already using the destroyed signal (calling beginRemoveRows / endRemoveRows from there) the invalid objects are accessed on beginInsertRows when something is added after something else got deleted.
I tried using reset model but that collapses all tree items in the view which is annoying.

At the moment I'm using a dirty work-around that keeps all pointers in a set and add/removes them recursively when something changes, then checking for validity to avoid segmentation faults.
But I still haven't figured why Qt is sending me invalid indexes from rows that have already been deleted.