Note the existence of two different concepts: selection and "current item". These are independent.
The selection is what you know (blue background), the current is a rather hard to see dotted rectangle.

Clicking selects the item (and also makes it current). Note that often the whole row is selected, but only one cell of it is current.

With the arrow keys you usually move the current item, not the selection.


If you insist on being able to do an extended selection, there is no way around that behaviour: how would you select rows 1 and 3 (with keyboard only) if hitting arrow keys moved the selection=?
If single selection should suffice, try QAbstractItemView::SingleSelection as selection mode. Maybe current and selection are held in sync then.

HTH