Borrow-checker hindered me to keep the same HashSet around all the time
and just clear it, so it feels a little inefficient to have a new set
every time.
I feel like there should be away for me to read from the back and move
things over directly instead of having a temporary Vec for it, but I'm
not sure how this would work with the owner-checking, so I didn't try.
I'm not sure why this is so much to write. It feels like there should be
a more generic way to check this instead of duplicating those lines, but
maybe it's not worth it for 2 ranges ...