Great suggestion. :) I've thought the same thing on several occasions.
I'm not sure that it makes sense all the time, so the question then becomes "When do we try to stay the same distance and when to we float to the borders of each photo?".
The problem becomes even more tricky when you consider cases besides simple panoramas or perfectly round orbits of photos around an object, for example photos from different distances all around the outside of a building. Imagine that you've zoomed in on a particular window in your house and you expect to stay centred on it as you shift between different photos. Is that really what you want when you come to the really wide shots and that window is composed of very few pixels? It might be difficult for people to realise that they needed to zoom out, as it would resemble the beginning stages of any new photo filling in from low resolution to high resolution.
I'm interested in any further ideas you've got.