I actually did try something similar with VE3D, and tried to incorporate regular images as well. It works relatively well.
http://photosynth.net/view.aspx?cid=a65da837-270c-4fd5-b487-08875dddf92b
Pretty cool! Why don't you do this with a spherical world?