From a strictly practical standpoint, video has a few great uses in the current version of Photosynth. The most compelling use is the "Rotational Move" that results in the donut-like functional button in the Photosynth viewer. Using videos to circle a single object greatly increases the fluidity of the motion and naturally creates a path of images for the rotation to follow. So long that a circular arch was show around a set point, Photosynth should have an easier time following a handheld video arch than a shot-by-shot series of single-shot guesses.
I recommend manually setting cameras to "shoot fast" when capturing such videos where the camera actually moves because otherwise the shots will be blurred. Consumer video cameras also lag far behind SLRs' many interchangable lenses, making matters even moredifficult. Furthermore, I expect that few people use video decompilers often. Nonetheless, with recent advances in HD Camera tech, I want to see testing.