This has been discussed at some length at getsatisfaction.
MathMaestro, giving Photosynth too many images (even if they are the low resolution stills that a video yields) almost always gives poor results. A more likely solution is if we are given the ability to tag specific parts of individual photos to point out common objects in photos that fail to synth.
Fred, it's worth noting that if your images' filenames are such that the photos taken earlier come earlier when sorted by filename then you can step through a synth where time is important (a sunset synth) in correct order by using the fullstop/period key to advance and the comma key to step backwards.
The truth is that with all the EXIF data that Photosynth reads it should, by all rights, read the date and time that a photo was taken and give us a way to step through the photos in true order, rather than simply filename order. This is an important part of photography and Photosynth should recognise that.