Joe, you're coming into this with wrong expectations.
By default Photosynth attempts to figure out the positions of all of your photos relative to each other. If it can't discover how some fit into the rest, it puts them at the end of the list, regardless of the name of the file, when they were taken, or what order they were placed into the synther.
Of the photos that *do* link together (in your case 328 of 337) Photosynth cares about moving through them spatially, rather than chronologically. It assumes that someone may be making a synth from photos taken at different times or by different people or that the person shooting may have taken photos of one part of the area, moved on, and then come back to another part for a bit more coverage. Again, it initially only cares about moving through the photos from one image to its nearest neighbor, etc., rather than what order they were taken in.