The permission/opt in idea is a good one, and the ability for others to email each to say, "hey I got some great shots! you want to share synths?" Another option is if Photosynth checks in the background that other people in the area have compatible 3d-ness for a combined point cloud, Photosynth could put a reminder on that users page that says "Your synth is compatible with others! Do you want to share?"
Its not all or nothing idea(i'm no programmer, so i dont know), just try a popular area and selectively ask others, maybe based on synth compatibility if they want to join.
This probably is a massive # crunch for Phtosynth and I can see why it hasn't been implemented.
How do display everyone's pictures of times square, you can't show them all! Maybe a Photosynth algorithm based on contrast/sharpness, then start an up vote option to pare the pics down to the best, with an option of the user pics promninent and other users for fill in for transitions.