Many Photosynth users are, like you, unhappy with the speed/performance of the current Silverlight viewer. It boils down to Silverlight 4 (and earlier versions) not having support for GPU accelerated graphics. Silverlight 5 will have such support but it does not release until fall of 2011.
The Silverlight viewer was important so that more people could view Photosynths in more browsers + with Macs + could do so potentially without needing to install Silverlight just for Photosynth, as they may have installed it for Netflix, the Olympics, or some other site.
The Photosynth team does acknowledge that the Silverlight viewer is not as fast as the Direct3D viewer for the time being, and so as a concession makes the Direct3D viewer available for every synth via a link on the lower left corner of the page, but take note that unlike the Silverlight viewer which works in all five major browsers on Windows, and the top four on Mac, D3D needs Firefox 3 or IE.