The Silverlight viewer can't make much use of your GPU so has to do all 3D calculations on your CPU, which does take some power.
The older D3D viewer uses your GPU, so is both faster and will use less of your CPU cycles.
Try the Direct3D viewer if you have:
2) Photosynth installed
3) Updated video drivers
4) Firefox 3 or Internet Explorer. (Firefox 4 Beta breaks the D3D viewer.)
The link to the Direct3D viewer is on the bottom left corner of each synth 'view' page. Alternately, just replace 'view.aspx' in any synth URL for 'd3d/photosynth.aspx'
Note that the old D3D plugin will not work in Opera, Safari, or Chrome. Also note that it's just for synths: not panos.
On the CPU-munching bug-reporting side of things, though, it's been my experience that both viewers will eat your CPU cycles when you leave them in a tab and look at something in another tab. I'm not sure why that is, but they both calm down once you look back at their tab.