The hope, really, is that Silverlight 4 offers, unless I am misunderstanding things rather badly, access to the full hardware of the computer (this would mean your video card, most importantly, which frees up your CPU as it simultaneously performs graphics tasks faster) IF you run your Silverlight 4 app as an out of the browser app with full trust privileges.
The big question, then, is, "Can you run that same app (once you have verified that you trust it out of the browser) inside the browser with that same full blown GPU acceleration?". If so, then the Photosynth viewer built in Silverlight 4 is set to be the one that brings us back to Direct3D performance levels. (Someone on the PS team shoot this idea down ASAP if I've got this wrong.)
If the above is correct, then you have to ask yourself if the 3D Virtual Earth control might not be ported to Silverlight 4 and at that point you wonder why Worldwide Telescope wouldn't join the party.