Hi, Cenobyte, the Silverlight 4 Photosynth viewer works in all four of the browsers you mentioned above on Windows as well as Safari, Firefox, and Chrome on Mac OSX.
If you are only trying to launch the synther (the Photosynth app for Windows that does the computer vision and the uploading to your account) it is a separate application from the browser and can be launched from any browser or from a shortcut icon or the Start menu.
To answer your question, no, Photosynth doesn't work only in Internet Explorer. The viewer should work in any browser that Silverlight 4 (or Moonlight 4 on Linux) runs in. If you're still having problems with the viewer, just be sure that you haven't blocked Silverlight from running in your other browsers.