I can't find any way to delete a panorama from the device. Any suggestion?

Touch catalog icon @ bottom left. Now touch EDIT @ top right. Now an X appears on each thumbnail. Simply touch to delete.

That will delete them when stored on the phone iteself. What about the ones that are underneath under the photosynth.net header?

I can't see how to delete them from the phone or on this website as it doesn't show up under My Photosynths

Scratch that comment. I was logged in using a different user account - doh!

Click on edit to name the untitled Photosynth.  Once you name the Photosynth select edit a second time.  This time an X will show up over the Photosynth and you will be able to delete it.