Forum : Technical Preview

Get all your questions answered about our latest Photosynth Technical Preview.

Topic: spherical panoramas

Report Abuse Report Abuse
This is a feature we often use in conversations here, but its priority has not yet been discussed yet. Maybe it is taken as going without saying that it has high priority. I see it as a key experience feature, that consequently should have high priority. Any supporters and reasoning?
NateLawrence (4 months ago)
I'd love to hear the Photosynth team's thoughts on this as well.

Joscelin, I'm not sure if you're asking about:
:: standalone spherical panoramas, 
:: or multiple spherical panoramas being able to be combined in a single synth 
(nothing has ever been promised from the team in this direction although user requests for it have been consistent from the beginning) 
:: or perhaps you're just asking about the transition experience between individual spherical panos which have been individually uploaded by someone all densely blanketing the same area.

The Photosynth folks did say some time ago, "We definitely are planning on fully supporting arbitrary resolution stitched panos in WebGL.", so there's no reason to think that individual panos will be anything less than a first class citizen in the new site and apps but the real question for me is whether we'll see them generate and|or borrow any geometry for them from neighboring panos and synths when transitioning.
No, I am not talking about stitched panos, I am talking about Ps2 panos, standalone ones to start with.

If I am correct, current Ps2 synthing of panos supports cylindrical panos, but no spherical ones. Frames should be in a horizontal plane, because currently Ps2 cannot resolve to a proper auto view path.

I would say, simply take the vertical middle. That would be start at least. I feel there is something to say for Googles urge to shoot spherical as good as one can, it does contribute to the viewing experience more than average.
NateLawrence (4 months ago)
Ah. So spherical panning synths. 
Just to understand each other's terms, when I say "spherical panorama" in other threads, I'm thinking of stitched panoramas.
I thought that Andre had an interesting step toward breaking the current limitations with his fisheye lens panning synths but I'll be the first to admit that the lack of ability to actually drag the view up and down while panning hurts the effect even if you still see quite far both up and down.

I too look forward to seeing support for synth types to be opened up to "2-dimensional navigation (as opposed to the 1D nav we currently have in the new technology)" (as the Photosynth team phrases it).

For Panning synths, that would mean the ability to shoot either multiple rows or multiple columns to cover a full sphere. 
(Note that even today, I believe there's nothing stopping you from shooting a single vertical loop/column panning synth.)
NateLawrence (4 months ago)
For Spins, it would mean the ability to take a sphere of photos around something (again - multiple rows or columns).

For Walks, it would mean being able to take an entire grid of photos (say, at least one photo pointed North, South, East, and West from each position in multiple rows/columns on the ground) throughout a room or field and choose to move forward, backward, left, or right and turn in each direction.

For walls, it would mean being able to take close ups of an entire facade or super high res shots of an important document in multiple columns or rows and pan up, down, right, and left.

The great thing about all of these steps is that it gives us much more freedom to move around and the ability to shoot things handheld that still hold together.
The difficulty for Photosynth's UI programmers is that when you're dealing with handheld shots (like the vast majority from upcoming apps will be) the rows and columns are not likely to be evenly/equally spaced.
NateLawrence (4 months ago)
When they're dealing with only a single row or a single column, uneven spacing from handheld shooting doesn't matter because you're only moving left and right or up and down and as long as there are sufficiently overlapping photos in the direction you're panning/moving, you can continue in it.

What happens in a panning synth, though, when each row of the panorama, as it looks up and down, do not line up with each other?

What I mean is that in each row of the pan, you could look left and right just fine, since they were captured in rows.
The problem comes in that there are enough photos to make columns out of for looking up and down, but they are dreadfully crooked columns. This becomes problematic for smoothly animating in the up and down direction.

(Alternately, if you shot in columns originally you would, more than likely, have very crooked rows, which gives you smooth panning up and down but very poor left and right movement.)
NateLawrence (4 months ago)
Perhaps equally difficult in a grid of photos (as opposed to the current single row/column of photos) is being able to load the correct photos before the user navigates to where they should be shown.

In the current situation, it's simple to load the user at the beginning of a Walk synth + load as fast as you can down the path in front of them - or delay movement momentarily until you can download enough. Pans and Walls are the same. Spins are interesting in that the default view for a non-360-degree spin is always in the center of the arc, so that actually gives an extra little bit of download time for the photos in the second half of the spin.

In a grid, because the freedom is there, the user may suddenly decide to deviate from the default play order to go look in the direction they want to (and those photos may be toward the end of the default download order). The more freedom there is, the less predictable the user's movement + greater risk of unloaded imagery.
NateLawrence (4 months ago)
As far as photo loading order when the user begins moving in their own direction (or even in the same direction as the default playback order but further ahead than photos have downloaded, as can happen today in PS2 synths - especially on slower connections like the Wi-Fi at an electronics store) you can skip to where they have moved their viewport to and continue loading from there (this is actually the behavior of the Tech Preview viewer).
NateLawrence (4 months ago)
In a grid of photos, though, it isn't as simple as skipping to the point in the line of photos that they jumped to and resuming downloading from there. 

In a grid, you could detect the direction vector that they're heading in and assume that they'll keep moving that direction and attempt to load as many photos that are intersected by that path as possible before they get there. If they change directions, though, you're sunk.

Alternately, you could always try to download all neighboring photos of the position that the user is currently navigated to so that no matter what direction they turn next, there is a little loaded there. (That would be similar to how Seadragon always loads image tiles from the center of the screen outward in larger and larger radii.) The problem with that is if the user actually is trying to move in a straight line you'll waste a lot of time downloading surrounding photos instead of those that are ahead in their path.
Pfff Nate, you always give me a real challenge with your tech explanations, but they are good and I think I understand what you are telling in the above.

Yes, semantics of terminology has become an issue.
I had a look at the Andre's panorama. Are you sure this was shot with a fish eye? I think that if I would do a portrait panorama with my Lumix TZ30 with 16:9 aspect ratio sensor, then I would get the same effect.
NateLawrence (4 months ago)
It's true that Andre (mostly) de-fisheyed his shots before creating the panning synth I linked to, but you can read his own words here:

To take things back around to your initial question, though, I think that even though I am very interested in moving beyond the four different 1-Dimensional navigation synth types that we have today, we haven't heard much from the Photosynth team about this.

The first sign of what we got in December 2013 was first shown to be operational in February 2011, so there has been a big gap between tech being developed + getting released, so I share your desire to see much quicker progress. 

What is worrisome to me is that we haven't seen an actual running example of a new synth type with multiple rows|columns, so how much of a priority this is to them remains uncertain. We heard David say, "We'll get there." but when?
In another discussion you have pointed out that from a set of cylindrical panoramas in a given space, other synth types can be created in that given space. I imagine this works easily with frame panoramas, but would it work as easily with stitched panoramas? I feel having frames is more versatile and fit for all sorts of reuses, while stitched material is not. So, let's forget the stitched material.
Yes, but we might be able to come up with a stepwise approach.

Taking the current Ps2 cylindrical panorama as a starting point. Let's set the other types aside for a while.

What I am looking for is how can we realize a more being in the space experience. Now, I miss what is in between where I am standing and what is represented of the space in the view. On average, I would say one is not as much interested in what is over our heads.
NateLawrence (4 months ago)
I'm sure you're aware of all of the times that the Photosynth team has mentioned that they also plan to offer the stitching of panoramas on their server (just as they currently synth the new photosynths on their server) so although that wasn't your question here, I think that is what they're thinking with regards to spherical panoramas. A bunch of the quotes to that end are collected in Andre's thread that I linked to above.

The quote from the the Expert Shooting Guide in this older related thread where you and I talked to hefeweiss also points in that direction.
NateLawrence (4 months ago)
That may not be what you wanted to hear, but if you read Photosynth's reply in xRez Studio's thread today, you'll hear more about what percentage of the input photos must have foreground in them in order to create the geometry for panning synths, which probably gives you a more definitive answer to your question about PS2 style spherical panning synths.

The quote: 
"Be aware that right now you need 3-view-matchable foreground detail in about 80% of the frames before we process the pano as a parallax pano. 

Among other things this means that a pano where 1/2 the sphere is a distant view (from a lookout for example) and the other half is the near-field view of the area behind you will NOT be processed as a parallax pano. 

It will be processed as "planes without geometry", which is equivalent to but mostly worse than, a stitched pano. 
We hope to get more sophisticated about this in the future, but for now get foreground detail in all frames."
NateLawrence (4 months ago)
Sorry. I think we're typing at the same time.

As to your objection to stitched panoramas, I don't necessarily follow. 
A single stitched panorama can easily be a single frame in a sequence. 

That is to say that a medium resolution cube map can just as easily be stored and transferred in a single file as a medium resolution regular photo.

Alternately, if you want to divide the data up a little more, then a cube map from a stitched panorama can easily be divided into its six sides and transferred individually.

I'll hold off for a little while so that you can add to your thoughts here and then respond later.
NateLawrence (4 months ago)
Well, it's been 3 hours, so I'll just add this before I sleep. 

There's one other relevant Photosynth team quote in a reply to Jeremy from about 4 weeks back. I was trying to find this to include earlier.

"When you want wider field of view than is offered in a single photo, then you definitely want the stitched panorama processing (referred to in this thread as Photosynth V1). We plan to keep that around because it's exactly what you want if there is very little parallax or motion in the scene. We'd like to stitch these in the cloud to remove the requirement of using desktop ICE. It's on the list. Photosynth V2 (as it's informally known) does not supersede stitched panoramas. It's an alternative to them. Does that help?"


For panoramas, in particular, you need to communicate your vision for what parallax for the floor really adds to the pano.
NateLawrence (4 months ago)
Also, I'm curious about your bias against stitched panoramas. 
Google uses them, as I'm sure you know, but I don't hear you criticizing them for it.

If I could have what I really wanted with panoramas, it would be to recover as much geometry as possible for the entire scene, project all the photos out onto that (with the distant parts without sufficient parallax being projected out to a sphere whose surface lies behind the most distant foreground geometry) and then to have textured 3D meshes for the foreground elements or at least multiple alpha matted 2D layers for foreground objects.
NateLawrence (4 months ago)
If we could add to that, looping video for every direction looked, with objects observed at different distances being projected appropriately, then we'd have something really quite compelling. 

Imagine seeing the wind blowing the grass and leaves on trees everywhere you looked from a field in a park as foreground moved correctly against background - or watching a video taken from the same position that the pano was, correctly embedded in the pano - scrolling and zooming the pano as the footage did and allowing the pano to fill in the edges left by the difference between the aspect ratio of the video and the aspect ratio of the screen (or by empty corners left by stabilizing the rolling rotation left or right of a handheld camera).

I would love to get to the point where it is a standard feature on a video player that it performs motion tracking and can fill in the difference between video and screen aspect ratio by using information from other frames in the video.
First, let me admit that I did come across much of what you are referring to before, but that it was not in my mind as an active memory. Even not the Photosynth Team being clear about having server side stitching, which I really like. (I feel low res smartphone stitching for immediate show of should be an option that can be turned off, because of battery depletion.)

So, thank you for the effort to bringing this all back to the table here.
My primary objection to stitched spherical panoramas is because of the following.
1. Moving objects cause a problem. In public space, this often is the case. ICE is doing a great job here, because I have the impression it tries to avoid artefacts.
2. Stitching may have serious artefacts, if the camera is not kept in position properly and frames are not shot evenly. This calls for a tripod up to more professional equipment to be carried along. ICE is doing a great job here again, because it compensates for moving out of position up to a certain level.

All no problem for the expert enthousiast/professional, but it is for mainstream usage. 

I definitely criticize Google on its policies for accepting contributions to Gmaps. Not just any panorama, but it should be full 360 with no jagged top and bottom borders, good vertical coverage and no serious stitching artefacts. I expect this to fail to include the mainstream user. I.e. crowdsource fails too.
I bet there are plenty of viewpoints worthwhile for a pano, but behind me while shooting, plenty of people, a wall or anything else that would spoil a cylinder and even more a sphere. Google's policies results in exclusion. That is why I very much appreciate Photosynth's policies not to be that strict. 

Also, I feel that crowd sourced material is not the same as Street Side and that should be clear in the presentation. Google goes from the idea that crowd sourced material is an addition to Street View and is presented as a Street View feature. That is why they have these restrictive policies, to have crowd sourced material like Street View. I consider this very unwise.
Although I do not really have an idea about what a cube map is, I do get from you Nate, that my idea of versatility of use of individual frames versus a stitch does not hold truth.
Back to my wish for a more in the scene experience with spherical panning synths.
Moving objects in the scene do not cause artefacts with current Ps2 panorama functionality. I even like it, because it gives some natural dynamics in the viewing experience. So, I very much prefer it over a stitch.
But, adding a more in the scene viewing experience by extra vertical coverage with extra frames introduces many challenges. You have pointed out the technical once above. Thinking about adding an extra vertical row. Then, moving objects are a capital problem in a stitch. Thinking about it anew, I realize it would introduce oddities in a spherical panning synth too. E.g. a head in the upper frame, while there is no body in the lower frame. This trade-off between more in the scene experience and the artefacts could be a creators decision in what to aim for.
I have the impression that integrating 2 dimensional navigating into the current Ps2 viewer might be a mismatch. It is good as it is. For navigating a spherical panning synth a Ps1 style viewer might be better, I feel. However, having different viewer paradigms is likely to confuse a major part of the audience.
In the current Ps2 panorama functionality, extra vertical coverage with extra vertical frames does not cause a frame matching issue, but it does cause an auto path issue. So, frame matching based on Date Taken works fine and saves a lot of processing compared to image matching only. But, would it be an idea to use another way to define the auto path, just taking the vertical middle for example?
NateLawrence (4 months ago)
Sorry, I'm not meaning to lose you in terminology. Here's a helpful illustration of what a cube map is.

Think of a panorama being projected onto the inside of a sphere or cylinder. 
This is good for stitching, but for rendering, that might be more polygons than you want to load at one time. 

(Back in the 1990s when this rendering method was becoming common, having a GPU was not common, so a lower polygon object to project photography onto when actually rendering the graphics for the end user was desirable and although every modern computer, tablet, and phone has a GPU which is more than capable of running a single sphere or cube with a high resolution texture, people these days are very used to having many tabs and apps open, meaning that having a less complex model is still desirable.)
NateLawrence (4 months ago)
(Sorry - here's the link again with the protocol attached: )

There are also some texturing issues when rendering onto spheres. 
Namely, the columns becoming very small at the apex and nadir of the cube (the north and south poles, if you will).

One way to simplify what you are rendering and how you store the texture is to center the sphere inside a cube and shine rays of light out from the center of the sphere in all directions, through the surface of the sphere, and onto the walls of the cube.

The result is six simple square images which are easy to store, easy to load, and easy to render. 
A cube is a simple geometric object, so it's easy to render with very few polygons so it should always remain very responsive.

There is a little trickiness in the corners of the cube where the imagery will always be further away from the virtual camera in the center of the cube than the center of any of the walls, but this is surmountable.
NateLawrence (4 months ago)
Apologies again. "columns becoming very small at the apex and nadir of the cube" ought to read "columns becoming very small at the apex and nadir of the sphere".


I think that your strongest point above is that, since the majority of people shooting panoramas will be doing so handheld (and therefore changing the camera lens' position between shots) and because sharing a spherical panoramic view remains compelling from a single point of view, that we ought to be able to take Photosynth's innate ability to cope with the camera lens' position changing between shots and leverage that to line the photos up correctly.
NateLawrence (4 months ago)
I've wanted to be able to run Photosynth as a pre-process to ICE before, when trying to process a handheld panorama, in order do a 3D model of the foreground and figure out how it overlaps with the background. 

This would allow the foreground to be modeled in 3D, have the photography from slightly differing camera positions correctly mapped onto that model for a unified alignment, and then render that from a single central position and let ICE do a stitch of the background and a single instance of the foreground overlaid on top of it, which ought to get rid of all alignment artifacts.

I grant that it will still not help in cases where only part of a moving object was captured in one of your frames - a car or a person, etc. but I feel that is more the responsibility of the photographer than the software at this point in time. 

When shooting a scene with movement, I try to shoot perpendicular to the motion (e.g. people walking horizontally so shoot in columns)
NateLawrence (4 months ago)
That sort of thing is, of course, only good for capturing + not for viewing order, as you pointed out above.

Something I've only begun to use but am finding very useful is to use a tripod + longer exposures when shooting panoramas in public spaces. This alleviates people's fear of being photographed + gives me a cleaner view of the environment. It takes time, though.


ICE always attempts to set the stitch vertically centered so, what you've said above about a good suggestion for the default playback path for a multi-row panning synth (or multi-column panning synth) being the vertical center, or what hefeweiss said before about 'locking the pano to the horizon' is certainly a fair expectation (if we are ever given the option to upload multiple row/column panning synths).


While I do enjoy the unified exposure of a stitched panorama, I wholeheartedly agree with you about the more dynamic feeling that you get when crossfading through the actual input frames.
Nathanean, you do lose me in jargon from time to time. ;-) But, I learn from the explanations. I understand CubeMap storage mow. I do not understand issues of columns becoming very small at the apex and nadir of the sphere and textures.
Anyway, curious now about what the PhotosynthTeam makes of this conversation.
You need to be Signed In to add a comment. (Are you new? Sign Up for a free account.)