Topic: Photosynth constellation viewer requirements

Report Abuse Report Abuse
Joscelin.Trouwborst (Over 1 year ago)
Dismantling 2 Photosynths 1 into its spins, walks and panos, it made me think about viewing and navigating in such constellations. Now, I feel some things could have been better in the Photosynth 1 viewer.

How about having a 2D overhead map overlayed transparantly in a corner of the 3D photo view. Photosynth constellation components should be projected on this 2D overhead map. It should also indicate the current 3D viewing direction on the map.
Compare Google Views.
Joscelin.Trouwborst (Over 1 year ago)
In Photosynth 1 we have the control with its 4 arrows for navigating the 3D view. This control does not distinguish between moving from spot to spot and turning on the spot. I feel we need navigational controls that have this distinction.
Pano/sphere: turn up, down, left, right
Spin:  move left, right
Wall/walk: move forward, backward
and exit/enter/jump from one constellation component to the other.
NateLawrence (Over 1 year ago)
There should definitely be some clear indicator that other nearby synths and/or panos are available when navigating linked synths.

Photosynth 1 had Overhead view and Photosynth 2 already has the Map feature (which, I'm sure, could be used to show surrounding camera paths and could/should use Bing Maps as a backdrop for geotagged and geo-aligned synths).

When the Silverlight synth editor was first added to the site along with geo-alignment abilities, it was laid out a bit differently than it was in later months.
The primary window still was locked in overhead view, but in the right hand panel, there was a small standard synth viewer in 3D view embedded on the lower right which you could control like normal. 

What I loved about this was that you could view the virtual camera moving from camera position to camera position on the map which is what you're talking about as far as showing the direction being looked on the map.
NateLawrence (Over 1 year ago)
I always wanted that experience to be part of the main synth viewing experience as well. (See and )

As far as Photosynth 1 navigation being lacking, the Photosynth team has said publicly multiple times over the past several years that they feel the same.
One of the most recent was Blaise's talk at the Hockney show last fall:

There is actually recognition in the original Photosynth viewers for the need for both turning and moving controls, but these really applied more to navigating the point cloud than navigating the photos. The WASDEC keys moved forward, left, backward, right, up, and down; the {L:" keys looked up, left, down, and right.

There were also keyboard controls for moving in and out of the screen to other photos, which differ from the up and down cursor keys, but the limiting factor there is people having taken photos to move to from any given photo.
NateLawrence (Over 1 year ago)
In any case, the severe limits on Photosynth 2 synths are specifically designed to make navigation simpler and predictable, as expressed in the blog post about panorama support under "A down payment on the future".

If we get to the point where we can upload a walk synth composed of spherical panoramas, there ceases to be a difference between wall synths and walk synths. 
The only difference in such a synth is which way the viewport is turned while moving through the camera positions.
If your viewport is pointed straight up, down, left, or right then such a synth is a Wall synth.
If your viewport is pointed forward or backward, then you're experiencing a Walk synth.
And of course if you stop at any point along the way, you're in a panorama.
NateLawrence (Over 1 year ago)
I mentioned it above, but note that (although I'm not claiming that it was particularly effective, even in synths with more structured shots to move between) in the Direct3D viewer and earlier versions of the Silverlight viewer there actually are different arrows for next photo up, down, left, and right and moving further into or out of the screen to neighboring photos.

In the Direct3D viewer, four of these were around the edges of the screen (and become usable as there are neighboring photos to move to) and the in and out arrows at the bottom of the screen on either side of the 'down' arrow and the Silverlight viewer later consolidated these to the menu bar at the bottom of the viewer. (The first version of this after the Silverlight viewer changed the navigation UI can be seen in the screenshot in this web log entry: so there were attempts to address this before.
NateLawrence (Over 1 year ago)
Out of curiosity, Joscelin, what are you envisioning happening exactly when you talk about exiting/entering/jumping from one synth to the other?
What I mean to ask is, what are you envisioning seeing if you tap the command to exit a synth? 
Bing Maps with pushpins or some other representation of nearby synths and panos?

One one hand, I don't really want to have to press a key to move to a nearby linked synth. 
In Photosynth 1, I could make a synth consisting of multiple walks, spins, pans, and walls and if I was in a pan in the center of a garden that was surrounded by six nearby spins, I could just click on one of the quads from those nearby synths to swoop over to it.

On the other hand, I'm a firm believer in having a way to do things with only keyboard, only mouse, only touch, only voice, only joypad, only motion (Kinect), etc. so in that sense I agree there ought to be a keyboard shortcut to move to another synth.
NateLawrence (Over 1 year ago)
In previous viewers the M key would jump you to the beginning of the next point cloud in a synth that wasn't 100% synthy.
(This seems to be broken in the current version of the Silverlight viewer and only takes you to the beginning of the largest point cloud, rather than moving through the list of subsynths and back around to the largest, but still works in the Direct3D viewer.)

That's the nearest thing that I can think of to a keyboard shortcut to move to another synth in the area, but I don't know if one key that moves you to the nearest neighboring synth is particularly effective when there are many near each other on the map.
Joscelin.Trouwborst (Over 1 year ago)
Yes, I feel the Photosynth 2 constraints to have synths to be more elementary is a good thing. It will bring structure to shooting and viewing, thereby increasing its usefulness.

I like Nates vision of Sphere constellations which allow Walk and Wall experience.
(I feel we should change over from the word/concept Panorama to the word/concept of a (partial) Sphere. So, a Wall and a Walk are constructs of partial (maybe unfinished) spheres.

Not to forget the Spin as an element on its own.
Joscelin.Trouwborst (Over 1 year ago)
With exit/enter/jump I mean the following.
One exits one elementary synth to enter another one. They can be smoothly connected or it can be some sort of transitional jump if they are further apart in real space. All connections between elements should be possible in real life too. That is, there should not be a walking obstruction and one surely cannot jump from a tower or something. (I have seen these in Google Sphere Constellations in spite of the guidelines.)
Joscelin.Trouwborst (Over 1 year ago)
I have been addressing navigation using the keyboard paradigm. However, I do not mean this to be about the keyboard or mouse way, but about the fundamental navigation possibilities and experience. Of course there should be implementations for touch, keyboard, mouse and whatever more.
Joscelin.Trouwborst (Over 1 year ago)
It seems to me that as human beings we have the following basic options to explore our environment with sight.
1. Look up or down.
2. Look right or left by turning our head or body.
3. Change position.

I feel the viewing experience in a Constellation of elementary Photosynths should be based on this. 

In general we follow paths. Generally we do not jump in or fly over water, we do not fly from skyscrapers, we do not climb over obstacles. This was readily possible in the experience so far. That is not a natural real life experience to me.
Joscelin.Trouwborst (Over 1 year ago)
In other words..
=> 1. + 2. is spherical experience.
=> 3. is moving along a path: straight/curved, circular/oval spin of an object.
Joscelin.Trouwborst (Over 1 year ago)
I have no gaming experience in virtual environments. I have had a look at Street Side and Street View occasionally. The lack of this experience might limit my thinking.

I took a look at Street Side just now. I feel Street Side navigation is more troublesome than Street View navigation. This to such a level that I would not use Street Side to have a virtual look around.
NateLawrence (Over 1 year ago)
As far as panorama terminology goes, a spherical panorama is only as complete as a panorama can be. 
Anything less is simply a partial panorama - but a panorama none the less. 

Enough people take partial panos that I wouldn't insist that they use an adjective to qualify their panos as partial but neither do I feel any need to qualify a full panorama as whole (unless I need someone to understand that I'm speaking only about complete panos).

In other words a spherical panorama is not something more than a panorama.
It is only the most complete field of view that a panorama is capable of having.

The word 'sphere' is completely generic so we would have to rely completely on context to make sure that people are not using it in some other sense in discussions here (which may not automatically translate very clearly).
NateLawrence (Over 1 year ago)
As far as limiting navigation between single synths to only reflect what would be typical in real life, I don't agree.

If I can see the same object in both synths (for example, a tree in the background of a wall synth taken from a rooftop and a spin synth around that tree) then I want to simply move from where I am to go look at that other thing, after Photosynth knows how they relate to each other.

It shouldn't matter that people don't typically jump from my roof to my front yard. 

One of the things that I wanted before I had ever heard of Photosynth (in fact, years before they began making it) was to spatially connect all photos of the neighborhood where I grew up - both aerial photos and ordinary shots of events and buildings taken from the ground. So, to me, having the system understand how all of these views fit together and being able to swoop out of the sky down to shots of a sports tournament or class photo in front of the school is exactly what I want.
NateLawrence (Over 1 year ago)
Similarly, if I've synthed the exterior and interior of a public building and want to go directly to the inside of the nearest room from the corner outside the building where I currently am navigated to, I don't really want to navigate all the way around to the door and go through the entire building to find that room if Photosynth can link the exterior views to the interior from looking out the window from inside and make the connection (or from exterior to interior if there is a big open window which allows the interior to be seen from outside enough to be matched).

For that matter, if both have been geo-aligned but there is no obvious visual connections (say the inside and outside were shot in different years or during different seasons), I still would want to be able to transition directly from one to the other (perhaps via a swoop up to the map and back down) simply because they relate to the same map.
NateLawrence (Over 1 year ago)
I'm interested to hear more detailed feedback on Streetside.
One problem I've had with it is that it is a little too easy to click on the next street over and float through a building when all I intended was to click on a building's façade to zoom in.

There are plenty of other improvements to be made such as preloading neighboring views so that they are loaded before you begin rendering them when the user begins to move (as Photosynth does in new Walk synths).

There are also currently issues with performance and stability which stem from development of Silverlight being canceled and therefore all development of Bing Maps' Silverlight controls being canceled as well and not using the latest version of Silverlight which might have given decent performance.

I expect the performance and stability problems to go away once Bing Maps moves to WebGL in the not very distant future and we'll see whether they have learned anything from new synths at that point.
NateLawrence (Over 1 year ago)
Part of what I began to say yesterday with synths composed of spherical panos blurring the lines between the new synth types is that I believe that we should think about navigation from that perspective as it adds some considerations such as whether you expect to be able to rotate your view as you control movement along the path.

For that matter, if we had a way to lock the center of the viewer on something to the left or right of the path in the center of a walk synth composed of full panos, you essentially have a spin synth of that object or point. It follows a linear path, rather than an elliptical path, but will yield many of the same angles of the subject.
NateLawrence (Over 1 year ago)
Also when capturing a full spherical pano in every shot (say with the Ricoh Theta), there turns out to be no practical difference between the navigation of a parallax pan synth and a spin synth, except for where the viewport is oriented. The camera path is elliptical in both cases.

For that matter, you can imagine being able to specify a subject to lock onto in the viewer on one side of a spin synth of full panos and have something of a small wall synth or walk synth of that background subject which was not the primary subject in the center of the spin synth.


If we're using something like an Oculus Rift to view a synth composed of full panos, then we can simply turn our heads to face what we intend to look at as we use keyboard or trackpad or joypad commands to move through the synth's path, but when only using a touch screen or only keyboard this becomes more complicated unless we have a way to specify what to lock onto before we begin moving.
NateLawrence (Over 1 year ago)
Think of that 'locking on' as perhaps a video Highlight (except I want any user to be able to specify this whenever they want instead of only the synth's author).


Like you, I'm wanting to generalize about navigation concepts, but I find thinking about how to make the desired functionality work with different input types to be productive.

Since we cannot currently make synths with full panoramas, here is our current Photosynth 2 functionality set: 
A) Change position +/- along the synth's path in walk, wall, and spin synths.

B) Pan in place in parallax pan synths and stitched panos.

C) Zoom In/Out. (Currently only available while stationary)

D) 2D pan around a single photo while zoomed in. 
(Currently only available when stationary. Also steals other functions' controls when the viewer is in this mode)
NateLawrence (Over 1 year ago)
I'm not very happy about only being able to zoom and pan while stationary.

Panning, especially, doesn't make sense to lock into only working when at rest when considering synths of full panos. 

When thinking about video highlights, it would be really ideal if the viewer would zoom in and load higher resolutions of a highlighted area when further away in the synth and then zoom out slowly as one draws nearer to it and zoom back in gently as you withdraw from it again to keep it as close to the same size and filling the screen throughout playback.

This would necessitate Seadragon working in 3D view, but if I've specified a particular part of the scene, it should still be able to pre-load the tiles for that area in advance of my moving to those images.

I will admit that being able to freely look around while moving is probably one of the bigger challenges technologically for the team to solve (at least when trying to keep the viewer filled with decent resolution).
NateLawrence (Over 1 year ago)
On the other hand, having turning/panning controls that work consistently at all times can solve the problem of zoomed panning stealing positioning controls while zoomed (from a control scheme point of view anyway - the technological side is something else).

Ideally moving, zooming, and panning always use separate control schemes which can be used simultaneously in conjunction with each other in the hands of experienced users (or even inexperienced users if the interaction method is user friendly enough, as with the Oculus Rift example).


With a joypad movement is typically controlled with the left joystick and head movement is controlled with the right joystick. 
Adding two buttons for zooming (ideally analog buttons such as the triggers) would be very simple to handle.

With a mouse, mouse movement could be used for mouselook (panning/turning), scrollwheel for zooming, and left and right buttons for zooming.
Joscelin.Trouwborst (Over 1 year ago)
Agreed on Panorama Nathan. I should have checked Wikipedia or the like before my suggestion. On the other hand, my conceptual misunderstanding is interesting for the Team when promoting Photosynth in comparison to others to the general public.
NateLawrence (Over 1 year ago)
Sorry, that was meant to say left and right mouse buttons for moving/changing positions.


On a keyboard, cursor keys could either be used for walking or looking and a second set of keys for whichever is not taken by the cursor keys (WASD for example) while zooming are easily handled by + and - keys.


Touch screens are probably one of the more difficult control schemes to pack much functionality into. 

Pinch to zoom is par for the course (as is double tap to zoom) but the question is whether they can be effectively performed while performing other inputs.

Stroking the screen up and down or left and right makes sense for panning, but the question is then what to do about moving.

Multi-finger strokes could be an easy solution to manual moving.

Tap to hold could be used to specify a desired target to track when moving, either by simply centering the viewport there or by bringing up a bounding box to be fitted to the desired portion of the scene.
NateLawrence (Over 1 year ago)
I think it's clear that if one is in a synth with ordinary narrow angle photos instead of full panos, then you want panning controls to be constrained when moving through a synth zoomed out, so that the user doesn't accidentally scroll the photos out of view while moving. 

Completely separating moving and panning keys (like you suggested at the top) or moving panning to mouselook/screen-stroke allows consistent control - completely independent of: 
what field of view you're looking at or 
whether you're in a synth or a pano or 
whether you're moving and/or zooming or not.
NateLawrence (Over 1 year ago)
Going back to intersecting or overlapping synths... 

If multiple people have all contributed spins around the same subject then clearly you're expecting to view them one at a time and I think that a key to select between them is great.

On the other hand if I've shot an extensive walk all the way around something large like a mall or park that obviously will never fit in the confines of 200 photos and I have therefore uploaded it in chunks of 200 photos (perhaps overlapping the beginning of one with the beginning of the next slice by at least one photo) then when they have linked online, I don't really want to have to press a button every time I move 200 photos forward.

If the system knows that these walk synths all connect and form a larger walk, it ought to begin loading the camera positions and photos from the next walk so that I can just keep going when I reach the end of the current segment, rather than waiting for a loading screen or clicking 'Continue'.
Joscelin.Trouwborst (Over 1 year ago)
So, Walks, Spins and Walls are a composite of Panoramas.
I agree. To me it matches to what I feel as the basic things in exploring ones environment.
- Spherical viewing including zooming.
- Changing viewing position. 
Depending on the real life situation both are executed at the same time or one has priority over the other.
So, I agree approaching navigation from this perspective.
Joscelin.Trouwborst (Over 1 year ago)
I feel the vision which Nate is outlining compelling.
I do not have a clue how far off we are of realizing such a thing.
From where we are today, then what would be the steps forward?
NateLawrence (Over 1 year ago)
One of the things I do like about Streetside is what you asked for - it clearly displays where you're looking on the mini-map (just like the early Photosynth geo-aligning editor) + the compass in the lower center of the screen rotates to show the compass bearing that you're currently looking in.

I remember being surprised that this compass doesn't carry over to viewing Photosynth panoramas on Bing Maps (or even geo-aligned panos on especially since the current Silverlight control has that disc there that you can use to keep track of where you are in relation to your starting view).

Since Photosynth wants to keep the viewer as uncluttered as possible I'm not expecting to see a compass bearing displayed prominently on every embedded new synth, but it makes sense:
1) if the optional Map view sticks around + matures 
2) when viewing embedded in Bing Maps 
3) if the new view page has dedicated page space for a more visible map than the current one does.
Joscelin.Trouwborst (Over 1 year ago)
I feel we should have some segmentation of usage scenarios. Using Photosynth for enhancing the map experience is rather different from showing an object with a spin, a skydive, documenting a pyramid and so forth.

I myself am focused on the Map and personal story telling experience, which will bias my contributions.

Allow me to illustrate the conversation with the Cuba, Trinidad, Plaza Mayor synth photo sets outlined here
Joscelin.Trouwborst (Over 1 year ago)
Example 1.
a. pano:
b. pano:
c. spin:
d. pano:

Between a and b there is a real life connection in a straight line. However, these Panos are at some distance from each other and there is no Walk between them. I think that even though there is no Walk to fill the gap, there should be a move option in the view port.

Should it be possible to move from b to d from an option in the view port? I think not, because spin c is in between. The move option would be from b to c.

To move from b to d would be through navigation options on the mini-map, allowing to jump to any shooting spot. (Note that I am using 'jump' different from previously in this conversation.

Moving in on an object regardless of the real life situation
Joscelin.Trouwborst (Over 1 year ago)
Moving in on an object regardless of the real life situation is a different thing, I think. It needs a separate navigation solution.
Joscelin.Trouwborst (Over 1 year ago)
This is a Constellation of Spheres with a way to navigate alongside the wall.,23.728435,3a,75y,348.47h,82.45t/data=!3m5!1e1!3m3!1s6l5HxV6YVmSf4mV8pGjr5w!2e0!3e4
NateLawrence (Over 1 year ago)
The forum didn't like the commas in the URL. Here's the whole thing inside a short link:

It's a good example. The big differences are that:
1) no depth information from comparing all the panos is being used to project the panos onto + the warping is bad to the sides
(even though we see that they have some information about the scene from how the cursor changes while mousing over the scene) + 
2) Google doesn't appear to have any good navigation method for strafing sideways as PS2 Wall synths do. 

Clicking the right or left arrow repeatedly + only loading the tiles for the next pano after you start moving to it isn't nearly as good as: 
pre-loading the imagery to either side, 
projecting the panoramas onto the recovered geometry, 
+ being able to constantly move the camera instead of just moving in little jumps from the center of one image to the next.


Your a,b,c,d example above reminds me of Noah Snavely's
Joscelin.Trouwborst (Over 1 year ago)
Agreed, Nate. Have you noticed the appearing x markers while mousing over the scene, indicating another pano position?
Thanks for the reference. I had not seen it before. Impressive!