Topic: Improved pointcloud? Maybe, you be the judge

Report Abuse Report Abuse
jimcseke (Over 1 year ago)
I think I may have found a way to improve the point cloud. I was experimenting with a single pic.
With a program I have, I am able to rotate the image cw or ccw to any angle. As it rotates the image, it also zooms in each time.
So, for fun I decided to rotate the pics 360 degs c.w. 5 degs at a time. Photosynth treated each pic as a unique image. Here is what I ended up with.
Then I tried it with an older synth with all the pics. It did affect the point cloud but it was not what I wanted.

I realized I was onto something, so I took 18 pics (lava rock) from part of a pending synth, and uploaded as is. 
Then, with the original 18, I rotated each pic twice cw in increments of 5 degs, and uploaded those along with the original unrotate pics. Creating 36 “individual angles”. This method seemed to create a  denser point cloud. But still was not what I was hoping for.
jimcseke (Over 1 year ago)
I then uploaded 40 original pic of the lava rock, where I rotated each photo 4 times (2 incr. ccw and 2 inc. cw at 5 degs each.
Totalling 120 pics from the original 40. This method seem to work really cool. 
I don’t know if anyone has tried this method yet, but if it works it could mean taking less photos and then doing the cw and ccw editing.
This is only a theory, but if you were to do an orbital of say a box. Shoot the 4 sides, 4 angles, and maybe 4 of the top totalling 12 photos. Then the editing would create a total of 60 pics.
Remember, the lava rock has 40 unique pics but Photosynth treated them as 120.  I will give the links to each synth for you to compare and hopefully comment. Perhaps I am way out in left field with this and a waste of time but it was fun experimenting.
jimcseke (Over 1 year ago)
When use the links, open them in separate tabs, to get good side-by-side comaparisons to fully see the differences.
Cheers everyone, way past my bedtime!
Nathanael (Over 1 year ago)
Right on, Jim.

A different take on the same idea is this synth. In its case, the author didn't rotate and crop, but instead cropped out portions in place. It's easier to see to understand than to read my words.

Another important concept is that the synther only analyses 1.5 or 2 megapixel versions of your photos for details. There are at least two reasons for this:
1) Keep the number of features to be matched down so that even people with weaker computers can make synths.
2) If the images are all different sizes, using 1.5 megapixel versions of them all means that if you have a 20 megapixel photo of a mountainside and a 4 megapixel version of that same mountainside where the mountain takes up roughly the same % of the frame in both that Photosynth will be able to successfully match them, not worrying that one is 500 times larger than the other.
Nathanael (Over 1 year ago)
The obvious drawback and what struck me the hardest when I began synthing and experienced some failure was that not every pixel was being analysed to match the images, but, as I came to find out, only 1.5 - 2 megapixels of each image were being used to match, leading to a less detailed pointcloud and also possibly missing matches that could have otherwise been made (although making more matches is theoretical being that the scales could possibly be quite different still, thus requiring extra steps in the existing algorithms).

The obvious cheat, then, is that if Photosynth only uses 1.5 versions of anything that I give it, I can give it not only my original images, but also 1.5 megapixel chunks of my photos (or maybe even 1 megapixel chunks, just to make sure that every last pixel is looked at). The pieces should match their parent photos pixel for pixel, thus generating all the features possible for that image to be considered by the synther.
Nathanael (Over 1 year ago)
Of course, with only one photo and its pieces, as displayed in your first link 
( ) 
and the baseball stadium synth that I linked to above
( ), 
since all subimages are derived from the single parent image, there is absolutely no parallax and therefore Photosynth cannot estimate any depth, resulting in a pointcloud pancake.
Nathanael (Over 1 year ago)
Clearly, when we mix this cheat or hack to get Photosynth to look at every pixel of our photos and combine it with photos from angles which stress the differences in the depth of a surface, you get all of what Photosynth is capable of.

You may end up needing fewer photos in arcs around things since every last detail is able to be analysed, but I think that you still cannot make leaps of 90 degrees difference.
Nathanael (Over 1 year ago)
Ah, I just re-read your closing statement there with four photos per side of the hypothetical box and that should certainly provide close enough angles to make the jumps.

Although your method of rotating and cropping provides more overlap than FlyPhoto2020's, I think that given photos which are right side up, it's going to be better to skip the rotating and go straight for cropping the copies or the original. This lessens the chance of Photosynth miscalculating which way is 'up' for the pointcloud. 

In cases of extremely large original images, 1 megapixel crops may be so much smaller in comparison with their representation in the 1.5 megapixel version of their parent image that the synther may fail to make the connection between them. In this case, a hybrid of your technique (Progressively cropping in a little at a time) and FlyPhoto2020's (providing fixed resolution pieces all right side up) would probably work best.
Nathanael (Over 1 year ago)
Providing the synther with something similar to Seadragon's own tiling system is actually ideal for guaranteeing that the pieces will fit their parents. Essentially Seadragon's multi-resolution format takes the original image, makes a copy that is 50% of the original, 25%, 12.5%, and so on and so forth until you hit one pixel. It also goes to a second step of dividing each of those resolutions into tiles of a predetermined size, usually 256 pixels, but not always.

Dividing each resolution into tiles is not what we want. For packing as much detail as possible into a single synth before the number of image features becomes too great for our computers to complete reconstruction, we want the absolute minimum number of child photos from each parent photo.
Nathanael (Over 1 year ago)
1) We know that the synther cannot make jumps between different copies of the same feature unless one is double the size or less than the size of the other. You should think of this in terms of percentage of the frame that any given feature is filling, being that all images will be resized to 1.5 megapixels for synthing.
2) We know that we want to upload the original image for viewing purposes.
3) We know that we want to use crops of the original pixels in small enough chunks that the synther will not make smaller copies and thus skip pixels.
4) We know that we want to use the minimum number of crops neccessary per parent photo.
Nathanael (Over 1 year ago)
From this we can determine that: 
1) As long as the parent image is small enough it may be sufficient to provide only the original image and crops of 1 megapixel tiles that fit each other precisely.
2) If the parent image is larger, we may need to introduce a crops of the original image split into quarters in order that the 1 megapixel crops match the original.
Nathanael (Over 1 year ago)
If the pointcloud is the only goal then this serves us very well. If, however, we are concerned with the photo viewing experience, mixing all of these crops of the original photos into the synth makes navigating the synth a chore.

Ideally there could just be an option in the synther that would let us choose to analyse the original photos at original resolution or have a slider with original resolution as the maximum and 1.5 megapixels as the low end of the spectrum.

References where I get this 1.5 - 2 megapixel talk:

A good visualisation of Seadragon's multiple resolutions before they are tiled:

For a discussion around large synths:
Nathanael (Over 1 year ago)

"go straight for cropping the copies or the original" should read 
"go straight for cropping the copies *of* the original".

Also in the entire part where I describe Seadragon's multiresolution system, I only meant to say that between the original image and our example of 1 megapixel crops from the original we should use crops of the original photo which correspond to 50% of the original photo's size, crops which correspond to 25% the size of the original photo, etc. until we reach a point where our most recent crops are double or less than our 1 megapixel crops.

This is probably making things muddier, but I just wanted to say that I wasn't suggesting giving the synther exactly what Seadragon generates as all resolutions of the image greater than 1.5 megapixels would look identical to the synther when it resizes them all to 1.5 megapixels.

Rather, any crops between the original and the smallest that are useful to us (1 MP) should follow the halving rule.
Nathanael (Over 1 year ago)
... Sorry for thinking out loud in text here, rather than getting organised offline and then posting, but this is something that I come back to touch on probably roughly once a week on average. (You'd think I'd have my head on straight by now. =)

Another benefit of coming up with a more evenly distributed cropping scheme is that you should get every last pixel in the original image analysed, rather than only the centre of the photo. (Your corkscrews make very pretty designs with the quads, but ultimately mean that the edges of the photos are less dense than the centres.)
michaeldenis (Over 1 year ago)
Nathanael: Great(extensive)discussion! Your posts are a real bibliography of what's-what in "Photosynth's Technical Workings."  Your posts within your posts are also very helpful.

jimcseke: Interesting tests! While I have limited experience with 3D construction using Photosynth, I can see the improvements over my own works in your synths.  The density of your synths is outstanding.

- Give me the space to synth and I will move the world. -
jimcseke (Over 1 year ago)
Hey Nathanael, thank you for taking the time to comment. I want you to know that I value your opinion very much. And also appreciate that you take the time to research and use your own experiences to set me in the right direction. I do read your replies, and in some cases more than once.I fullheartedly agree with michaeldenis who wrote here, that "you are a real bibliography of what's-what in Photosynth". If I don't reply in some cases, I just haven't gotten around to it yet. But your advice is sincerely appreciated, and seriously considered. 
Just thought I would let you know that, and just throw a big thank you blanket over all the advice and critique you have kindly given to date.
This is a great site, and you are one of the reasons it's focus is maintained.You should be given an official title, and be put on the payroll :) 
Thanks again, and you can bet I'll be hitting you up again shortly.
Nathanael (Over 1 year ago)
Thanks for the kind words, guys.
jimcseke (Over 1 year ago)
Hey again Nathanael. So far the experiments I have done with my Zoom Edit have be done to the extreme to show just how this type of editing can improve the density of the pointcloud.And I am not suggesting that each image has to be zoomed down to the smallest detail. The more I used my zoom method, I became more convinced that this type of editing can have many applications. I have created some additional synths with this method, trying to determine approximately how it can be used in a synth. Rather than try to imagine how others might use it, I will give a scenario of the typical synths that I will make using this method:
I will shoot all my pics at 12mps. Now that I can use the zoom method, I have a better idea of which pics to take.
For example, when I shot my train synth, above the large springs assembly there was some raised lettering. The letter B and the word PAT'D. I took a lot of shots of this area because..... continued
jimcseke (Over 1 year ago)
I thought it would make for a more interesting pointcloud. Because I have just started experimenting with this process I do not know yet as to the max/min zoom edits to use, but I know that I could have taken a lot fewer pics of the raised letters, and I guess perhaps 2 zooms edits to each of those pics (cw and ccw). When we go out to take pictures, we tend to eventually take more pics of the areas which we hope will be the focal points of our synths. With my zoom method, that is basically what you are doing. My zooms, so far have been accepted as an original by the synther. So it is the same as you shooting more closeups of a particular area. Giving the synther more points of reference to create the pointcloud. 
Now when it comes to editing, I will look through each image looking for the ones I can by now assume won't reference well. Those are the ones I will use the zoom edit on. The raised lettering for example. I will do all my editing with the....continued
jimcseke (Over 1 year ago)
pics at 12mps. This gives me alot of detail and elbow room for editing. After the zoom editing is finished, I can now reduce the 12mps down to a smaller size. This way there is less processing needed. I can reduce my final images down to 8mps or smaller yet. The latest single photo synth I made was reduced to 10mp (just to be safe) and the absolute smallest zoom I edited was still over 2mps (more than enough for the synther to reference)and that smallest zoom can still be zoomed in on.Plus it surprizingly creats a defined pointcloud. The latest and final test I did is of three sequential images reduced to 8mps. Keep in mind when you view this synth I used only 3 original images.
Please let me know what you think. :) 
And anyone else reading this discussion please give me your feedback.
jimcseke (Over 1 year ago)
Oh and I forgot I uploaded a 10mp version of the same set of pics just to see what if any difference it makes in quality and upload time. For my own curiosity, I am going to try to determine the smallest mps I can use to create a good pointcloud. Less mps = less mb = less space taken up in my 20 gig quota = and I'm not sure about this one yet, but less upload time. In this instance I think that "Less is More" definetly applies.
jimcseke (Over 1 year ago)
The next thing I was think about working on, is occular occlusion (sp)with the zoom edit. Just for fun. If it doesn't work, I still enjoy the process.
Nathanael (Over 1 year ago)
Hey... you deleted your tests! Maybe next time consider just unlisting them?
jimcseke (Over 1 year ago)
I'm sorry Nathanael,but they really didn't seem to be getting much attention. I felt kind of foolish putting them out there, kinda like a kid yelling out to everyone "hey look what I can do" and nobody looks :(  I made a sad face.
Seriously I am getting kind of tired of editing and my wife is starting to give me dirty looks when I spend too much time on the computer. I honestly think SHE needs to get a hobby. And my daughter complains that her internet is running slow because of my uploading.
I am doing some different editing test and will be posting them to YOU shortly.That way I know that you have them and will eventually read and comment on them. I thought that you were not able to get a chance to view them. If after viewing,you think they merit making public, I will do so.  "Hey, look what I can do!" : ) . Now I made a happy face.
Have a great weekend Nathanael! And sorry about the delete.
Sorry again Nathanael. I will upload a few with links for you to view
jimcseke (Over 1 year ago)
"Have a great weekend" and the "delete appology" was suppose to be my last sentence. How'd that otherlast sentence get in there?
Nathanael.Lawrence (Over 1 year ago)
No worries, mate. Family obviously comes first.

I still have some reservations about your particular approach of digitally rotating a single image. I actually wrote up a long post on my computer to try to straighten all my thoughts out and tried to explain exactly why I thought you were getting the results you were and why you might not actually get what you wanted from it in the end but I wanted to test some of my theories with actual tests with the synther and it behaved slightly differently than I had anticipated, so I'm glad I didn't post right away.
Nathanael.Lawrence (Over 1 year ago)
I still think that my fundamental objections to artificially retaining points in the pointcloud by causing the image to match itself are valid. 

I also have real issues with rotating raster images to begin with (being that it irreversibly jumbles the pixels unless used in increments of 90 degrees), not to mention editing copies of copies of images, especially when the images are JPEGs, means reapplying compression with each save, and thirdly I have a real dislike of magnifying pixels unless there is no other option. 

I have things to say about why it was possible to zoom further into your magnified crop than that portion of the original image, other methods to make Photosynth consider duplicate images as different files besides reshuffling the pixels, etc. but I'm going to hold off a bit more. 

I didn't want to just make a huge post just flaming everything about your discovery because there are some valid things that it was accomplishing, but I need more time.
jimcseke (Over 1 year ago)
You're a good man Nathanael. As alway I appreciate your input. Regardless of validity or not, having an open mind to new thoughts and ideas is exactly the reason we are able to chat back and forth in cyberspace (is that term still valid?),and do the very things we do on this site. And thankfully the open-minded people are winning most of the battles. And, with more individuals such as yourself who take the time to openly debate, I think we can keep the close-minded at bay.(For now.)
Wow, that's way too deep for this forum. Ah! what the heck., click the little green check icon and post it.
michaeldenis (Over 1 year ago)
Just to recap (now that we've lost what we're talking about here), jimcseke's method maximizes the density of the pointcloud image by "repeating" an image by *slightly* modifying its copies by cropping and zooming.  I was impressed with the point clouds he had made.  Good work jimcseke!

I seem to remember a synth of a baseball or football field that exemplified the idea of this method.  It took a single shot of a football field from a blimp, helicopter, or plane and then cropped into it to gain the "Photosynth effect."  In addition, the pointcloud was extremely dense for this synth.  It looked a lot like the wide image because of all of the overlaps between the more detailed crops.  I'll be interested to see how jimcseke continues his projects.  By the way, good discussion guys.  Your comments enliven the place.
Nathanael (Over 1 year ago)
I never have finished organising that post that I wrote offline, but I thought I'd post a link to the last version of it for the conversation's sake. You can find it here: 

It is actually the result of two different attempts to explain my thinking on this thread. I had moved most of my conversational rambling up into bullet point format, but I had taken different logical paths through the same material and it became too exhausting to sort it all out. I may finish cleaning it up sometime and post links to some of my follow up test synths, but I can't promise anything.

MichaelDenis, your last comment before this one really makes me smile every time that I read it. I assume that you're referring to the first link that I posted in my first post above. Was the thread really that long? ;)

Jim has since updated his ideas:
michaeldenis (Over 1 year ago)
Ha!  You're right, Nathanael.  Good old, flyphoto2020 and his amazing flying synths.  I could use one of those flying machines: to soar away from each of my synth subjects and provide such beautiful context.  Oh well.

Back onto topic, I can't help but be happy to see this thread re-emerge on top.  Mainly, I always thought you and jim were a bit bonkers with all this pointcloud capture obsession, and on the other hand, of course, I was eager to catch the latest developments your experiments.

Unfortunately, jim's latest opus has left me scratching my head more than reaching for my camera, so it's nice to look forward to a cogent write-up of the state of things in point capture land.

I've always been more directed towards using Photosynths as reality substitutes, rather than taking things to literally the next level by stripping away Photosynth's image-based interface.  That is the future.  I'll look forward to the read; thanks for writing.
jimcseke (Over 1 year ago)
Hey Nathanael,
Thank you for taking the time to set up this discussion. Above and beyond as always. Much appreciated. 
michaeldenis, I love your sense of humor. Glad to see you join in, if as only an observer, but glad none the less.
Well I guess we just sit and wait until more synthies give their 2 cents worth.
Again Nathanael, thank you.
Did anyone bring doughnuts? :)
Nathanael (Over 1 year ago)
Here are some simple illustration test synths to accompany my Dense Point Cloud Theory doc:

A demonstration of how cropping can be used to increase the density of an actual synth:

The original synth:


Unrelated: FlyPhoto2020 now goes by TriState360 and his synth which I first link to above has been re-uploaded at