I love projecting the photos onto geometry, but although the outlines around an object sometimes surprise me with how close they are, they are usually very rough.
In research from years ago, by having a user highlight the edges of an object, the demonstrated software was able to generate a nearly perfect matte differentiating foreground from background.
(See a video here: http://vis.berkeley.edu/papers/softscissors/ )
You are already using the point cloud as a signal for which edges to cut along for your geometry. Couldn't you use a more sophisticated edge detecting algorithm like the one above?
I look at the foreground on many of the new synths and all I can think is how much better it would look (or how much less awful, depending on your outlook) if the moss on the branches in the forest or the pine trees by the tent on the mountain were properly matted out from the background instead of crudely cropped out.