PhotoSynthTileDownloader
In my previous post I have released my PhotoSynth Tookit but the PhotoSynth tile downloader wasn’t available yet. You can now download the picture of your Synth in HD using PhotoSynthTileDownloader1.zip. But please respect author’s copyright ! I have included a new confirmation dialog box that warn you about the Synth status (unlisted, public) but this is not shown in the video presentation of PhotoSynthTileDownloader.
BundlerMatcher
I have finally released BundlerMatcher, a feature extraction and picture matching tool built with SiftGPU. The main goal of this tool was to replace the slow matching step packaged with Bundler by a faster one using GPU without needing to modify Bundler’s code. You can download BundlerMatcher1.zip or checkout the code on my google code. Warning: this tool needs a 64bit windows OS (tested on 7 and vista) and a decent GPU. All my demos are available under MIT license but SiftGPU isn’t released under MIT so you should take a look at SiftGPU license.
OpenSynther
I’m proud to introduce OpenSynther which is under heavy development. The first goal of this tool is the same as BundlerMatcher: provide a faster matching engine for Structure from Motion tools. To achieve high performance it is coded in C++ and using both multi-core and GPU. Furthermore OpenSynther is using Surf instead of Sift and may in the future also compute the 3d reconstruction (as PhotoSynth does).
OpenSynther current feature list:
- Jpeg loading + Exif reading
- multi-threaded Surf feature extraction (based on OpenSurf & ParallelSurf)
- multi-threaded Surf feature matching (based on OpenSurf matching + Cuda GPU matching)
- Quadratic complexity O(n²) with n number of pictures
OpenSynther TODO:
- multi-threaded Surf feature approximate matching (based on FLANN)
- Linear complexity (hypothetical)
- integrate 3d reconstruction ?
Benchmarks
These benchmarks were done on an Intel Corei7 920 (8 cores @ 2.66Ghz) + Nvidia GTX 285. The pictures size were 1600×1200 (~2M pixels), I heard that it was the limit size at which PhotoSynth extract features (it can of course use bigger pictures but feature extraction are done at a max size of 2M pixels).
Note:
- Bundler: I had to reduce picture to 640×480, otherwise I got too many features (would be unfair).
- PhotoSynth: I didn’t take into account the time spent in tile creation.
- OpenSynther: in the future I may consider using GPUSurf for feature extraction.
- BundlerMatcher: to be fair with PhotoSynth I didn’t write the ascii key file (so in fact it’s slower).
Note:
- PhotoSynth: they did a very good job and managed to have a linear matching !
- OpenSynther: the GPU is in fact doing the most part of the job as you can see in the table below:
| App Name | PhotoSynth | OpenSynther | BundlerMatcher | Bundler | ||||||||
| Extraction in sec | 10 | 22 | 39 | 21 | 40 | 78 | 7 | 12 | 24 | 105 | 220 | – |
| Matching in sec | 15 | 31 | 58 | 31 | 104 | 322 | 40 | 170 | 542 | 75 | 440 | – |
| % matching done by CPU |
100 | 100 | 100 | 22 | 12 | 13 | 0 | 0 | 0 | 100 | 100 | – |
| % matching done by GPU |
0 | 0 | 0 | 78 | 88 | 87 | 100 | 100 | 100 | 0 | 0 | – |
| Nb pictures | 49 | 98 | 196 | 49 | 98 | 196 | 49 | 98 | 196 | 49 | 98 | 196 |
As you can see OpenSynther matching is in fact done by the GPU at 85%.
Great news! Keep up the awesome work.
By the way, i’ve been using your output from photosynthToolkit into linux 64 bit pmvs2, works great. It’s been very useful since Bundler output for me has been a complete disaster, haven’t been able to compile it in linux and windows ply output is totally off. Probably has to do with the focal extraction since my camera is not in the extractfocal.pl, but i’ve changed the pl file to include the ccd of my camera with no good results.
Will give BundlerMatcher a serious try. Can’t wait to take a look into Opensynther!
Tried BundlerMatcher. It is much faster indeed.
But, as before, i’m getting geometry wrong. The ply resulting file is useless, even though the ET example gives pretty good outcome (and comparatively fast!)
My camera is a Canon EOS rebel XS which is supposed to have 1/2.5″ ccd width. So what i’m doing is changing the exif tags to Canon Canon DIGITAL IXUS 800 IS which has the same ccd width. extractfocal.pl will accept the data under that tag but, as said before, the outcoming ply files are useless.
Somehow, photosynth does do it correctly, but bundler won’t.
I don’t know what’s missing, but i’ve had no luck with Bundler.
Just a thought, Henri:
PhotoSynthTileDownloader could read the .JSON file of the synth to obtain the Copyright or Creative Commons license for the photos of that synth and simply refuse to download photos with a Copyright license.
Most people use the default liberal Creative Commons license with their synths at this point, so there really shouldn’t be many problems, but this would save your tool some criticism from professional photographers who have already explicitly chosen the Copyright license.
@Nathanael: I’ve allowed the download of copyrighted synth because you may want to download your own synth (that could be copyrighted). And I think that people should be able to download all pictures anyway, they just should know that they have to respect the copyright.
@Henri, fair enough. ?
Is there any chance of getting PhotoSynthTileDownloader to resume a previously begun download, rather than completely restart from the beginning in the event that something interrupts the download the first time?
Thank you for addressing the copyright issue.
The ease of ripping the synths has caused me to hesitate to publish many locations I have done depending on the expense and effort it took to capture a location. I have already been told of cases of my other hdr & gigapixel photography getting ripped off and used in both major films and video games. I don’t doubt some people will take “shortcuts” with synths as well since it will be even easier to hide your source by changing the perspective it is rendered from.
Nate’s suggestion of blocking content that has a copyright is appealing, but the obvious problem is that someone who posts and copyrights their work may want to download their own data. I would fall into that category.
oops, I missed the updates by Nate and Henri from above. sorry for the redundancy.
Astre,
You had mentioned Insight3D and I had tested that a few weeks back. Do you know if that software is at a stand still or if there are any future developments planned?
I still want to chime and and make sure you don’t forget some of us (uh-hum) want the 64 bit Win version of the PMVS2 so “we” can do some really dense matching!
@Eugene: last update on Insight3D svn was 13 months ago, so I’m not sure that it is still under developments. But Insight3D has the cleanest code I saw which use SBA for structure from motion, so this is why I mentioned it. Concerning the win 64-bit version of PMVS2, this is not my top priority now, I’m working on OpenSynther for my outdoor tracking algorithm. But if you want a 64bit version of PMVS2 so badly, you can switch to the linux version anyway (Cesar Lopez mentioned that it was working). I’m sorry but I can’t help you more on that because I can’t work on all project at the same time and I first want to have some results with OpenSynther.
@Nathanael: I’ve added the resume function on my todo list, that should be easy to implement.
One question, Astre, Do you expect OpenSynther to generate output appropiate for CMVS?
@Cesar Lopez: if I manage to integrate Insight3D camera calibration code, I’ll generate the same output as Bundler. So it will be compatible with CMVS.
Astre, no problem on the 64 bit version. You have all these separate individual tools that seem related to one another. Do you plan at some point to put them all together and provide a complete workflow from photos to point cloud to exporting camera stations and camera parameters?
@Eugene: this is indeed the goal of OpenSynther, but if I don’t manage to integrate a sfm code (Insight3D or Bundler) I’ll keep only the matching step and provide a faster matching input for Bundler.
Release early, release often
Regarding BundlerMatcher have you looked at allowing it to swap instead of relying only on RAM I would love to run it on very large image sets, I am running out of RAM on just over 1000 images. Got access to computers with 16GB RAM. Or is this something that will be limited to RAM only.
Got a question on running it on computers with ATI cards as well, I am getting
“Fragmented shader(s) failed to link, no vertex shader(s) defined.
Fragment Shader not supported by HW
Error : can not initialize opengl context for SiftGPU”
Just wondering if you have an idea, as you mention it should run on both Nvidia and ATI cards.
Using PhotoSynthTileDownloader I noticed it stops very often downloading and says its done, it would be so fantastic to have a resume download function where it looks into the folder for already downloaded tiles.
Again thanks a lot for your hard and fast work, really look forward seeing some more on this blog.
@Olafur Haraldsson: I’m not sure to understand: BundlerMatcher was running out of memory with a 16Gb computer ? Or you are willing to give it a try on such a computer ? Anyway it should indeed only be limited by available ram. Swapping to the disk could be doable but would slow down the process a lot. So I won’t implement it right now as OpenSynther is kinda time consuming
Concerning your ATI issue: I don’t own an ATI card so I can’t debug this (but according to SiftGPU author, it should works with an ATI card).
All right, I’ll implement the resume feature (you are now 2 waiting for this) !
I ran out of memory on a computer with 12GB and have access to 16GB ram computer. so I do have a little more extra I could use. But I would love to be able to run it on larger sets, someway or another. Some way to batch it (break it up) then merge it together in the end. I guess the answer is simply more RAM.
One other things I forgot to ask you about, when I run images through BundlerMatcher at lets say a 4K size per image it extracts maybe about 2.000-5.000 features max per image. But if I hand it 1.5K same images just down-scaled it finds about 15.000-30.000 features per image. Is there some kind of size limit for the SiftGPU that you know of…
@Olafur Haraldsson: I understand now how you manage to eat 12GB
! Extracting more than 6000 features per picture is useless, I recommend you to have around 4000 features. I’m not aware of SiftGPU size limits but generally GPU doesn’t handle well picture larger than 4096px. To reduce the number of feature found you have 2 possibilities: downscale the picture (1600×1200 is generally enough) or augment the last parameters in BundlerMatcher (firstOctave). I’m curious about your motivation in using BundlerMatcher with that much pictures: are you using CMVS after Bundler to get dense point cloud ? If not, why aren’t you using PhotoSynth ?
I am indeed using CMVS due to the problems I have with PMVS, I have managed to compile a 64 bit version of PMVS under Ubuntu, however I still run out of RAM there as well. Therefore again I am using CMVS. Photosynth does run out of memory as well and output does not support CMVS as you well know. I will play around with these 2 settings and report back.
What is the reason for your default value of -1 for firstOctave? Where you getting better results, I do wonder why I was getting up to 30.000 features with that setting.
Again, yes I did reduce the size from around 4000px down to 1536px and the difference was from 2.000-5.000 features with 4000px up to 15.000-30.000 features with the 1536px. I do find it curious down-sampling creates more features per image. But I will try and play with the settings like you suggested and see how many I can process.
@Olafur Haraldsson: this is indeed a strange behavior. -1 for the firstOctave parameters means that SiftGPU is upsampling the first octave to extract feature on the first octave. I’ve used this parameter value for the BundlerMatcher package because otherwise i didn’t get enough feature for the ET sample packaged with Bundler. But you shouldn’t really use this value width 4000px pictures: 0 or 1 or even 2 should give more predictable behavior. Good luck with your 1000 pictures dataset !
I have gotten blue-screen 2 times on the 2140 image data set, it does not seem to be running out of RAM so I wonder what it is. Has anyone else seen this behavior?
Astre,
I am not sure if this was just posted, but when I tried to download the file (several times now over a 15 minute period), it says it’s not found.
Thanks for working this out!
Sorry, my last post may not have been clear. I was referring to the 64 bit version of PMVS2
@Eugene: … maybe it’s because you’ve posted your comments in the wrong post
Just kidding! I’ve downloaded the 64 bits version right now and it’s working without any trouble.
i’m a bit puzzled: your bundlermatcher requires windows? i use windows and was pretty annoyed to need linux for bundler and thus must run it via cygwin.
how does now bundlermatch fit into this picture?
or is there a way to run the full set of bundler steps directly in windows?
thanks in advance,
daniel / sirleto
@daniel: for now BundlerMatcher still needs cygwin (because of perl script used to extract focal parameters using jhead). But I already have a working version that runs without cygwin, I just needs to make it more robust. The new version will provide a full Bundler toolchain: BundlerMatcher, Bundler, CMVS, PMVS2 x64 and a Bundler viewer using Ogre3D).
oh, great!
thats exactly what i was waiting for. if you need some help finishing it (tests, etc.), feel free to email me.
best,
daniel / sirleto
@daniel: there is a beta available on : http://www.pgrammetry.com/forum/index.php -> SFMToolkit -> First beta available.
oh, great! (you could’ve told me sooner via email
will give it a try (hopefully next weekend)!
give it one test run – sorry, had not had more time – and it crashed. (i did not even have enough time to check what the problem seems to be – i came back later when the crash did already happen).
i will try to give you more details,
the cpu is a core i7 and the gpu a nvida gta 460 – so these two should not create any trouble – the os is windows 7 64bit, may this cause problems?
atleast the first “steps” seemed to run fine, it was one of the later ones that crashed …
@daniel: it’s working fine for me with a Corei7 and an Nvidia GTX470… so I need indeed more information: what is crashing ? what the error message ?
sorry, working currently full weekends – release schedule tightening
will mail you as soon as i get time to test it thoroughly and report errors.
I have tried bundle matcher with 150 images of 5184*3456 pixel size. it gave error message as too many features, reallocate texture. so i have tried with a small group of image (10 image) and also reduced the resolution to 1024*683. But it still give the same error. How can i solve this? (16Gb RAM and 1Gb GPU – windows 7 OS)
@swee: this is more a warning than an error: the matching process should continue even if you can see this message.