This page was last updated March 30th, 2016.
By utilizing the Eigen and Ceres Solver libraries in addition to custom scalable algorithms, Theia achieves state-of-the-art performance in terms of efficiency and accuracy on large-scale datasets. We measure the performance of Theia in terms of efficiency and accuracy on benchmark datasets for small and large-scale problems to provide meaningful context for the strengths and weaknesses of the library.
Small Dataset Benchmarks¶
To demonstrate the viability of Theia for small scenes, we measure the performance on the Strecha MVS datasets datasets. These datasets consist of small scale scenes, though they are extremely high-resolution and rather dense image sampling so it may be considered an “easy” dataset by some. Nonetheless, it is a commonly used dataset to measure reconstruction accuracy.
All reconstructions were generated using the parameters found in this configuration file. Each reconstruction generated from this file was then compared to the ground truth reconstruction (which can be generated from the Strecha dataset using the “create_reconstruction_from_strecha_dataset.cc” program). The camera intrinsic calibration files can be easily generated from the information provided by datasets.
|Dataset||N (input)||N||Median Error (mm)||Mean Error (mm)||Timing (s)|
Large Scale Benchmarks¶
We use the 1DSfM Datasets as benchmarks for large-scale reconstructions. These datasets provide 2-view matches and epipolar geometry as input, and a reference reconstruction from incremental SfM (computed with Bundler) for measuring error. The reference reconstruction is not necessarily considered ground truth, but it is a meaningful reference point since incremental SfM algorithms are rather robust and accurate.
We measure the accuracy of camera positions (approximately in meters) and timing results. We report N, the number of cameras that could be succesfully reconstructed, in addition to the mean and median camera position errors after robust alignment (via RANSAC) to ground truth camera positions.
All reconstructions were generated using the “build_1dsfm_reconstruction.cc” program and the parameters found in this config file. Each reconstruction generated was then compared to the ground truth reconstruction provided in the 1dSfM dataset (these are provided as Bundler files, but can be converted to Theia reconstructions with the “convert_bundle_file.cc” program).
|Dataset||N (input)||N||Median Error (m)||Mean Error (m)|
|Piazza del Popolo||328||326||1.03||3.91|
|Tower of London||572||456||1.36||17.38|
|Piazza del Popolo||328||0.47||8.37||46.30||61.31|
|Tower of London||572||0.47||8.03||129.65||154.45|