☎

Experiment design

After the talk with all supervisors, one of the biggest questions is how the system evaluation will be performed. A small discussion on AR system evaluation seems appropriate.

Previously, I mentioned the systems of Bleser et al. and Ta et al.. While I like certain aspects of their approach, something certainly struck me while reading more and more AR papers; system evaluation is hard and often slightly neglected.

As an objective measure, Bleser et al. evaluate their system by reporting the root mean square prediction error in pixels for their different sensor fusion models, and the standard deviation. But for as far as I can tell, this is based on estimated errors of the system, and not on measurements of an external system/observer.

Ta et al. have a different approach. They plot the estimated path as an overlay on the ground floor of the environment. I take this as a valid approach for them; for most purposes, a robot does not need to know a sub-milimeter precise location. However, it is hard to deduce a quantitative evaluation from this map.

References

  • Gabriele Bleser and Didier Stricker. Advanced tracking through efficient image processing and visual-inertial sensor fusion. In Virtual Reality Conference, pages 137–144. IEEE, 2008. [ bib | DOI ]
    
@inproceedings{bleser2008advanced,
  author = {Bleser, Gabriele and Stricker, Didier},
  booktitle = {Virtual Reality Conference},
  title = {Advanced tracking through efficient image processing and visual-inertial sensor fusion},
  year = {2008},
  pages = {137--144},
  organization = {IEEE},
  doi = {10.1109/VR.2008.4480765}
}
  • Duy-Nguyen Ta, Kyel Ok, and Frank Dellaert. Monocular parallel tracking and mapping with odometry fusion for MAV navigation in feature-lacking environments. In IEEE/RSJ IROS'13 International Workshop on Vision-based Closed-Loop Control and Navigation of Micro Helicopters in GPS-denied Environments, November 2013. [ bib | 1585 ]
    
@inproceedings{ta2013monocular,
  author = {Duy-Nguyen Ta and Kyel Ok and Dellaert, Frank},
  title = {Monocular Parallel Tracking and Mapping with Odometry Fusion for {MAV} Navigation in Feature-lacking Environments},
  booktitle = { {IEEE/RSJ IROS}'13 International Workshop on Vision-based Closed-Loop Control and Navigation of Micro Helicopters in {GPS}-denied Environments},
  location = {Tokyo, Japan},
  month = nov,
  year = {2013},
  url = {http://hdl.handle.net/1853/51585}
}