Reviewing my implementation design choices.

last week I spoke with Arnoud. One idea was that Bleser et al.’s approach might be outdated, as it comes from 2008. Back when I started with my thesis, that was „fresh enough”, but now it is 10 years old and a lot has happened in the field of tracking for both virtual and augmented reality applications. So, let’s do some literature research again.

The focus of this part of the literature study is recent developments in vision-hybrid pose tracking systems. „Recent” means that I will focus on papers from 2014 up to now. „Vision-hybrid” indicates systems using multiple sensors, of at least one camera. This camera might work with infrared, optical light, or structured light approaches. „Pose tracking systems” are those systems that estimate the 6D pose (3D position and orientation) over time. This excludes one-shot localization systems.

Billinghurst et al. is a big survey from 2014 over the complete width of AR technologies. It subdivides AR tracking into methodologies based on the sensor used: magnetic, vision, inertial, GPS, and hybrid for sensor-fusion systems. It does not include technical depth of these approaches.

In their semi-final chapter, „Research Directions”, they state that „Tracking is one of the most popular areas for AR research because it is one of the most important of the AR enabling technologies. […] However, there is still significant work that needs to be done before the vision of ‚anywhere augmentation’ is achieved, where users will be able to have a compelling AR experience in any environment.”. They list some robust indoor tracking methods, includng a method with ceiling-mounted ultra-sounic receivers by Dissanayake et al. in 2001; a method where camera images are combined with a 2D laser tracker for range finding [Scheer and Müller, 2012]; „computer vision trakcing from markers pasted over the walls [Wagner, 2002, Schmalstieg and Wagner, 2007]” and „hybrid systems with computer vision, inertial sensors and ultra-wide-band tracking [Newman et al., 2006] among others”.

They note that handheld depth sensors for indoor tracking seem very promising, such as The Google Tango Project and Intel’s RealSense. I would say that these devices are not yet small enough to be embedded in consumer devices to be widely used.

Another challenge this paper observes is seamless switching „from” (why not bothways „between”?) outdoor to indoor tracking environments. All cited sources in this section use at least GPS, with only [Piekarski et al., 2003] fusing that with vision. This brings them to 20 cm precision, too low in my case.

As a follow up on Billinghurst et al., I looked at Google Tango. Google has ended support for Tango on 1 March 2018, in favor for Google ARCore. Looking at the required permissions, ARCore only uses the camera, and requires no other sensor permissions.

Related to ARCore is Apple’s ARKit. I need to investigation that API.

Bloesch et al. presented a visual-inertial tracking method using the EKF. It might be a proper alternative for Bleser’s models, although I think Bleser et al. have been a bit more precise. Compare Bleser’s models to the following:

with coordinate frames at a global world coordinate system \(\mathcal W\), robot’s center \(\mathcal R\), IMU center \(\mathcal S\) and camera center \(\mathcal C\), and (quoted from Bloesch et al., replacing their notation with Bleser’s):

• \(\mathbf{s}_{rs,t}\): robocentric position of IMU (expressed in \(\mathcal S\))
• \(\mathbf{v}_{rs,t}\): robocentric velocity of IMU (expressed in \(\mathcal S\))
• \(\mathbf{q}_{sw,t}\): attitude of IMU (map from \(\mathcal S\) to \(\mathcal W\))
• \(\mathbf{b}^a_{s,t}\): additive bias on accelerometer (expressed in \(\mathcal S\))
• \(\mathbf{b}^\omega_{s,t}\): additive bias on gyroscope (expressed in \(mathcal S\))
• \(\mathbf{c}_{s,t}\): translational part of IMU-camera extrinsics (expressed in \(\mathcal S\))
• \(\mathbf{z}_{sc,t}\): rotational part of IMU-camera extrinsics (map from \(\mathcal S\) to \(\mathcal C\))
• \(\mathbf{\mu}^{(i)}_{c,t}\): bearing vector to feature \(i\) (expressed in \(\mathcal C\))
• \(\rho^{(i)}_{c,t}\): distance parameter of feature \(i\).

Their prediction model is given below as a differential equation, with \(\mathbf{u}_t^T = \begin{bmatrix} \mathbf{y}^\omega_{s,t} & \mathbf{y}^a_{s,t} \end{bmatrix}\):

with:

• superscript \(\mbox{}^\times\) denoting the skew symmetric matrix of a vector,
• \(\mathbf{N}^T(\mathbf{\mu}^{(i)}_{c,t})\) linearly projects a 3D vector onto the 2D tangent space around the bearing vector \(\mathbf{\mu}^{(i)}_{c,t}\),
• \(d(\rho^{(i)}_{c,t}) = \frac{1}{\rho^{(i)}_{c,t}}\) model of that distance parameter \(\rho^{(i)}_{c,t}\),
• corrected measurements \(\bar{\mathbf{y}}^a_{s,t} = \mathbf{y}^a_{s,t} - \mathbf{b}^a_{s,t} - \mathbf{v}^a_{s,t}\) and \(\bar{\mathbf{y}}^\omega_{s,t} = \mathbf{y}^\omega_{s,t} - \mathbf{b}^\omega_{s,t} - \mathbf{v}^\omega_{s,t}\).
• \(\mathbf{g}_{w}\): gravity vector

One thing that pops out is that they do not take into account variable intervals between updates. In other words, \(T = 1\).

I need to study their observation model further.

Mur-Artal is the main author on a series on monocular, stereo and RGB-D and DBoW2 Place recognizer ORB-SLAM papers. Another addition to that series by Mur-Artal and Tardós integrates IMU information for accurate tracking. Nothing seems to indicate that this system is limited to the robotics settings (where estimates of a control vector are reliable), and it seems equally applicable for augmented reality. In the conclusion of IMU they state:

We have presented a novel tightly coupled Visual-Inertial SLAM system, that is able to close loops in real-time and localize the sensor reusing the map in already mapped areas. This allows to achieve a zero-drift localization, in contrast to visual odometry approaches where drift grows unbounded. The experiments show that our monocular SLAM recovers metric scale with high precision, and achieves better accuracy than the state-of-the-art in stereo visual-inertial odometry when continually localizing in the same environment. We consider this zero-drift localization of particular interest for virtual/augmented reality systems, where the predicted user viewpoint must not drift when the user operates in the same workspace. Moreover we expect to achieve better accuracy and robustness by using stereo or RGB-D cameras, which would also simplify IMU initialization as scale is known. The main weakness of our proposed IMU initialization is that it relies on the initialization of the monocular SLAM. We would like to investigate the use of the gyroscope to make the monocular initialization faster and more robust.

Bold is mine. First bold comment is done in their final paper from this series, ORB-SLAM2, but no simplification of IMU initialisation is presented. The second bold comment can be investigated further.

It is a resource-intense method, using three threads for tracking, localization and loop closing, but that should not limit its applicability.

TODO

1. Investigate what Apple’s ARKit actually does. Does it use sensor fusion? Did they reveal the papers that inspired their methods?
2. Is combining the inertial and stereo/RGB-D ORB-SLAM2 interesting enough?