Multiple ORBSLAM3 runs of `mono_euroc` on EuRoC MH01 with same params

12 May 2021

thesis

There seems to be some variation occurring in running ORBSLAM3 with similar arguments. Let’s see what the noise is.

In this experiment, we will run ORBSLAM3 multiple times on the same EuRoC datasets. We will then analyse the results with the provided evaluation/evaluate_ate_scale.py script, and report the differences here. In rows „Paper”, I’ll report the information from the published paper. In rows labeled „#1”, I’ll report the results generated in the previous post. Data for rows „#2” to „#6” are generated for this experiment, to test for noise.

The mean, median, standard deviation, minimum error and maximum error colums are expressed as absolute numbers, and as percentages of the RMS ATE. Providing the percentages enables the reader to make comparisons between rows within a table. All absolute numbers are based on meters.

What I hope to see is that the mean/median/std/min/max would change linearly in terms of the RMS ATE, and that the variation might be blamed by scale errors. The monocular scenarios seem to hint on that. However, the same analysis on the other scenarios do not display the same consistency.

Monocular
- MH01
- MH02
- MH03
- MH04
- MH05
- V101
- V102
- V103
- V201
- V202
- V203
Stereo
- MH01
- MH02
- MH03
- MH04
- MH05
- V101
- V102
- V103
- V201
- V202
- V203
Monocular inertial
- MH01
- MH02
- MH03
- MH04
- MH05
- V101
- V102
- V103
- V201
- V202
- V203
Stereo inertial
- MH01
- MH02
- MH03
- MH04
- MH05
- V101
- V102
- V103
- V201
- V202
- V203

Monocular

MH01

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.017	-	-	-	-	-	-	-	-	-	-
#1	1.152231	1.059942	91.99%	1.185462	102.88%	0.451840	39.21%	0.238597	20.71%	2.000678	173.64%
#2	3.393642	3.123164	92.03%	3.494480	102.97%	1.327650	39.12%	0.731295	21.55%	5.898523	173.81%
#3	3.367350	3.099083	92.03%	3.467830	102.98%	1.317092	39.11%	0.725999	21.56%	5.851425	173.77%
#4	2.366724	2.176176	91.95%	2.424396	102.44%	0.930398	39.31%	0.518139	21.89%	4.093447	172.96%
#5	2.987140	2.749144	92.03%	3.073724	102.90%	1.168424	39.12%	0.641827	21.49%	5.181438	173.46%
#6	2.988189	2.749682	92.02%	3.073283	102.85%	1.169839	39.15%	0.639896	21.41%	5.181658	173.40%

MH02

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.014	-	-	-	-	-	-	-	-	-	-
#1	6.040316	5.478086	90.69%	6.453422	106.84%	2.544797	42.13%	0.321352	5.32%	10.192074	168.73%
#2	6.131629	5.558318	90.65%	6.532935	106.54%	2.588817	42.22%	0.368228	6.01%	10.294247	167.89%
#3	2.086861	1.887522	90.45%	2.203295	105.58%	0.890084	42.65%	0.125153	6.00%	3.484115	166.95%
#4	1.582292	1.431296	90.46%	1.674025	105.80%	0.674567	42.63%	0.082394	5.21%	2.647174	167.30%
#5	4.164156	3.861103	92.72%	3.561454	85.53%	1.559513	37.45%	0.335164	8.05%	6.504753	156.21%
#6	0.611884	0.551016	90.05%	0.634668	103.72%	0.266052	43.48%	0.028194	4.61%	1.029154	168.19%

MH03

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.031	-	-	-	-	-	-	-	-	-	-
#1	2.532367	2.300038	90.83%	2.133642	84.25%	1.059578	41.84%	0.170972	6.75%	6.324798	249.76%
#2	0.655750	0.595135	90.76%	0.552682	84.28%	0.275360	41.99%	0.027516	4.20%	1.680590	256.29%
#3	9.631822	8.756025	90.91%	8.084771	83.94%	4.012982	41.66%	0.656701	6.82%	23.779104	246.88%
#4	1.374518	1.247061	90.73%	1.138974	82.86%	0.578047	42.05%	0.080078	5.83%	3.439590	250.24%
#5	4.501101	4.094876	90.97%	3.821999	84.91%	1.868663	41.52%	0.288191	6.40%	11.133745	247.36%
#6	6.576417	5.983288	90.98%	5.579440	84.84%	2.729382	41.50%	0.404369	6.15%	16.201346	246.36%

MH04

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.066	-	-	-	-	-	-	-	-	-	-
#1	0.278285	0.244803	87.97%	0.202243	72.67%	0.132339	47.56%	0.061703	22.17%	0.624734	224.49%
#2	4.884112	4.358938	89.25%	3.397331	69.56%	2.203227	45.11%	0.895398	18.33%	9.391822	192.29%
#3	9.847925	8.805982	89.42%	6.824810	69.30%	4.408663	44.77%	1.758954	17.86%	19.013275	193.07%
#4	11.287675	10.116204	89.62%	8.078140	71.57%	5.007398	44.36%	2.215250	19.63%	21.371842	189.34%
#5	0.418444	0.363307	86.82%	0.287109	68.61%	0.207614	49.62%	0.127697	30.52%	1.029064	245.93%
#6	8.662571	7.669742	88.54%	5.661755	65.36%	4.026810	46.49%	1.495891	17.27%	16.843464	194.44%

MH05

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.044	-	-	-	-	-	-	-	-	-	-
#1	2.079829	1.861468	89.50%	1.736438	83.49%	0.927698	44.60%	0.180973	8.70%	3.990731	191.88%
#2	8.392823	7.530071	89.72%	6.872701	81.89%	3.706416	44.16%	1.776133	21.16%	15.707105	187.15%
#3	13.788697	12.363484	89.66%	11.403111	82.70%	6.105116	44.28%	1.293492	9.38%	25.953673	188.22%
#4	13.221017	11.851884	89.64%	10.930936	82.68%	5.859023	44.32%	1.162522	8.79%	24.858435	188.02%
#5	5.351978	4.796701	89.62%	4.415727	82.51%	2.373884	44.36%	0.448257	8.38%	10.132903	189.33%
#6	18.216040	16.313811	89.56%	15.031308	82.52%	8.104547	44.49%	1.545573	8.48%	34.740511	190.71%

V101

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.033	-	-	-	-	-	-	-	-	-	-
#1	0.999117	0.911711	91.25%	0.931047	93.19%	0.408678	40.90%	0.113726	11.38%	1.851312	185.29%
#2	0.986085	0.899916	91.26%	0.919913	93.29%	0.403130	40.88%	0.112684	11.43%	1.825103	185.09%
#3	0.996911	0.909765	91.26%	0.929524	93.24%	0.407624	40.89%	0.115657	11.60%	1.844356	185.01%
#4	0.995759	0.908653	91.25%	0.929462	93.34%	0.407291	40.90%	0.116122	11.66%	1.842985	185.08%
#5	1.003113	0.914896	91.21%	0.933999	93.11%	0.411341	41.01%	0.113959	11.36%	1.854899	184.91%
#6	0.986760	0.900323	91.24%	0.919214	93.15%	0.403873	40.93%	0.112957	11.45%	1.826076	185.06%

V102

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.016	-	-	-	-	-	-	-	-	-	-
#1	10.211572	9.514054	93.17%	9.499267	93.02%	3.709312	36.32%	0.246598	2.41%	18.747270	183.59%
#2	15.552612	14.476459	93.08%	14.400741	92.59%	5.684707	36.55%	0.666041	4.28%	28.654867	184.24%
#3	14.749435	13.737878	93.14%	13.697747	92.87%	5.368103	36.40%	0.280025	1.90%	27.200777	184.42%
#4	18.831797	17.539681	93.14%	17.372531	92.25%	6.855376	36.40%	0.439138	2.33%	34.616715	183.82%
#5	11.582065	10.786769	93.13%	10.746202	92.78%	4.217802	36.42%	0.261647	2.26%	21.295688	183.87%
#6	10.593163	9.872335	93.20%	9.692751	91.50%	3.840847	36.26%	0.407990	3.85%	19.280299	182.01%

V103

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.037	-	-	-	-	-	-	-	-	-	-
#1	0.772792	0.690450	89.34%	0.616459	79.77%	0.347110	44.92%	0.098469	12.74%	1.508639	195.22%
#2	0.926426	0.825022	89.05%	0.736283	79.48%	0.421431	45.49%	0.113477	12.25%	1.826568	197.16%
#3	0.999168	0.901993	90.27%	0.838491	83.92%	0.429822	43.02%	0.121647	12.17%	1.974418	197.61%
#4	0.814266	0.728885	89.51%	0.653619	80.27%	0.362982	44.58%	0.109272	13.42%	1.587317	194.94%
#5	0.871117	0.777627	89.27%	0.697344	80.05%	0.392608	45.07%	0.114845	13.18%	1.714302	196.79%
#6	0.697906	0.625055	89.56%	0.561105	80.40%	0.310450	44.48%	0.087168	12.49%	1.354062	194.02%

V201

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.021	-	-	-	-	-	-	-	-	-	-
#1	1.462772	1.418872	97.00%	1.363137	93.19%	0.355673	24.31%	0.735491	50.28%	2.263611	154.75%
#2	0.948541	0.919790	96.97%	0.883955	93.19%	0.231768	24.43%	0.474893	50.07%	1.466752	154.63%
#3	0.934954	0.906635	96.97%	0.871114	93.17%	0.228364	24.43%	0.463328	49.56%	1.447335	154.80%
#4	0.943776	0.915380	96.99%	0.879204	93.16%	0.229766	24.35%	0.469502	49.75%	1.459138	154.61%
#5	0.937475	0.909304	97.00%	0.872813	93.10%	0.228090	24.33%	0.469872	50.12%	1.450201	154.69%
#6	0.956372	0.927526	96.98%	0.890070	93.07%	0.233113	24.37%	0.470079	49.15%	1.476676	154.40%

V202

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.022	-	-	-	-	-	-	-	-	-	-
#1	2.250168	2.112660	93.89%	2.174163	96.62%	0.774549	34.42%	0.264579	11.76%	3.749173	166.62%
#2	0.856666	0.463127	54.06%	0.343302	40.07%	0.720687	84.13%	0.095174	11.11%	7.848757	916.20%
#3	1.044070	0.834646	79.94%	0.734199	70.32%	0.627254	60.08%	0.182948	17.52%	5.176099	495.76%
#4	0.713411	0.668880	93.76%	0.681166	95.48%	0.248103	34.78%	0.022354	3.13%	1.186038	166.25%
#5	9.692430	9.110235	93.99%	9.419850	97.19%	3.308598	34.14%	1.234334	12.74%	16.127159	166.39%
#6	0.846450	0.591494	69.88%	0.504687	59.62%	0.605486	71.53%	0.037977	4.49%	6.236230	736.75%

V203

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	-	-	-	-	-	-	-	-	-	-	-
#1	0.952695	0.873713	91.71%	0.861690	90.45%	0.379807	39.87%	0.161705	16.97%	1.831130	192.21%
#2	0.836914	0.756478	90.39%	0.731555	87.41%	0.358004	42.78%	0.133100	15.90%	1.933404	231.02%
#3	0.866973	0.790264	91.15%	0.769099	88.71%	0.356545	41.13%	0.146864	16.94%	1.734405	200.05%
#4	0.905904	0.830651	91.69%	0.818039	90.30%	0.361498	39.90%	0.154277	17.03%	1.776229	196.07%
#5	0.886416	0.804502	90.76%	0.785623	88.63%	0.372169	41.99%	0.090518	10.21%	1.764058	199.01%
#6	0.630023	0.577353	91.64%	0.576421	91.49%	0.252176	40.03%	0.091639	14.55%	1.714085	272.07%

Stereo

MH01

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.025	-	-	-	-	-	-	-	-	-	-
#1	0.043952	0.040174	91.40%	0.046375	105.51%	0.017827	40.56%	0.003390	7.71%	0.080410	182.95%
#2	0.036767	0.033707	91.68%	0.036525	99.34%	0.014684	39.94%	0.000860	2.34%	0.070428	191.55%
#3	0.029278	0.026801	91.54%	0.030729	104.95%	0.011787	40.26%	0.000114	0.39%	0.063684	217.51%
#4	0.026547	0.024449	92.10%	0.026596	100.18%	0.010342	38.96%	0.004086	15.39%	0.064030	241.20%
#5	0.029209	0.026852	91.93%	0.027881	95.45%	0.011497	39.36%	0.001001	3.43%	0.057717	197.60%
#6	0.030022	0.027574	91.85%	0.030190	100.56%	0.011873	39.55%	0.000654	2.18%	0.065001	216.51%

MH02

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.022	-	-	-	-	-	-	-	-	-	-
#1	0.026427	0.022936	86.79%	0.019489	73.75%	0.013127	49.67%	0.001532	5.80%	0.079795	301.95%
#2	0.021561	0.018731	86.87%	0.015537	72.06%	0.010680	49.53%	0.001266	5.87%	0.097317	451.35%
#3	0.018642	0.016822	90.23%	0.014899	79.92%	0.008036	43.11%	0.001636	8.77%	0.050326	269.95%
#4	0.022824	0.020182	88.42%	0.016910	74.09%	0.010659	46.70%	0.001362	5.97%	0.064902	284.36%
#5	0.017622	0.016093	91.33%	0.014128	80.17%	0.007179	40.74%	0.001471	8.35%	0.054949	311.83%
#6	0.018619	0.016518	88.71%	0.013756	73.88%	0.008593	46.15%	0.000670	3.60%	0.056752	304.81%

MH03

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.027	-	-	-	-	-	-	-	-	-	-
#1	0.025911	0.022125	85.39%	0.017550	67.73%	0.013486	52.05%	0.001655	6.39%	0.119099	459.65%
#2	0.031740	0.027021	85.13%	0.023129	72.87%	0.016651	52.46%	0.000701	2.21%	0.192969	607.97%
#3	0.035092	0.022438	63.94%	0.015949	45.45%	0.026982	76.89%	0.000863	2.46%	0.213402	608.12%
#4	0.030860	0.026947	87.32%	0.025059	81.20%	0.015039	48.73%	0.001742	5.64%	0.101618	329.29%
#5	0.026519	0.021842	82.36%	0.015998	60.33%	0.015039	56.71%	0.001836	6.92%	0.091513	345.09%
#6	0.026957	0.023482	87.11%	0.019242	71.38%	0.013239	49.11%	0.001530	5.67%	0.107764	399.77%

MH04

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.089	-	-	-	-	-	-	-	-	-	-
#1	0.140880	0.132633	94.15%	0.149731	106.28%	0.047492	33.71%	0.024947	17.71%	0.252013	178.89%
#2	0.108798	0.102045	93.79%	0.088641	81.47%	0.037732	34.68%	0.031726	29.16%	0.217211	199.65%
#3	0.105818	0.095700	90.44%	0.091478	86.45%	0.045155	42.67%	0.016591	15.68%	0.255129	241.10%
#4	0.062272	0.056092	90.08%	0.046211	74.21%	0.027046	43.43%	0.009529	15.30%	0.216785	348.13%
#5	0.167465	0.158262	94.50%	0.179872	107.41%	0.054751	32.69%	0.036752	21.95%	0.334245	199.59%
#6	0.098254	0.092599	94.25%	0.079852	81.27%	0.032851	33.43%	0.028924	29.44%	0.241760	246.06%

MH05

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.058	-	-	-	-	-	-	-	-	-	-
#1	0.043626	0.036605	83.91%	0.030049	68.88%	0.023733	54.40%	0.002438	5.59%	0.133921	306.98%
#2	0.048248	0.041354	85.71%	0.034368	71.23%	0.024853	51.51%	0.003828	7.93%	0.137719	285.44%
#3	0.055547	0.047131	84.85%	0.038068	68.53%	0.029396	52.92%	0.001928	3.47%	0.203519	366.39%
#4	0.046779	0.039573	84.60%	0.032492	69.46%	0.024945	53.33%	0.003624	7.75%	0.137858	294.70%
#5	0.051432	0.042249	82.15%	0.035765	69.54%	0.029331	57.03%	0.002661	5.17%	0.148105	287.96%
#6	0.049461	0.043202	87.35%	0.037509	75.83%	0.024082	48.69%	0.001850	3.74%	0.144932	293.02%

V101

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.035	-	-	-	-	-	-	-	-	-	-
#1	0.034244	0.031838	92.98%	0.031596	92.27%	0.012607	36.82%	0.002576	7.52%	0.070539	205.99%
#2	0.035947	0.033697	93.74%	0.033531	93.28%	0.012517	34.82%	0.001914	5.32%	0.079815	222.04%
#3	0.035595	0.033204	93.29%	0.032900	92.43%	0.012823	36.03%	0.002045	5.75%	0.078728	221.18%
#4	0.035653	0.033531	94.05%	0.034485	96.72%	0.012117	33.98%	0.003590	10.07%	0.073212	205.34%
#5	0.035509	0.033169	93.41%	0.033542	94.46%	0.012677	35.70%	0.002951	8.31%	0.072046	202.89%
#6	0.036833	0.034420	93.45%	0.033807	91.78%	0.013113	35.60%	0.004678	12.70%	0.094891	257.62%

V102

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.021	-	-	-	-	-	-	-	-	-	-
#1	0.020474	0.018529	90.50%	0.016937	82.72%	0.008709	42.54%	0.002414	11.79%	0.052816	257.97%
#2	0.025233	0.020172	79.94%	0.017100	67.77%	0.015159	60.07%	0.001927	7.64%	0.210832	835.53%
#3	0.024184	0.021186	87.60%	0.018938	78.31%	0.011662	48.22%	0.002227	9.21%	0.127612	527.67%
#4	0.022095	0.019807	89.64%	0.019129	86.58%	0.009792	44.32%	0.001326	6.00%	0.055129	249.51%
#5	0.022914	0.020313	88.65%	0.017923	78.22%	0.010603	46.27%	0.001346	5.87%	0.099498	434.22%
#6	0.023223	0.020567	88.56%	0.018974	81.70%	0.010785	46.44%	0.001487	6.40%	0.134451	578.95%

V103

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.049	-	-	-	-	-	-	-	-	-	-
#1	0.045956	0.036320	79.03%	0.025014	54.43%	0.028158	61.27%	0.003225	7.02%	0.211876	461.04%
#2	0.112539	0.099553	88.46%	0.101154	89.88%	0.052480	46.63%	0.009814	8.72%	0.438102	389.29%
#3	0.100634	0.070851	70.40%	0.051021	50.70%	0.071466	71.02%	0.003769	3.75%	0.393342	390.86%
#4	0.067465	0.055579	82.38%	0.046942	69.58%	0.038243	56.69%	0.004029	5.97%	0.437921	649.11%
#5	0.062202	0.045230	72.72%	0.027353	43.97%	0.042700	68.65%	0.003493	5.62%	0.308131	495.37%
#6	0.062910	0.044041	70.01%	0.033702	53.57%	0.044923	71.41%	0.000997	1.59%	0.331739	527.32%

V201

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.032	-	-	-	-	-	-	-	-	-	-
#1	0.128184	0.108340	84.52%	0.081472	63.56%	0.068510	53.45%	0.014107	11.00%	0.266790	208.13%
#2	0.047879	0.038802	81.04%	0.031688	66.18%	0.028050	58.58%	0.002617	5.47%	0.184206	384.73%
#3	0.041998	0.036090	85.93%	0.028017	66.71%	0.021479	51.14%	0.006318	15.04%	0.143826	342.46%
#4	0.019118	0.016717	87.44%	0.014209	74.32%	0.009277	48.52%	0.001512	7.91%	0.116128	607.43%
#5	0.037793	0.031604	83.62%	0.024394	64.55%	0.020725	54.84%	0.004801	12.70%	0.156563	414.26%
#6	0.036645	0.031037	84.70%	0.026088	71.19%	0.019482	53.17%	0.002986	8.15%	0.126994	346.55%

V202

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.027	-	-	-	-	-	-	-	-	-	-
#1	0.027646	0.022814	82.52%	0.018943	68.52%	0.015616	56.48%	0.001004	3.63%	0.149668	541.37%
#2	0.025489	0.022050	86.51%	0.019702	77.30%	0.012786	50.16%	0.000697	2.73%	0.111142	436.04%
#3	0.033087	0.028772	86.96%	0.027281	82.45%	0.016337	49.38%	0.000567	1.71%	0.181642	548.98%
#4	0.020961	0.018101	86.36%	0.016333	77.92%	0.010570	50.43%	0.000829	3.96%	0.098367	469.28%
#5	0.036632	0.031681	86.49%	0.026496	72.33%	0.018390	50.20%	0.002373	6.48%	0.099620	271.95%
#6	0.028327	0.024377	86.05%	0.021729	76.71%	0.014429	50.94%	0.001279	4.51%	0.157033	554.35%

V203

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.361	-	-	-	-	-	-	-	-	-	-
#1	0.264040	0.222920	84.43%	0.205428	77.80%	0.141506	53.59%	0.022561	8.54%	0.708936	268.50%
#2	0.417937	0.316697	75.78%	0.218610	52.31%	0.272717	65.25%	0.036198	8.66%	1.435672	343.51%
#3	0.466434	0.400134	85.79%	0.357285	76.60%	0.239694	51.39%	0.025907	5.55%	1.533754	328.83%
#4	0.618172	0.545486	88.24%	0.455044	73.61%	0.290829	47.05%	0.066288	10.72%	1.552020	251.07%
#5	0.628483	0.505231	80.39%	0.312738	49.76%	0.373808	59.48%	0.078758	12.53%	2.167842	344.93%
#6	0.357440	0.325323	91.01%	0.286067	80.03%	0.148082	41.43%	0.046215	12.93%	1.059697	296.47%

Monocular inertial

MH01

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.032	-	-	-	-	-	-	-	-	-	-
#1	0.035645	0.031488	88.34%	0.032053	89.92%	0.016705	46.86%	0.002303	6.46%	0.091242	255.98%
#2	0.021853	0.020282	92.81%	0.018008	82.40%	0.008136	37.23%	0.003141	14.37%	0.056321	257.73%
#3	0.021027	0.018936	90.06%	0.016360	77.81%	0.009141	43.47%	0.002873	13.67%	0.057799	274.88%
#4	0.028647	0.026082	91.04%	0.027284	95.24%	0.011849	41.36%	0.002869	10.02%	0.069231	241.67%
#5	0.015272	0.013268	86.87%	0.011756	76.97%	0.007564	49.53%	0.000370	2.42%	0.045547	298.24%
#6	0.022571	0.021298	94.36%	0.020030	88.74%	0.007475	33.12%	0.000678	3.01%	0.060143	266.46%

MH02

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.053	-	-	-	-	-	-	-	-	-	-
#1	0.082254	0.075413	91.68%	0.072602	88.27%	0.032842	39.93%	0.007680	9.34%	0.154529	187.87%
#2	0.056738	0.052691	92.87%	0.055307	97.48%	0.021045	37.09%	0.008498	14.98%	0.119037	209.80%
#3	0.067402	0.062132	92.18%	0.064991	96.42%	0.026126	38.76%	0.005574	8.27%	0.133817	198.54%
#4	0.068326	0.064221	93.99%	0.061172	89.53%	0.023326	34.14%	0.002539	3.72%	0.117918	172.58%
#5	0.037561	0.033851	90.12%	0.028971	77.13%	0.016277	43.33%	0.003554	9.46%	0.098667	262.68%
#6	0.045198	0.041780	92.44%	0.038853	85.96%	0.017240	38.14%	0.002655	5.87%	0.090236	199.65%

MH03

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.033	-	-	-	-	-	-	-	-	-	-
#1	0.029338	0.025720	87.67%	0.023483	80.04%	0.014114	48.11%	0.002570	8.76%	0.091812	312.94%
#2	0.038308	0.034931	91.19%	0.033563	87.61%	0.015726	41.05%	0.000766	2.00%	0.104735	273.40%
#3	0.043278	0.038076	87.98%	0.032931	76.09%	0.020573	47.54%	0.004374	10.11%	0.098920	228.57%
#4	0.036678	0.032192	87.77%	0.028456	77.58%	0.017578	47.92%	0.003180	8.67%	0.093641	255.31%
#5	0.034312	0.029872	87.06%	0.025982	75.72%	0.016880	49.20%	0.002095	6.11%	0.084669	246.76%
#6	0.041638	0.036018	86.50%	0.031498	75.65%	0.020891	50.17%	0.001762	4.23%	0.101845	244.59%

MH04

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.099	-	-	-	-	-	-	-	-	-	-
#1	0.151054	0.138862	91.93%	0.119804	79.31%	0.059453	39.36%	0.027332	18.09%	0.422357	279.61%
#2	0.159759	0.147312	92.21%	0.131152	82.09%	0.061822	38.70%	0.018981	11.88%	0.287995	180.27%
#3	0.139501	0.126978	91.02%	0.112555	80.68%	0.057768	41.41%	0.013103	9.39%	0.295948	212.15%
#4	0.152827	0.141777	92.77%	0.125301	81.99%	0.057054	37.33%	0.026792	17.53%	0.278339	182.13%
#5	0.113980	0.075486	66.23%	0.060424	53.01%	0.085400	74.93%	0.012665	11.11%	0.822744	721.83%
#6	0.154039	0.142664	92.62%	0.128755	83.59%	0.058095	37.71%	0.007534	4.89%	0.295469	191.81%

MH05

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.071	-	-	-	-	-	-	-	-	-	-
#1	0.106432	0.093753	88.09%	0.077468	72.79%	0.050380	47.34%	0.010954	10.29%	0.253673	238.34%
#2	0.079666	0.070852	88.94%	0.062369	78.29%	0.036423	45.72%	0.002417	3.03%	0.245323	307.94%
#3	0.070335	0.061874	87.97%	0.055833	79.38%	0.033447	47.55%	0.007620	10.83%	0.170868	242.93%
#4	0.073537	0.065101	88.53%	0.059224	80.54%	0.034199	46.51%	0.005083	6.91%	0.267769	364.13%
#5	0.079124	0.071514	90.38%	0.068431	86.49%	0.033858	42.79%	0.011092	14.02%	0.219313	277.18%
#6	0.108005	0.094782	87.76%	0.080168	74.23%	0.051784	47.95%	0.013481	12.48%	0.268640	248.73%

V101

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.043	-	-	-	-	-	-	-	-	-	-
#1	0.040610	0.037661	92.74%	0.036644	90.23%	0.015191	37.41%	0.003583	8.82%	0.088182	217.15%
#2	0.040412	0.037592	93.02%	0.036840	91.16%	0.014832	36.70%	0.002929	7.25%	0.085471	211.50%
#3	0.046303	0.042553	91.90%	0.040019	86.43%	0.018256	39.43%	0.008606	18.59%	0.096887	209.24%
#4	0.045433	0.041964	92.36%	0.039993	88.02%	0.017412	38.32%	0.007702	16.95%	0.092190	202.91%
#5	0.043162	0.040008	92.69%	0.038604	89.44%	0.016196	37.52%	0.007517	17.42%	0.089678	207.77%
#6	0.040974	0.037984	92.70%	0.036693	89.55%	0.015365	37.50%	0.004063	9.92%	0.084557	206.37%

V102

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.016	-	-	-	-	-	-	-	-	-	-
#1	0.015545	0.013402	86.22%	0.012001	77.20%	0.007875	50.66%	0.000794	5.11%	0.145124	933.60%
#2	0.013919	0.012161	87.37%	0.011042	79.33%	0.006771	48.65%	0.000466	3.35%	0.113319	814.14%
#3	0.015426	0.013532	87.72%	0.011757	76.22%	0.007407	48.01%	0.001415	9.17%	0.085898	556.83%
#4	0.020227	0.015727	77.75%	0.013388	66.19%	0.012720	62.89%	0.000475	2.35%	0.134999	667.40%
#5	0.022283	0.016868	75.70%	0.014390	64.58%	0.014561	65.34%	0.002053	9.21%	0.166966	749.29%
#6	0.013238	0.012203	92.18%	0.011537	87.15%	0.005132	38.76%	0.000162	1.23%	0.028526	215.49%

V103

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.025	-	-	-	-	-	-	-	-	-	-
#1	0.024250	0.020184	83.23%	0.016178	66.72%	0.013440	55.43%	0.001686	6.95%	0.101304	417.76%
#2	0.040327	0.032804	81.35%	0.025516	63.27%	0.023455	58.16%	0.007883	19.55%	0.117025	290.19%
#3	0.036869	0.029485	79.97%	0.025245	68.47%	0.022135	60.04%	0.003468	9.41%	0.110311	299.20%
#4	0.043106	0.035791	83.03%	0.031099	72.15%	0.024023	55.73%	0.002659	6.17%	0.110960	257.41%
#5	0.044632	0.037255	83.47%	0.031620	70.85%	0.024578	55.07%	0.006270	14.05%	0.130644	292.71%
#6	0.037413	0.030144	80.57%	0.024616	65.79%	0.022161	59.23%	0.006466	17.28%	0.118722	317.32%

V201

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.041	-	-	-	-	-	-	-	-	-	-
#1	0.042169	0.034767	82.45%	0.027942	66.26%	0.023865	56.59%	0.011294	26.78%	0.124032	294.13%
#2	0.013364	0.011691	87.48%	0.010832	81.05%	0.006473	48.44%	0.001164	8.71%	0.088296	660.70%
#3	0.018338	0.013115	71.52%	0.011813	64.42%	0.012816	69.89%	0.000598	3.26%	0.281970	1537.65%
#4	0.012884	0.011313	87.81%	0.010541	81.81%	0.006165	47.85%	0.000678	5.26%	0.146894	1140.10%
#5	0.020537	0.018038	87.83%	0.015719	76.54%	0.009818	47.81%	0.001801	8.77%	0.172314	839.04%
#6	0.021599	0.017056	78.97%	0.014249	65.97%	0.013251	61.35%	0.000319	1.48%	0.241966	1120.28%

V202

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.015	-	-	-	-	-	-	-	-	-	-
#1	0.016182	0.014900	92.08%	0.014525	89.76%	0.006312	39.01%	0.001369	8.46%	0.046770	289.03%
#2	0.013364	0.011691	87.48%	0.010832	81.05%	0.006473	48.44%	0.001164	8.71%	0.088296	660.70%
#3	0.018338	0.013115	71.52%	0.011813	64.42%	0.012816	69.89%	0.000598	3.26%	0.281970	1537.65%
#4	0.012884	0.011313	87.81%	0.010541	81.81%	0.006165	47.85%	0.000678	5.26%	0.146894	1140.10%
#5	0.020537	0.018038	87.83%	0.015719	76.54%	0.009818	47.81%	0.001801	8.77%	0.172314	839.04%
#6	0.021599	0.017056	78.97%	0.014249	65.97%	0.013251	61.35%	0.000319	1.48%	0.241966	1120.28%

V203

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.037	-	-	-	-	-	-	-	-	-	-
#1	0.020387	0.017168	84.21%	0.014430	70.78%	0.010996	53.93%	0.000939	4.61%	0.076446	374.98%
#2	0.026414	0.022763	86.18%	0.020170	76.36%	0.013400	50.73%	0.000760	2.88%	0.095114	360.09%
#3	0.035476	0.032920	92.79%	0.031417	88.56%	0.013222	37.27%	0.003798	10.71%	0.096725	272.65%
#4	0.025870	0.021763	84.12%	0.017973	69.47%	0.013987	54.06%	0.002173	8.40%	0.088328	341.43%
#5	0.023435	0.019589	83.59%	0.017653	75.33%	0.012865	54.89%	0.000566	2.41%	0.084802	361.86%
#6	0.022944	0.018382	80.12%	0.014787	64.45%	0.013729	59.84%	0.000619	2.70%	0.069309	302.09%

Stereo inertial

MH01

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.037	-	-	-	-	-	-	-	-	-	-
#1	0.037852	0.034436	90.98%	0.039788	105.11%	0.015714	41.51%	0.001918	5.07%	0.079224	209.30%
#2	0.032869	0.030104	91.59%	0.031116	94.67%	0.013194	40.14%	0.002413	7.34%	0.070068	213.18%
#3	0.036464	0.033275	91.25%	0.036995	101.46%	0.014914	40.90%	0.002596	7.12%	0.074800	205.13%
#4	0.042572	0.038988	91.58%	0.042308	99.38%	0.017096	40.16%	0.001186	2.79%	0.084610	198.75%
#5	0.035707	0.032525	91.09%	0.036705	102.79%	0.014733	41.26%	0.001223	3.42%	0.072748	203.74%
#6	0.032415	0.029415	90.75%	0.029860	92.12%	0.013619	42.01%	0.000691	2.13%	0.079440	245.07%

MH02

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.031	-	-	-	-	-	-	-	-	-	-
#1	0.028810	0.024418	84.75%	0.020967	72.78%	0.015290	53.07%	0.000583	2.02%	0.075949	263.62%
#2	0.034388	0.027585	80.22%	0.020992	61.05%	0.020533	59.71%	0.000959	2.79%	0.112813	328.06%
#3	0.033808	0.028134	83.22%	0.023102	68.33%	0.018746	55.45%	0.003056	9.04%	0.102893	304.35%
#4	0.033192	0.027990	84.33%	0.022029	66.37%	0.017840	53.75%	0.002160	6.51%	0.096487	290.70%
#5	0.042474	0.033594	79.09%	0.023492	55.31%	0.025991	61.19%	0.004645	10.94%	0.138514	326.12%
#6	0.035685	0.028192	79.00%	0.021346	59.82%	0.021878	61.31%	0.000877	2.46%	0.114866	321.89%

MH03

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.026	-	-	-	-	-	-	-	-	-	-
#1	0.027117	0.024185	89.19%	0.021501	79.29%	0.012265	45.23%	0.001371	5.05%	0.090918	335.27%
#2	0.032197	0.028635	88.94%	0.025471	79.11%	0.014721	45.72%	0.002992	9.29%	0.091769	285.02%
#3	0.031424	0.027607	87.85%	0.024348	77.48%	0.015010	47.77%	0.001029	3.27%	0.090113	286.77%
#4	0.036330	0.032493	89.44%	0.029166	80.28%	0.016250	44.73%	0.002208	6.08%	0.094611	260.42%
#5	0.034860	0.030909	88.67%	0.026404	75.74%	0.016119	46.24%	0.000444	1.27%	0.094995	272.50%
#6	0.035272	0.031163	88.35%	0.025749	73.00%	0.016521	46.84%	0.003235	9.17%	0.095905	271.90%

MH04

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.059	-	-	-	-	-	-	-	-	-	-
#1	0.051604	0.046460	90.03%	0.041353	80.14%	0.022460	43.52%	0.001371	2.66%	0.147616	286.06%
#2	0.051812	0.046904	90.53%	0.046433	89.62%	0.022011	42.48%	0.003548	6.85%	0.144343	278.59%
#3	0.053374	0.048319	90.53%	0.048178	90.26%	0.022674	42.48%	0.004968	9.31%	0.146372	274.24%
#4	0.054860	0.049556	90.33%	0.045819	83.52%	0.023534	42.90%	0.005897	10.75%	0.153974	280.67%
#5	0.053507	0.048894	91.38%	0.047028	87.89%	0.021734	40.62%	0.003459	6.47%	0.152675	285.34%
#6	0.053269	0.048566	91.17%	0.046752	87.77%	0.021886	41.09%	0.001878	3.53%	0.151143	283.73%

MH05

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.086	-	-	-	-	-	-	-	-	-	-
#1	0.098118	0.087398	89.07%	0.083855	85.46%	0.044595	45.45%	0.006070	6.19%	0.196570	200.34%
#2	0.076391	0.069011	90.34%	0.060749	79.52%	0.032758	42.88%	0.006394	8.37%	0.154765	202.59%
#3	0.079331	0.073052	92.09%	0.070796	89.24%	0.030931	38.99%	0.008687	10.95%	0.149877	188.93%
#4	0.073361	0.066772	91.02%	0.060460	82.41%	0.030387	41.42%	0.008123	11.07%	0.143368	195.43%
#5	0.103696	0.093132	89.81%	0.089236	86.06%	0.045599	43.97%	0.014678	14.15%	0.199569	192.46%
#6	0.099290	0.089898	90.54%	0.086309	86.93%	0.042152	42.45%	0.014765	14.87%	0.189289	190.64%

V101

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.037	-	-	-	-	-	-	-	-	-	-
#1	0.035701	0.033659	94.28%	0.034121	95.57%	0.011903	33.34%	0.002261	6.33%	0.069804	195.52%
#2	0.037168	0.034908	93.92%	0.035789	96.29%	0.012761	34.33%	0.002142	5.76%	0.074087	199.33%
#3	0.037063	0.035006	94.45%	0.036044	97.25%	0.012174	32.85%	0.002927	7.90%	0.072006	194.28%
#4	0.036590	0.034457	94.17%	0.035392	96.73%	0.012309	33.64%	0.002325	6.36%	0.071884	196.46%
#5	0.037389	0.035134	93.97%	0.035690	95.45%	0.012789	34.20%	0.004863	13.01%	0.078881	210.97%
#6	0.036969	0.034702	93.87%	0.035540	96.13%	0.012748	34.48%	0.001558	4.22%	0.075013	202.91%

V102

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.014	-	-	-	-	-	-	-	-	-	-
#1	0.013438	0.012433	92.52%	0.012158	90.48%	0.005100	37.95%	0.000943	7.02%	0.035607	264.98%
#2	0.015192	0.013642	89.80%	0.012047	79.30%	0.006685	44.00%	0.001831	12.06%	0.036863	242.65%
#3	0.014300	0.013058	91.31%	0.011957	83.61%	0.005830	40.77%	0.001158	8.10%	0.036238	253.40%
#4	0.014785	0.013631	92.20%	0.013266	89.72%	0.005726	38.73%	0.000723	4.89%	0.041786	282.62%
#5	0.011891	0.010835	91.12%	0.010403	87.49%	0.004898	41.19%	0.001048	8.81%	0.043370	364.74%
#6	0.012398	0.011416	92.08%	0.011141	89.87%	0.004835	39.00%	0.001136	9.16%	0.039223	316.37%

V103

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.023	-	-	-	-	-	-	-	-	-	-
#1	0.019834	0.018158	91.55%	0.017463	88.05%	0.007979	40.23%	0.001282	6.47%	0.046956	236.75%
#2	0.022068	0.020668	93.66%	0.020744	94.00%	0.007736	35.05%	0.000253	1.15%	0.046159	209.16%
#3	0.027217	0.025227	92.69%	0.024524	90.10%	0.010216	37.53%	0.002147	7.89%	0.064151	235.70%
#4	0.022933	0.021613	94.24%	0.021177	92.34%	0.007668	33.44%	0.003491	15.22%	0.054455	237.46%
#5	0.027515	0.025167	91.47%	0.024902	90.50%	0.011122	40.42%	0.002419	8.79%	0.065718	238.84%
#6	0.024260	0.022740	93.74%	0.022329	92.04%	0.008450	34.83%	0.000727	3.00%	0.048540	200.09%

V201

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.037	-	-	-	-	-	-	-	-	-	-
#1	0.039833	0.037206	93.41%	0.039432	98.99%	0.014224	35.71%	0.006492	16.30%	0.095619	240.05%
#2	0.025012	0.023481	93.88%	0.023576	94.26%	0.008616	34.45%	0.003719	14.87%	0.077494	309.83%
#3	0.032415	0.030194	93.15%	0.031664	97.68%	0.011794	36.38%	0.005502	16.97%	0.086639	267.28%
#4	0.026755	0.024925	93.16%	0.025724	96.15%	0.009725	36.35%	0.002899	10.84%	0.079271	296.28%
#5	0.022258	0.020424	91.76%	0.018732	84.16%	0.008847	39.75%	0.002057	9.24%	0.093225	418.84%
#6	0.026974	0.025161	93.28%	0.025987	96.34%	0.009722	36.04%	0.006854	25.41%	0.083376	309.10%

V202

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.014	-	-	-	-	-	-	-	-	-	-
#1	0.013806	0.012374	89.62%	0.011381	82.43%	0.006124	44.36%	0.001122	8.13%	0.037093	268.66%
#2	0.011977	0.010918	91.16%	0.010639	88.83%	0.004924	41.11%	0.000876	7.32%	0.034818	290.70%
#3	0.014142	0.012800	90.51%	0.011865	83.90%	0.006013	42.52%	0.000856	6.05%	0.032479	229.67%
#4	0.024537	0.021404	87.23%	0.018945	77.21%	0.011997	48.89%	0.002353	9.59%	0.063322	258.06%
#5	0.011869	0.010820	91.16%	0.010427	87.85%	0.004879	41.11%	0.001055	8.89%	0.035173	296.34%
#6	0.012561	0.011251	89.57%	0.010742	85.52%	0.005585	44.47%	0.000483	3.85%	0.036513	290.68%

V203

Run	RMS_ATE	Mean		Median		Std		Min		Max
Paper	0.029	-	-	-	-	-	-	-	-	-	-
#1	0.060111	0.052893	87.99%	0.045140	75.09%	0.028560	47.51%	0.006004	9.99%	0.132159	219.86%
#2	0.056788	0.047470	83.59%	0.038655	68.07%	0.031168	54.88%	0.002057	3.62%	0.278297	490.06%
#3	0.052929	0.046955	88.71%	0.043783	82.72%	0.024427	46.15%	0.001185	2.24%	0.199298	376.54%
#4	0.028151	0.025139	89.30%	0.023954	85.09%	0.012670	45.01%	0.002340	8.31%	0.137291	487.69%
#5	0.020663	0.017816	86.22%	0.015888	76.89%	0.010467	50.65%	0.001361	6.59%	0.074749	361.75%
#6	0.034629	0.030187	87.17%	0.026051	75.23%	0.016968	49.00%	0.001774	5.12%	0.153984	444.67%

References

Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. Martínez Montiel, and Juan Domingo Tardós. ORB-SLAM3: an accurate open-source library for visual, visual-inertial and multi-map SLAM. CoRR, abs/2007.11898, 2020. [ bib | DOI ]


@article{campos2020orbslam3,
  author = {Carlos Campos and
               Richard Elvira and
               Juan J. Gómez Rodríguez and
               José M. Martínez Montiel and
               Juan Domingo Tardós},
  title = { {ORB-SLAM3:} An Accurate Open-Source Library for Visual, Visual-Inertial
               and Multi-Map {SLAM}},
  journal = {CoRR},
  volume = {abs/2007.11898},
  year = {2020},
  url = {https://arxiv.org/abs/2007.11898},
  archiveprefix = {arXiv},
  eprint = {2007.11898},
  timestamp = {Wed, 29 Jul 2020 15:36:39 +0200},
  biburl = {https://dblp.org/rec/journals/corr/abs-2007-11898.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org},
  doi = {10.1109/TRO.2021.3075644}
}