vision

The repository is private for now. This page describes what the package does and how it was checked.

What it does

• Images. Grayscale images, bilinear sampling, pyramids, a box blur.
• Features. FAST-9 corners spread over the frame, pyramidal Lucas-Kanade flow with a forward-backward check, oriented BRIEF descriptors and Hamming matching.
• Geometry. Rotations and poses, two-view translation with RANSAC, triangulation, pose from 3D-2D matches, Gauss-Newton refinement and sliding-window bundle adjustment.
• Direct alignment. A pose solved photometrically against a pyramid, with an epipolar depth search to seed it.
• Inertial. A complementary filter for attitude and gravity, IMU preintegration and a visual-inertial metric scale.
• Planes. Detection in a point cloud, tracking across updates with boundary polygons, and ray casts for hit tests.

An example

Corners in one frame, followed into the next with Lucas-Kanade flow:

import <list>;
import <vision/image>;
import <vision/fast>;
import <vision/klt>;

ImageRef first = a.ref();
FastConfig fast;
List<Corner> corners;
vision_fast(&first, &fast, &corners);

List<V2> from;
for (i in 0..corners.size) from.push(corners.data[i].px);
Pyramid pa;
Pyramid pb;
pa.build(first, 3);
pb.build(b.ref(), 3);
KltConfig klt;
List<FlowResult> flow;
vision_klt_track(&pa, &pb, from.data, from.size, null, 0, &klt, &flow);
// flow.data[i].ok() and flow.data[i].px: where corner i went

Every config struct carries the defaults of the original, so KltConfig klt; is a working setup. On two test frames the example prints 152 corners, 150 tracked, median motion (5.00, -2.00) px.

Same bits as the C++

The package's tests link tinyvio's C++ core, pinned at a commit and built with -ffp-contract=off as its WebAssembly builds are, against each module, and compare every output bit for bit on frames and an IMU stream rendered for the test. They run on every commit:

Two things a C++ build does implicitly are made explicit. Tied elements are ordered as libc++ orders them, since Eskiu's <sort> is libc++'s algorithm. And sin and cos of one angle go through one sincos call on Apple platforms, as clang merges them there.

Where it runs

vision is pure Eskiu, so it builds wherever Eskiu does: its tests run on macOS, Linux and Windows, and it cross-compiles for iOS, Android (arm64, armv7, x86_64) and WebAssembly. The results are the same bits on every platform whose C library rounds sin, cos and the other math functions the same way.

The same code, under tinyvio's own names, is tinyvio's Eskiu engine. Replaying a recording through its C API gives the same output as the C++ engine on macOS, WebAssembly, the iOS simulator and the Android emulator. See the ReactVision case study.