Corvus ISR Publishes Synthetic Benchmark Comparing Tracker Models

Corvus ISR tracker benchmark matrix (seed 1337) — The published matrix — every row reproducible. Source: corvusisr.com/benchmark

Corvus ISR, known for its wide-area motion imagery (WAMI) exploitation tools, has released a comprehensive public tracker benchmark that rigorously compares two distinct tracking models. By utilizing a fixed-seed synthetic scene with perfect ground truth, the benchmark isolates the true capabilities of each tracker without confounding variables. This approach highlights the importance of controlled testing environments in advancing sensor and algorithm development.

The benchmark evaluates two models: v1, a simple greedy nearest-neighbour baseline featuring two-pass association, constant-velocity prediction, and fixed 2-second coasting, and v2, a more advanced confirmed-track auction model implementing three-tier auction association, velocity-consistency gating, and confidence decay. Both models are tested under identical conditions, ensuring a fair comparison and revealing the significance of sophisticated data association techniques in reducing identity errors.

Results show that switching from v1 to v2 leads to a substantial 42% reduction in ID switches per minute. For instance, in a scene with 150 moving objects at 2fps, ID switches decrease from 2,042 to 1,183. When scaled to 400 objects in a dense scene, the reduction is similar: from 14,032 to 8,040. These numbers demonstrate the tangible improvements achieved with the new approach, even under challenging conditions like occlusion or low frame rate.

Why publish these failure metrics? The answer lies in the value of perfect ground truth in synthetic scenes. Unlike real-world data, where ground truth is often approximated, synthetic environments allow precise measurement of every identity switch, including fragmentations and re-acquisitions. The strictness of this metric emphasizes the need for continuous innovation and transparency in evaluating tracker performance, inviting a culture of measured improvement rather than marketing hype.

Corvus ISR live demo — The live demo — press “Run benchmark” to reproduce the numbers. Source: corvusisr.com/demo

From an engineering perspective, v2 achieves these results with an average processing time of about 1.2ms per sensor tick at a density of 400 objects, comfortably within real-time constraints. The entire benchmarking process is accessible via the live demo, where users can reproduce the benchmark without any signup or NDA. This open approach exemplifies the commitment to transparency and reproducibility in sensor evaluation, crucial for scientific rigor and technological progress.

Ultimately, the purpose of publishing these failure numbers is to underscore that even top-tier, synthetic benchmarks reveal thousands of identity errors per minute. Such data is invaluable for research, as it pinpoints where improvements are needed. Encouraging science-minded readers to explore the benchmark firsthand invites everyone to better understand the challenges and opportunities in multi-object tracking technology.