OpenMonkeyStudio: Automated Markerless Pose Estimation in Freely Moving Macaques

Bala, Eisenreich, Yoo, Hayden*, Park*, and Zimmermann* (* indicates joint last authors)
Nature Communications, 2020
Detail +

In our paper, we describe a novel integrated hardware and software system for pose tracking in freely moving rhesus macaques. We use 64 carefully placed machine vision cameras to generate a 13-point stick model of our subjects. Our system relies on a large library of hand-annotated images (>1M images), and a deep learning engine for predictive inference.
Paper link

Macaques Are Risk-averse in a Freely Moving Foraging Task

Eisenreich, Hayden, and Zimmermann
Scientific Reports, 2019
Detail +

Rhesus macaques (Macaca mulatta) appear to be robustly risk-seeking in computerized gambling tasks typically used for electrophysiology. This behavior distinguishes them from many other animals, which are risk-averse, albeit measured in more naturalistic contexts. We wondered whether macaques’ risk preferences reflect their evolutionary history or derive from the less naturalistic elements of task design associated with the demands of physiological recording. We assessed macaques’ risk attitudes in a task that is somewhat more naturalistic than many that have previously been used: subjects foraged at four feeding stations in a large enclosure. Patches (i.e., stations), provided either stochastically or non-stochastically depleting rewards. Subjects’ patch residence times were longer at safe than at risky stations, indicating a preference for safe options. This preference was not attributable to a win-stay-lose-shift heuristic and reversed as the environmental richness increased. These findings highlight the lability of risk attitudes in macaques and support the hypothesis that the ecological validity of a task can influence the expression of risk preference.
Paper link

MONET: Multiview Semi-supervised Keypoint via Epipolar Divergence

Yao, Jafarian, and Park
International Conference on Computer Vision (ICCV), 2019
Detail +

This paper presents an end-to-end semi-supervised learning framework for a keypoint detector using multiview image streams. In particular, we consider general subjects such as non-human species where attaining a large scale annotated dataset is challenging. While multiview geometry can be used to self-supervise the unlabeled data, integrating the geometry into learning a keypoint detector is challenging due to representation mismatch. We address this mismatch by formulating a new differentiable representation of the epipolar constraint called epipolar divergence--a generalized distance from the epipolar lines to the corresponding keypoint distribution. Epipolar divergence characterizes when two view keypoint distributions produce zero reprojection error.
Paper link

Multiview Cross-supervision for Semantic Segmentation

Yao and Park
Winter Conference on Applications of Computer Vision (WACV), 2020
Detail +

This paper presents a semi-supervised learning framework for a customized semantic segmentation task using multiview image streams. A key challenge of the customized task lies in the limited accessibility of the labeled data due to the requirement of prohibitive manual annotation effort. We hypothesize that it is possible to leverage multiview image streams that are linked through the underlying 3D geometry, which can provide an additional supervisionary signal to train a segmentation model. We formulate a new cross-supervision method using a shape belief transfer---the segmentation belief in one image is used to predict that of the other image through epipolar geometry analogous to shape-from-silhouette. The shape belief transfer provides the upper and lower bounds of the segmentation for the unlabeled data where its gap approaches asymptotically to zero as the number of the labeled views increases. We integrate this theory to design a novel network that is agnostic to camera calibration, network model, and semantic category and bypasses the intermediate process of suboptimal 3D reconstruction.
Paper link