Research

Blind Man's Buff - George Morland (1788)

Humans are provided with a remarkable cognitive framework that allows them to process complex perceptual information and create a rich representation of their external reality. These perceptual representations play a fundamental role in the planning and execution of tasks, even in challenging perceptual conditions such as the absence of modality-specific information (e.g. dark room) or if a given sensor is malfunctioning (e.g. visual impairments).

We aim at endowing agents with mechanisms to learn representations from multimodal sensory data and to employ them in the execution of tasks.

Research Question

How can artificial agents learn multimodal representations of their environment and leverage such representations in the execution of tasks, even in challenging perceptual conditions?

Major Contributions

A novel multimodal representation of human actions and an online algorithm to learn such representations in a sample-efficient way to employ them in few-shot classification tasks.

In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4288-4293). IEEE.

MHVAE

A novel hierarchical multimodal generative model that outperforms current approaches for cross-modality inference: the generation of absent modality data from available ones.

Under review (2021)

A novel architecture that allows a reinforcement learning agent to reuse a policy trained over a given sensory modality (e.g. image) on a different sensory modality (e.g sound)

In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1260-1268. 2020.

Research Overview

Doctoral Consortium of the 19th International Conference on Autonomous Agents and MultiAgent Systems (2020) - 9 min