Our Research

Our cutting-edge research spans many topics in computer vision and multimodal learning. Our current emphasis is on building reliable AI models and advancing multimodal large language models. We work on topics such as grounding, captioning, visual question answering, and multimodal fact-checking.

For the most up-to-date snapshot of our lab’s research topics, see also the lab leads’ Google Scholar (Prof. Anna Rohrbach, Prof. Marcus Rohrbach).

Research Project Areas

Selective Prediction

Selective Prediction, as a sub-field of reliable Artificial Intelligence, aims to develop methods and techniques to identify uncertain model predictions. Instead of having a model produce an output for every sample, an abstention mechanism is implemented that recognizes instances where predictions are likely inaccurate. Accordingly, the model only makes predictions for samples that are not filtered out by this mechanism and abstains from the others. Colloquially, with Selective Prediction, we are "creating models that know what they don't know."

Learn more

Multimodal Fact-Checking

Disinformation is arguably one of the biggest threats for society and democracy. Multimodal fact-checking aims to verify claims (e.g. from social media posts) that, beyond text, also include images or videos. The fact-checking process includes the interpretation of the actual claim, the retrieval of textual and visual evidence from the open web, and reasoning to find if the claim holds true. We are pushing the boundaries of current multimodal LLM approaches to elevate fact-checking to an effective and scalable level.

Learn more

Modularity

Modular AI systems are designed with a modular architecture, where complex tasks are broken down into smaller, more manageable components or modules. Each module handles specific aspects like perception, reasoning, or decision-making. With current end-to-end models growing into monolithic black boxes, there's a clear argument for more modular systems composed of smaller, manageable components.

Learn more

Visual Relationship Detection

Visual relationship detection (VRD) is a computer vision task aiming to recognize relationships or interactions between objects within an image. Although it is a more challenging task than object recognition as it faces more ambiguities, it plays a crucial role in understanding images and the visual world more comprehensively. In our group, we are leveraging multiple modalities to allow a finer-grained distinction between these ambiguities.

Learn more