Benchmarking Vision Language Models on German Factual Data

Abstract

Similar to LLMs, the development of vision language models is mainly driven by English datasets and models trained in English and Chinese language, whereas support for other languages, even those considered high-resource languages such as German, remains significantly weaker. In this work we present an analysis of open-weight VLMs on factual knowledge in the German and English language. We disentangle the image-related aspects from the textual ones by analyzing accuracy with jury-as-a-judge in both prompt languages and images from German and international contexts. We found that for celebrities and sights, VLMs struggle because they are lacking visual cognition of German image contents. For animals and plants, the tested models can often correctly identify the image contents according to the scientific name or English common name but fail in German language. Cars and supermarket products were identified equally well in English and German images across both prompt languages.

Mehr zum Titel

Titel	Benchmarking Vision Language Models on German Factual Data
Medien	21st International Conference on Artificial Intelligence Applications and Innovations, 26 – 29 June, 2025, Limassol, Cyprus (accepted)
Verlag	---
Heft	---
Band	---
ISBN	---
Verfasser/Herausgeber	Prof. Dr. René Peinl, Vincent Tischler
Seiten	---
Veröffentlichungsdatum	15.04.2025
Projekttitel	---
Zitation	Peinl, René; Tischler, Vincent (2025): Benchmarking Vision Language Models on German Factual Data. 21st International Conference on Artificial Intelligence Applications and Innovations, 26 – 29 June, 2025, Limassol, Cyprus (accepted).