Responsive image





Interactive WebXR Hypertext Storytelling for Cultural Heritage

Zöllner, Michael; Gemeinhardt, Jan; Krause, Moritz (2024)

HUMAN '24: Proceedings of the 7th Workshop on Human Factors in Hypertext 2024, 7, S. 1-4.
DOI: 10.1145/3679058.3688635


Open Access Peer Reviewed
 

We are presenting our approach for interactive cultural heritage storytelling in WebXR. Therefore, we are describing our scenes’ structure consisting of (stylized) photospheres of the historic locations, 3D models of 3D-scanned historic artifacts and animated 2D textures of historic characters generated with a machine learning toolset. The result is a platform-independent web-application in an immersive interactive WebXR environment running in browsers on PCs, tablets, phones and XR headsets thanks to the underlying software based on the open-source framework A-Frame. Our pa- per describes the process, the results and the limitations in detail. The resulting application, designed for the Fichtelgebirge region in Upper Franconia, Germany, offers users an immersive digital time travel experience in the virtual space and within a museum setting connecting real artifacts and virtual stories.

more

Similarities of Motion Patterns in Skateboarding and Hydrofoil Pumping

Zöllner, Michael; Krause, Moritz; Gemeinhardt, Jan (2024)

iWOAR 2024 - 9th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence (iWOAR 2024), September 26--27, 2024, Potsdam, Germany 2024.
DOI: 10.1145/3694885.3695197


Open Access Peer Reviewed
 

Like skateboarding acceleration in surfing on a hydrofoil with muscle power is achieved by a constant sinusoidal motion. Both are challenging sports to begin with because learning the complex up and down movements takes time, skill and reflexion. The interplay of rotating joints and applying forces at the right time is hard to perceive, understand and to transfer into muscle memory.

Since the motions in skateboarding on pump tracks and hydrofoil pumping are similar, we are comparing both motion sequences with inertial measurement units and 3D pose estimation. We postulate that learning the physically challenging and expensive hydrofoil pumping can be improved and accelerated by training with skateboards. Therefore, we are capturing forces with inertial measurement units and validate them with 3D pose estimation. Finally, we are comparing and visualizing the motions and forces of the boards and the skeleton to show the similarities within the y- and z-momentum.

more

Shy Tech Floating Tablet Computers for Interaction with XR in Museums

Zöllner, Michael; Gemeinhardt, Jan; Krause, Moritz (2024)

GCH 2024 - Eurographics Workshop on Graphics and Cultural Heritage 2024.
DOI: 10.2312/gch.20241266


Open Access Peer Reviewed
 

Tablet computers like the iPad are user-friendly and reliable interaction devices for displaying XR content in museums. Although they are lacking the stereo view of a VR headset, they also avoid the hygiene, optics and usability challenges of head mounted displays in public spaces. In our paper we describe methods to improve the shortcomings of these devices: Placement, presentation and distracting branding. Our solution is a combination of a physical hanging system and custom designed acrylic cases disguising the hardware. In our example of an exhibition about a baroque architect the result is a series of floating baroque picture frames which users are grabbing and rotating to view into the XR scenes.

more

WebXR Cultural Heritage Tour with Generative AI Characters

Zöllner, Michael; Gemeinhardt, Jan (2024)

GCH 2024 - Eurographics Workshop on Graphics and Cultural Heritage 2024.
DOI: 10.2312/gch.20241266


Open Access Peer Reviewed
 

Tablet computers like the iPad are user-friendly and reliable interaction devices for displaying XR content in museums. Although they are lacking the stereo view of a VR headset, they also avoid the hygiene, optics and usability challenges of head mounted displays in public spaces. In our paper we describe methods to improve the shortcomings of these devices: Placement, presentation and distracting branding. Our solution is a combination of a physical hanging system and custom designed acrylic cases disguising the hardware. In our example of an exhibition about a baroque architect the result is a series of floating baroque picture frames which users are grabbing and rotating to view into the XR scenes.

more

Generative AI Tool Pipeline for Creating Artificial Historical Characters for Cultural Heritage XR

Zöllner, Michael; Gemeinhardt, Jan (2024)

International Conference on Human-Computer Interaction 2024, S. 41-46.
DOI: 10.1007/978-3-031-61950-2_5


Peer Reviewed
 

In our project, we aimed to create historically authentic and vivid virtual representations of historic personalities that are connected to the regional Fichtelgebirge (Bavaria, Germany) to support the storytelling of our immersive XR applications. We are describing the tools in detail, the process of the tool chain and the resulting media. Next, we are discussing the challenges in media production like historical correctness and the consultation of historians. In order to create visual reproducibility we are explaining the detailed text prompts, their limitations and how to cope with resulting errors of the human physiognomy. Finally, we are briefly describing the application of the animated and talking generated historic characters in an immersive interactive WebXR environment. The XR experience is presented in web browsers on smartphones, tablets and XR headsets and the underlying software is based on the open-source framework Aframe. Our paper will describe the process, the results and the limitations in detail. Furthermore, we will provide a flow chart of the tool pipeline with visual examples of these aspects. The animations and voices of the historic characters will be demonstrated in videos of the XR application.

more

Hydrodynamic Force Measurement on Surfboards During Surfing Maneuvers

Kniesburges, Stefan; Punger, Nina; Tur, Bocac; Zöllner, Michael; Buff, Joachim...

ESBiomech 2024: BIOMECHANICS FROM RESEARCH TO PRACTICE 2024.


Peer Reviewed
 

Surfing is a complex sport that encounters the interaction between the surfer and the highly unsteady water hydrodynamic of ocean or river waves. With the availability of off-shore river waves or wave pools, it becomes more and more popular being included in the Olympic Games 2021 in Tokyo/Japan. To analyze the biomechanical characteristics of surfers, all forces that act on the surfer have to be measured which has not yet been done before.

Four pressure sensors were integrated water-tightly in a 3D-printed surfboard fin, each two sensors on opposite side. The fin was mounted at the rear end of a surf board on the left side of the centerline within a three-fin-configuration. The sensors were connected to a micro controller with micro SD-card within the board that performed and stored the measurement data with an acquisition rate of 27 Hz. For the measurements, a male surfer performed a set of turning maneuvers on an artificial river wave as shown in figure 2. Thereby, the surfer’s motion was recorded by a camera using frame-rate of 30 fps. The synchronization of the pressure and the video signals was achieved in a post-processing step.

The pressure signals showed a negative pressure difference between the inside sensors and the outside sensors at all surf maneuvers, see figure 3. It generated an outwardly directed force that becomes largest during the turns. The smallest pressure difference was detected on the surf-path from the left to right side of the wave or the other way around.

Taking both lateral fins into account, the pressure forces are contrarily directed apart from the surfboard. This arrangement generates the stabilizing effect of the board’s rear end which helps the surfer to keep the board on the desired track. Especially during the turns, the forces became largest giving the surfer a stable center of rotation at the rear end of the board. This stabilizing effect of surfboard fins was reported and observed in computational models [1,2,3]. The measurement of hydrodynamic forces enables to develop a biomechanical model of a surfer that show the characteristic biomechanic stress level peak and location within the surfer’s body.

more

Web Based Data Visualization Dashboard for Sensor and Pose Estimation Data

Zöllner, Michael; Krause, Moritz (2024)

International Conference on Human-Computer Interaction, S. 291-296.
DOI: 10.1007/978-3-031-62110-9_31


Peer Reviewed
 

Our proposed visual dashboard aims to facilitate training progress evaluation in sports like skateboarding and foil pumping by offering real-time sensor and camera data processing through machine learning-based pose estimation. The web-based application, developed in HTML, CSS, and JavaScript utilizing p5js and Google Charts frameworks, enables athletes to visualize and analyze their movements for efficient and safe performance without the need for app installation on various devices. This paper will delve into the advantages and challenges of the web-based approach, detailing data structures, synchronization techniques, and plans for future enhancements like Web BLE connectivity and improved user experience across different devices.

more

Symotiv - Virtual Insights into the Symphony Orchestra.

Zöllner, Michael; Krause, Moritz; Bosl, Markus; Widmann, Dirk (2023)

AI in Museums 2023.


Open Access
 

The technologies regarding capturing motion and visualizing data enabled new possibilities describing complex systems to a broader audience. We used the latest methods of motion analysis through Machine Learning / Artificial Intelligence and visualization in Virtual and Augmented Reality (VR/AR) in order to analyze and explain the way the Hof Symphony Orchestra works. Therefore, we showed the diverse aspects from rehearsal to performance to a broad audience via interactive immersive Extended Reality experiences. In this paper we are describing the process and the implications for tracking motion and gestures in cultural places. 


Preliminary studies of measuring skateboarding forces by combining inertial sensors and camera-based pose estimation.

Zöllner, Michael; Krause, Moritz (2023)

iWOAR 2023 - 8th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence.
DOI: 10.1145/3615834.3615856


Peer Reviewed
 

Understanding acceleration forces and making progress in learning Skateboarding is a process of trial and error. In our paper we are describing our preliminary experiments for describing the complex interactions while pushing for speed in ramps and pump tracks. Therefore, we capture and visualize the body movement, the joint relations from hip to ankle and the resulting forces by joining iner- tial sensors on the skateboard and camera-based machine learning pose estimation of the athlete.

more

HomeGrid - Experimental Displays for SmartHome Devices and Interfaces

Krause, Moritz; Zöllner, Michael (2023)

iWOAR 2023 - 8th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence.
DOI: 10.1145/3615834.3615850


Peer Reviewed
 

In our society, displays are becoming increasingly prevalent. While it is nearly inconceivable to imagine a daily life without screens, scientific research indicates that our mental well-being is nega- tively affected by the increasing use and time spent on screens. The "HomeGrid" project has been developed as a smart home concept, aiming to explore how traditional screens in private environments can be reduced while still effectively conveying information in an intuitive manner. To achieve this, two main approaches were em- ployed. Firstly, experimentation was conducted on the capture and communication of artificial light, and secondly, precise monitoring and visualization of indoor air quality were explored. These factors are fundamental indicators of both our mental and physical well- being, as high air quality, for example, enhances concentration, and light serves as a crucial regulator of our circadian rhythm.

more

Explaining rehearsals of a symphony orchestra in virtual reality

Zöllner, Michael; Krause, Moritz; Gemeinhardt, Jan; Bosl, Markus; Widmann, Dirk (2023)

25th International Conference on Human-Computer Interaction - Workshop on Interactive Technologies for Music Analysis and Visualization.


Peer Reviewed
 

The project Symotiv’s (http://symotiv.de) goal is to describe the complex function of a symphony through an interactive experience of the orchestra’s digital twin. We are capturing the musicians, their instruments and their sounds with machine learning based 3D human pose estimation and visualize them in Virtual Reality in a three-dimensional, virtual concert hall. Our audience steps into the role of the conductor and experience the scene and the sound from a perspective within the orchestra. This opens the possibility of experiencing and understanding what is happening from new spatial, but above all sonic perspectives.

more

Preliminary Studies and Prototypes for Machine Learning based Evaluation of Surfers’ Performance on River Waves

Zöllner, Michael; Krause, Moritz; Gemeinhardt, Jan; Döllinger, Michael...

25th International Conference on Human-Computer Interaction - Late Breaking Work.


Peer Reviewed
 

In this paper we are describing our preliminary studies and proto- types for evaluating a surfer’s performance on a stationary wave. We are briefly describing the sport’s environment and development and the current state-of- the-art of machine learning based single camera tracking approaches we have evaluated and applied. The main part of the paper deals with the first implemen- tation, the tracking and the results of the evaluation of the movements of two surfers. We are closing with our lessons learned and our next steps.

more

Understanding a Symphony Orchestra by Reexperience in Virtual Reality

Zöllner, Michael; Gemeinhardt, Jan; Bosl, Markus; Widmann, Dirk (2023)

25th International Conference on Human-Computer Interaction - Late Breaking Work.


Peer Reviewed
 

The aim of the project Symotiv (http://symotiv.de) is to introduce people to classical music through interaction with a virtual orchestra. With the help of Virtual Reality, it does not only enable to immerse oneself in a three-dimensional, virtual concert hall, but also to experience images and sound from the perspective of a musician. This opened the possibility of experiencing and understanding what is happening from new spatial, but above all sonic perspectives. Therefore, we developed a workflow to capture the motions and sound of 50 musicians and the conductor by using a camera-based Machine Learning approach.

more

Evaluation of Machine Learning based Pose Estimation of Surfers on River Waves

Zöllner, Michael; Kniesburges, Stefan; Krause, Moritz; Döllinger, Michael...

PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments 2023.
DOI: 10.1145/3594806


Peer Reviewed
 

In this work we are describing our preliminary experimental evaluation of open-source machine learning based pose estimation solutions for tracking a surfer’s skeleton on a stationary wave. We are briefly describing the sport’s environment and development and the current state-of-the-art of machine learning based single camera 2D and 3D pose estimation solutions. The main part of the paper deals with the experiment’s setup and the interpretation of the resulting data of the movements of one surfer. We are closing with our lessons learned and our next steps. 

more

Prof. Michael Zöllner


Hochschule für Angewandte Wissenschaften Hof

Forschungsgruppe Interaction & Data Driven Design (iddd)
Alfons-Goppel-Platz 1
95028 Hof

T +49 9281 409-8563
michael.zoellner[at]hof-university.de